Re: FILE's _file can only hold a short

On Fri, Jan 11, 2013 at 7:44 AM, John Baldwin <jhb_at_freebsd.org> wrote:
> On Wednesday, October 31, 2012 02:12:55 PM mdf_at_freebsd.org wrote:
>> I seem to recall a thread earlier on this limitation, but looking at
>> actual libc/stdio sources, the 4 year old check for open(2)'s fd being
>> less than SHRT_MAX is still there.  I thought I saw a patch to change
>> this to an int, but it's not in the tree.  Was this in a PR or a
>> mailing list thread or am I just imagining things?
>>
>> We've run into this limitation at work, where some processes have
>> around 32k open file descriptors and then try to use the libc FILE
>> interface.  Since we control ABI we can just change this to int, but I
>> had been hoping there was a FreeBSD revision we could pull instead of
>> having another diff.
>
> I had been working on a port-exp run.  The problem I have run into is
> that perl actually reaches inside of FILE directly to clear out _file
> so it can control when the fd is actually closed (really gross).  I
> have extended my stuff so that old Perl binaries should still work, but
> wanted to figure out how to prevent future Perl binaries from growing
> the same dependency.  Also, I haven't had a chance to do a follow-up
> to find what else out in ports-land tries to use _file directly from
> FILE.

John,

I posted a problem I am having with a perl script that opens multiple
filehandles with output to a pipe (open $fh,"command|";). The script
has worked fine for over a decade, but an upgrade of FreeBSD from 7.2
to 8.3 caused the program to fail to transfer the entire output
(between 14 and 30KB of data in lines of between 1 and 80 bytes.
(Output of show bgp summary on a Juniper.) The sub-process just hangs
until my watchdog sends a sigterm after 15 seconds. When I read the
filehadle, I get only a truncated output from these processes. Others
complete normally with full output. the failures tend to be ones with
larger outputs, but not always.

I have worked around this, but I would love to know why it broke in
8.3 after working since at least v5 and probably 4.3. Since this is a
common technique or running multiple commands in parallel in perl, it
would be nice if I could at least understand what is going on. This
looks a bit like what this thread is discussing, but I am far from
certain.
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6558_at_gmail.com