Re: I/O semantics of pipe and FIFO.

Problem we had found was:

Executing dd with a closed stdout and stderr would cause the summary messages printed at the end to go into the destination output file.

For example,

dd if=/dev/zero of=/tmp/foo bs=1m count=1

Works fine, but the following:

dd if=/dev/zero of=/tmp/foo bs=1m count=1 >&- 2>&-

Will cause the summary statistics of dd to appear in /tmp/foo instead of on the console.

The issue is that the summary statistics are send to fd1, which if you close down stdout and stdin, fd1 is actually the output file since it got the lowest file descriptor available when open(2) was called on the output file.

This was never fixed because it was deemed “silly developer, don’t close stdout and stderr before invoking dd”.

The argument has been made by Jilles T. that it is generally a bad idea to close down any of the standard file descriptors because it cannot be predicted how a particular UNIX utility will react (e.g., in the case of dd, causing a simple printf(3) to go to an unexpected location).
— 
Devin


> On Mar 4, 2017, at 8:12 PM, Alfred Perlstein <alfred_at_freebsd.org> wrote:
> 
> Devin and I found this when we worked together.  I think it was due to some situation in dd(1) where short reads would exit pre-maturely, however I may be mis-remembering.  Devin, do you recall the specifics?
> 
> 
> On 3/4/17 7:44 PM, Julian Elischer wrote:
>> 
>> an interesting point to discuss? is our behaviour in this test right?
>>   from: "austin-group mailng list (posix standard discussion)"
>> 
>> ------ rest of email is quoted -------
>> On 5/3/17 5:48 am, Stephane Chazelas wrote:
>> 
>> 2017-03-04 13:14:08 +0000, Danny Niu:
>>> Hi all.
>>> 
>>> I couldn't remember where I saw it saying, that when reading
>>> from a pipe or a FIFO, the read syscall returns the content of
>>> at most one write call. It's a bit similar to the
>>> message-nondiscard semantics of dear old STREAM.
>>> 
>>> Currently, I'm reading through the text to find out a bit
>>> more, and I appreciate a bit of pointer on this.
>> [...]
>> 
>> (echo x; echo y) | (sleep 1; dd count=1 2> /dev/null)
>> 
>> outputs both x and y in all of Linux, FreeBSD and Solaris in my
>> tests.
>> 
>> That a read wouldn't read what's currently in the pipe would be
>> quite surprising.
>> 
>> I also wouldn't expect pipes to store the writes as individual
>> separate message but use one buffer.
>> 
>> In:
>> 
>> (
>>  dd bs=40000 count=1 if=/dev/zero 2> /dev/null
>>  echo first through >&2
>>  dd bs=40000 count=1 if=/dev/zero 2> /dev/null
>>  echo second through >&2
>> ) | (sleep 1; dd bs=100000 count=1 2> /dev/null) | wc -c
>> 
>> That is where the second write blocks because the pipe is full,
>> the reading dd still reads both writes in Linux and Solaris in
>> my tests (on Solaris (10 on amd64 at least), reduce to 20000
>> instead of 40000 or both writes would block).
>> 
>> On FreeBSD, I get only the first write (using 8000 followed by
>> 10000 for instance).
>> 
>> FreeBSD is also the only one of the three where
>> 
>> dd bs=1000000 count=1 if=/dev/zero | dd bs=1000000 count=1 | wc -c
>> 
>> Doesn't output 1000000. The others schedule both processes back
>> and forth during their write() and read() system call while the
>> pipe is being filled and emptied several times.
>> 
>