Re: sendfile() not detecting closed connections.

Andre Oppermann wrote:
> Ian FREISLICH wrote:
> > Andre Oppermann wrote:
> > 
> >>Ian FREISLICH wrote:
> >>
> >>>Hi
> >>>
> >>>System is 8.0-CURRENT.  I have the following piece of code:
> >>>
> >>>	rename(path, data);
> >>>	stat(data, &sb);
> >>>	len = snprintf(buffer, MAXBUFLEN, "BYTES %lld\r\n", sb.st_size);
> >>>	write(connection, buffer, len);
> >>>	sleep(10);
> >>>	if ((sendfile(fd, connection, 0, sb.st_size, NULL,
> >>>	    &sbytes, 0)) == -1 || sbytes != sb.st_size) {
> >>>		syslog(facility, "Problem writing data: %s, wrote %lld",
> >>>		    strerror(errno), sbytes);
> >>>		respool(fd, path);
> >>>		unlink(data)
> >>>		close(fd);
> >>>		return(-1);
> >>>	}
> >>>	else
> >>>		syslog(facility, "Download successful %ld", sbytes);
> >>>	close(fd);
> >>>	unlink(data);
> >>>
> >>>If, during the sleep, I terminate the connection so that netstat
> >>>reports:
> >>>
> >>>tcp4       0      0  127.0.0.1.666          127.0.0.1.58239        CLOSE_W
A
> > 
> > IT
> > 
> >>>tcp4       0      0  127.0.0.1.58239        127.0.0.1.666          FIN_WAI
T
> > 
> > _2
> > 
> >>>sendfile() reports success for files less than about 64k in size,
> >>>but I haven't been able to figure out where the threshold is.  It
> >>>erroneously reports that 41000 of the 64k were sent, but will say
> >>>the whole file was transferred up to about 64k.  The connection
> >>>filedescriptor is blocking.
> >>>
> >>>Any ideas?
> >>
> >>sendfile() reports the bytes written into the send socket buffer.  If
> >>there is a connection error it doesn't (and never did) look at how
> >>much data was still in the socket buffer. The sendfile(2) man page
> >>says: "[sbytes] If non-NULL, the system will write the total number
> >>of bytes sent on the socket to the variable pointed to by sbytes."
> >>This could be changed to subtract the remaining data in the socket
> >>buffer before reporting back.  One has to be careful though about
> >>other writes so that the number never goes negative.  There may be
> >>more data remaining in the socket buffer than from this write attempt
> >>alone.
> > 
> > 
> > The connection was closed about 6 seconds before I called sendfile().
> > Would sendfile() write to the socket buffer of a socket closed that
> > long ago?
> 
> The sendfile(2) syscall checks if the socket is still connected:
> 
> 	if ((so->so_state & SS_ISCONNECTED) == 0) {
> 		error = ENOTCONN;
> 		goto out;
> 	}
> 
> Due to timing circumstances the connection may still be technically
> connected while at the same time disconnecting when FIN exchange has
> not completed yet.
> 
> To prevent a sendfile(2) call on a disconnecting socket another test
> has to be added:
> 
> 	if (so->so_state & SS_ISDISCONNECTING) {
> 		error = EPIPE;		/* or ESHUTDOWN? */
> 		goto out;
> 	}
> 
> Please add this test after line 1841 in sys/kern/uipc_syscalls.c and
> test again.  The error code ESHUTDOWN seems more appropriate but is
> so far not documented in the sendfile(2) man page.  As it seems this
> bug has been present since the introduction of the sendfile syscall.

I'm not sure that this has changed much.  It errors with errno ==
EPIPE, but still claims it's placed 40960 bytes in the socket buffer.

A tight loop of netstat -anf inet never reveals a Send-Q greater
than 0 bytes, although it's entirely likely it missed the event.

But if I terminate the connection mid stream I can see a send queue
sticking around for a bit:
tcp4       0  43640  127.0.0.1.666          127.0.0.1.50916        CLOSE_WAIT

Ian

--
Ian Freislich