Re: wierd dsl performance with -CURRENT

Kenneth Culver wrote:
> > Just as an experiment, try setting "net.inet.tcp.newreno" to 0 using
> > sysctl(8).  It might help; it might not.  Please let us know.
> 
> It didn't help. I also tried setting several other sysctl OID's in
> net.inet.tcp, but nothing helped. I'm totally out of ideas for why this
> could be happening. I mean it IS only 16KB/sec, but it seems wierd that
> only FreeBSD is having problems out of all the OS's I use at home.

Your Macintosh processes network traffic differently than FreeBSD
does.  What it does is basically this:

o	Network interrupts kick off a Mach thread that runs the
	data from the card into a buffer in the IOKit driver,
	until there is no more data, or until the buffer hits
	a high watermark.

o	They kick another Mach thread telling it there's data,
	and that Mach thread begins servicing the data.  In
	the meantime, you may get more interrupts.  The second
	Mach thread loads the data through ether_input.  This
	in turn goes through the network stack, and is, in effect,
	the NETISR processing as well.

What this basically means is that there is a lot less latency in
the MacOS X stack than there is in the normal FreeBSD stack.

You can't do this without redesigning FreeBSD's network handling,
and getting away from the 5.x "interrupt thread" style architecture;
that's a long discussion, though.

But there are some things you can do.

You should visit the FreeBSD -performance list archives for a
(fairly) recent discussion on network performance (I believe
between a couple of us, we were able to come up with tuning
parameters that improved someone's file transfer performance by
about a factor of 10 for some tests).

The major part of this discussion was about turning on direct
dispatch, dealing with polling, tuning TCP paramters via sysctl,
and setting a number of kernel options.

I would *NOT* recommend adjusting the TCP timers in your case, but
you *could* make the NETISR happen more frequently by upping your
HZ value.

If you want to discuss performance issues in networking further,
-performance or -net are much better lists to use than -current,
which is kind of a generic catch-all list for work on FreeBSD 5.x.


Or... you could also just do the work and fix the code.

Now, you are using a link with a big downchannel and a large
upchannel; the ideal way to use such a link is called "rate based
ACK", which basically means that you send your ACK up at timed
intervals according to the data rate, instead of sending it up
at "window full" intervals and risking getting stalled.  There
is a FreeBSD committer who got her Master's degree based on work
in this area: Jennifer Yang.

One way to (almost) get this effect would be to set the "PUSH"
option on the socket; this won't effect ACK's in FreeBSD, though.

Another way is to implement TCP Rate Halving.  This is much better
than New Reno at congestion avoidance; it was implemented in a
fairly recent NetBSD stack at Carnegie-Mellon's Pittsburgh
Supercomputing Center (CMU PSC).  Their implementation is available
on the net for download, but you would have to hack it.

Another way is called Lazy Receiver Processing (LRP).  LRP comes
out of Peter Druschel's group at Rice University, and was developed
as part of the Scala Server Project.  Mohit Aron, who last I heard
wa working for Zambeel (a distributed storage area network FS
company in the Bay Area) did the most recent work on that for 4.x
FreeBSD; it has a pretty bad license.  There is also a port of it
to a more recent 4.x by a researcher from Duke University.  This
would be an easy port, but the code isn't really that production,
and if you were going to productize it, you should probably start
with the ancient 2.4 implementation, which has a free license.

I mention this last, because I'm always hopeful that someone who
has a chance of getting the code committed because they already
have commit bits, and it's easier to ask for forgiveness than
permission, will do the work on the other, better approaches.  ;^).

-- Terry