Re: New optimized soreceive_stream() for TCP sockets, proof of concept

On Mon, 5 Mar 2007, Andrew Gallatin wrote:

> Robert Watson writes:
> > On Mon, 5 Mar 2007, Andrew Gallatin wrote:
> >
> > > With the patch, we finally seem to be performance competative on the receive
> > > side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of
> > > those OSes do much better (saturate the link with jumbos) when CPU affinity
> > > is used to bind the interrupt handler and netserver process to different
> > > cores on the same socket.  I imagine FreeBSD may be able to do even better
> > > if it ever grows CPU affinity support for both interrupt handlers and
> > > processes.  With the patch, it performs at least as well, if not better
> > > than, Solaris and Linux do without CPU affinity.
> >
> > I don't have numbers in front of me, and am currently packing for a trip 
> > to Tokyo so won't find them before traveling, but my experience has been 
> > that binding the ithread to a specific CPU is very helpful in improving 
> > receive performance.  You can slap a sched_bind(0) into the interrupt 
> > handler the first time it runs and it should stick appropriately, and add 
> > a sysctl to sched_bind() for a user process as a hack to test it out.
>
> You lost me at adding the sysctl for the user process.. Does FreeBSD have, 
> or plan to have, an interface to bind threads to CPUs?

Right now we have a facility to bind kernel threads to specific CPUs 
(sched_bind(9)), but not an API or mature facility to bind user threads 
(accept in as much as the scheduling of the user thread reflects the 
scheduling of the kernel thread, which works but requires adding a system call 
or sysctl as a hack).  There was a recent discussion on arch_at_ about the 
relative merits of various such user APIs floating around.  Adding a basic 
model for user thread CPU binding would be fairly easy, but I'm not sure 
anyone owns the task currently.

The quick hack model for testing is to sched_bind() the user thread using a 
sysctl added for testing purposes, which I believe should work to demonstrate 
the general point, and can be done in about 15 minutes.  Since I'm running out 
the door now, it will have to be someone else's 15 minutes. :-)

Robert N M Watson
Computer Laboratory
University of Cambridge