RE: 4.7 vs 5.2.1 SMP/UP bridging performance

On Tue, 4 May 2004, Gerrit Nagelhout wrote:

> Thanks for all the responses so far.  WITNESS is definitely disabled, as
> are the other INVARIANTS.  I had a look through the netperf patches, but
> I don't think they will affect bridging very much.  They seem be
> directed more towards the socket layer and above. 

The reason to run with these patches is that, without them, it's not safe
to run without Giant over the lower half of the network stack, so all the
network interrupt threads are running with Giant otherwise.  Also, in that
patchset, network processing to completion runs in the ithread rather than
the netisr, so you get lower latency and more parallelism even for
bridging. 

Another known issue is the latency from interrupt generation to interrupt
task execution in the interrupt thread. 

> I still think that one of the bigger bottlenecks is the cost of all the
> mutexes in SMP mode, and some of the new bus_dma and mbuf code that was
> introduced.
> 
> With previous platforms I have worked on (vxWorks), we had similar
> issues, and ended up pushing buckets of packets through the data path,
> so each mutex was only taken once for every 10-100 packets. 
> 
> Also, polling is currently done by only one CPU at a time.  If this were
> changed to have multiple threads poll multiple devices at the same time,
> the performance should become much better. 

Getting polling and SMP to play nicely would be a very good thing, but
isn't something I currently have the bandwidth to work on.  Don't suppose
we could interest you in that? :-)

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research