Re: 4.7 vs 5.2.1 SMP/UP bridging performance

Gerrit Nagelhout wrote:

>>>>I would like to move to CURRENT for new hardware support, and the 
>>>>ability to properly use multi-threading in user-space, but can't do
>>>>this until the performance bottlenecks are solved.  I realize that 
>>>>5.x is still a work in progress and hasn't been tuned as well as 4.7 
>>>>yet, but are there any plans for optimizations in this area?  Does 
>>>>anyone have any suggestions on what else I can try?
>>>
>>>
>>>Try rwatson's netperf patches:
>>>
>>>  http://www.watson.org/~robert/freebsd/netperf/
>>>
>>>There is at least one outstanding panic condition known, but more
>>>testing will be a great help.
>>>
>>>Kris
>>>
>>>P.S. You didn't mention the status of WITNESS, but I'm assuming you
>>>read the docs and disabled it since it's a huge performance killer.
> 
> 
>>WITNESS and INVARIANTS are turned off for the 5.2.1 release bits.
>>However, the debug.mpsafenet sysctl is also turned off.  Turning this
>>on might give a significant performance boost for bridging.
> 
> 
>>Scott
> 
> 
> Thanks for all the responses so far.  WITNESS is definitely disabled, 
> as are the other INVARIANTS.  I had a look through the netperf patches, 
> but I don't think they will affect bridging very much.  They seem be 
> directed more towards the socket layer and above.
> 
> I still think that one of the bigger bottlenecks is the cost of all
> the mutexes in SMP mode, and some of the new bus_dma and mbuf code that
> was introduced.  
> 
> With previous platforms I have worked on (vxWorks), we had similar 
> issues, and ended up pushing buckets of packets through the data path, 
> so each mutex was only taken once for every 10-100 packets.
> 
> Also, polling is currently done by only one CPU at a time.  If this 
> were changed to have multiple threads poll multiple devices at the
> same time, the performance should become much better.
> 
> Thanks,
> 
> Gerrit

You are correct about the netperf patches being directed towards the 
socket layer.  The IP stack and below was locked for 5.2, but the
benefits won't be seen unless you turn on debug.mpsafenet.  During
the 5.2 development cycle I believe that benchmarking was done that
showed that mpsafenet bridging was significantly faster than non-
mpsafenet, and nearly as fast as 4.x if not a little faster.

I'd be interest to know more about your comments about polling from
multiple CPUs.  Did you have a thread bound to each CPU, and did
each thread poll every interface, or only an exclusive subset of the
interfaces?

Scott