Re: dev.bce.X.com_no_buffers increasing and packet loss

On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote:
> 
> > I successfully reproduced the issue with netperf on BCM5709. 
> > You can use UDP frame size 1 to trigger the issue.
> > 
> > > Changing the high level design of bce_rx_intr() and
> > > bce_rx_fill_chain() slightly to post a new buffer as each frame is 
> > > passed to the OS would likely avoid these gaps during 
> > bursts of small 
> > > frames but I'm not sure whether they'll have a negative 
> > impact on the 
> > > more common case of streams of MTU sized frames.  I've 
> > considered this 
> > > in the past but never coded the change and tested the resulting 
> > > performance.
> > > 
> > 
> > I guess this may slightly increase performance with additional
> > bus_dma(9) overheads but I think one of reason of dropping 
> > frames under heavy UDP frames may come from lack of free RX 
> > descriptors.
> > Because bce(4) just uses a single RX ring so the number of 
> > available RX buffers would be 512. However it seems it's not 
> > possible to increase the number of RX buffers per RX ring so 
> > the next possible approach would be switching to use multiple 
> > RX rings with RSS.  Even though FreeBSD does not dynamically 
> > adjust loads among CPUs I guess using RSS would be the way to go.
> 
> The bce(4) hardware supports a linked list of pages for RX 
> buffer descriptors.  The stock build supports 2 pages (RX_PAGES)
> with a total of 511 BD's per page.  The hardware can support a
> maximum of 64K BD's but that would be an unnecessarily large
> amount of mbufs for an infrequent problem.
> 

Thanks for the info. I guess 2048 or 4096 BDs would be necessary to
get satisfactory Rx performance. I'll have to experiment this.

> The middle road would probably involve changing RX_PAGES from a
> #define to a sysctl variable to allow tuning for specific
> environments along with a change in bce_rx_intr() to fill the
> ring after all frames have been processed or when more than
> 256 BDs have been consumed, whichever comes first.
> 
> RSS would be great as well though it wouldn't make a dent in
> this case since RSS is only supported for TCP, not UDP.
> 

Even though UDP is not supported in RSS, RSS can handle IP. This
wouldn't distribute UDP load coming from a single host but if
source IP address is different it may help, I guess.

> Dave