RE: dev.bce.X.com_no_buffers increasing and packet loss

> I successfully reproduced the issue with netperf on BCM5709. 
> You can use UDP frame size 1 to trigger the issue.
> 
> > Changing the high level design of bce_rx_intr() and
> > bce_rx_fill_chain() slightly to post a new buffer as each frame is 
> > passed to the OS would likely avoid these gaps during 
> bursts of small 
> > frames but I'm not sure whether they'll have a negative 
> impact on the 
> > more common case of streams of MTU sized frames.  I've 
> considered this 
> > in the past but never coded the change and tested the resulting 
> > performance.
> > 
> 
> I guess this may slightly increase performance with additional
> bus_dma(9) overheads but I think one of reason of dropping 
> frames under heavy UDP frames may come from lack of free RX 
> descriptors.
> Because bce(4) just uses a single RX ring so the number of 
> available RX buffers would be 512. However it seems it's not 
> possible to increase the number of RX buffers per RX ring so 
> the next possible approach would be switching to use multiple 
> RX rings with RSS.  Even though FreeBSD does not dynamically 
> adjust loads among CPUs I guess using RSS would be the way to go.

The bce(4) hardware supports a linked list of pages for RX 
buffer descriptors.  The stock build supports 2 pages (RX_PAGES)
with a total of 511 BD's per page.  The hardware can support a
maximum of 64K BD's but that would be an unnecessarily large
amount of mbufs for an infrequent problem.

The middle road would probably involve changing RX_PAGES from a
#define to a sysctl variable to allow tuning for specific
environments along with a change in bce_rx_intr() to fill the
ring after all frames have been processed or when more than
256 BDs have been consumed, whichever comes first.

RSS would be great as well though it wouldn't make a dent in
this case since RSS is only supported for TCP, not UDP.

Dave