Re: Re[4]: serious networking (em) performance (ggate and NFS) problem

On Sun, 2004-11-21 at 20:42 -0800, Matthew Dillon wrote:
> : Yes, I knew that adjusting TCP window size is important to use up a link.
> : However I wanted to show adjusting the parameters of Interrupt
> : Moderation affects network performance.
> :
> : And I think a packet loss was occured by enabled Interrupt Moderation.
> : The mechanism of a packet loss in this case is not cleared, but I think
> : inappropriate TCP window size is not the only reason.
> 
>     Packet loss is not likely, at least not for the contrived tests we
>     are doing because GiGE links have hardware flow control (I'm fairly
>     sure).

I have to disagree.  Packet loss is likely according to some of my
tests.  With the re driver, no change except placing a 100BT setup with
no packet loss to a gigE setup (both linksys switches) will cause
serious packet loss at 20Mbps data rates.  I have discovered the only
way to get good performance with no packet loss was to

1) Remove interrupt moderation
2) defrag each mbuf that comes in to the driver.

Doing both of these, I get excellent performance without any packet
loss.  All my testing has been with UDP packets, however, and nothing
was checked for TCP.

>     One could calculate the worst case small-packet build up in the receive
>     ring.  I'm not sure what the minimum pad for GiGE is, but lets say it's
>     64 bytes.  Then the packet rate would be around 1.9M pps or 244 packets
>     per interrupt at a moderation frequency of 8000 hz.  The ring is 256
>     packets.  But, don't forget the hardware flow control!  The switch
>     has some buffering too.
> 
>     hmm... me thinks I now understand why 8000 was chosen as the default :-)
> 
>     I would say that this means packet loss due to the interrupt moderation
>     is highly unlikely, at least in theory, but if one were paranoid one
>     might want to use a higher moderation frequency, say 16000 hz, to be sure.

Your calculations are based on the mbufs being a particular size, no?
What happens if they are seriously defragmented?  Is this what you mean
by "small-packet"?  Are you assuming the mbufs are as small as they get?
How small can they go?  1 byte? 1 MTU?

Increasing the interrupt moderation frequency worked on the re driver,
but it only made it marginally better.  Even without moderation,
however, I could lose packets without m_defrag.  I suspect that there is
something in the higher level layers that is causing the packet loss.  I
have no explanation why m_defrag makes such a big difference for me, but
it does.  I also have no idea why a 20Mbps UDP stream can lose data over
gigE phy and not lose anything over 100BT... without the above mentioned
changes that is.