Re: throughput and interrupts

On Tue, 15 Aug 2006, Bachilo Dmitry wrote:

> I am completely stuck. I have a router, that rules three subnets. At first 
> there were Allied Telesyn's netcards with realtek chipsets. I am using 
> ipfw+natd and I had like 3,5 megabytes per second with 20 per cent of 
> interrupt load. I had "discarded oversized frame" too often so I've replaced 
> two main cards with 3Com 905 and now my xl0 and xl1 never discard frames. 
> But I saw no speed boost and interrupts were also very high. I have 
> discovered some IRQ conflicts and removed them, now I have single device per 
> IRQ, but still i have near 20-30 % of interrupt load, when i download 
> something through this router (and natd takes rest % of CPU). Some people 
> advised me to use polling, so I've built kernel with HZ=1000, polling and 
> tried it. Interrupts grew to 40-50% and speed decreased to 200-300 kb/s, so 
> I had to turn the polling off.
>
> I just don't know what to try, to gain at least 9 mb/s, because this is what 
> I get at home with the same cards, the same processor, with no polling and 
> throgh the same provider.

I think it would be useful to compare the settled loads of the two systems, if 
they are otherwise identical, using systat -vmstat 1 and top -S.  I'd load the 
systems up, leave the tools running, and wait a couple of minutes, then 
compare snapshots of the output from both systems.  top -S is particularly 
informative, as it will tell you which ithreads are burning the most CPU, and 
systat -vmstat will tell us the context switch rate and interrupt rates of 
various sources.

There are some general optimization techniques that can be applied here -- for 
example, if it's an SMP box, you might find that net.isr.direct=1 improves 
performance.  However, I think the real problem here is the performance 
difference, which suggests a problem that needs resolving, rather than 
optimization being required up front.

Thanks,

Robert N M Watson
Computer Laboratory
University of Cambridge