Re: throughput and interrupts

On Wed, 16 Aug 2006, Bachilo Dmitry wrote:

> Here is what I see:
>
> CPU states:  9.6% user,  0.0% nice, 58.5% system, 31.9% interrupt,  0.0% idle
> Mem: 127M Active, 664M Inact, 138M Wired, 48M Cache, 111M Buf, 21M Free
> Swap: 64M Total, 272K Used, 64M Free
>
>  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
>   11 root     171   52     0K     8K RUN    422.7H 33.06% 33.06% idle
>  462 root     109    0  1496K   952K RUN    202:44 21.53% 21.53% natd
>   37 root     -44 -163     0K     8K WAIT    99:09  5.08%  5.08% swi1: net
>   29 root     -68 -187     0K     8K RUN     59:11  3.52%  3.52% irq19: xl0
>   30 root     -68 -187     0K     8K WAIT    40:38  3.47%  3.47% irq20: xl1
>
> This machine is Celeron 1,7 Ghz and 1 GB RAM and it gives me 3,7 MB per 
> second max. While my home router is Celeron 1,7 Ghz with 512 MB RAM, the 
> same xl NICs and I have almost 10 MB per second there with at least 48% of 
> free CPU. I'll keep on testing this stuff, so i'll keep reporting the 
> results.

You appear to have omitted the systat -vmstat 1 output.

BTW, there are typically three influences on processing cost for traffic: 
packets per second, size of packets, and contents of packets.  In the case of 
natd, the contents are almost irrelevant, so the costs are really based on 
number of packets processed, and the size of the packets.  You compare the two 
systems in terms of bandwidth, which is more often a property of packet size. 
Could you compare the systems in terms of number of packets per second through 
the system?  As each packet requires two (or more) system calls for natd, lots 
of small packets can cost more to process than a smaller number of bigger 
packets.  If your home system is processing larger packets, it may use less 
CPU than the same system processing many smaller packets at the office, for 
example.

Finally, are you running any tools such as tcpdump, trafshow, etc, which use 
bpf?  Any additional programs that perform per-packet operations will result 
in additional per-packet context switches, forcing full context switches 
between processes, not just switching in and out of the kernel, which is 
lighter-weight.  This can have a significant impact on performance, perhaps 
more significant than expected.

Robert N M Watson
Computer Laboratory
University of Cambridge