: I did simple benchmark at some settings. : : I used two boxes which are single Xeon 2.4GHz with on-boarded em. : I measured a TCP throughput by iperf. : : These results show that the throughput of TCP increased if Interrupt :Moderation is turned OFF. At least, adjusting these parameters affected :TCP performance. Other appropriate combination of parameter may exist. Very interesting, but the only reason you get lower results is simply because the TCP window is not big enough. That's it. 8000 ints/sec = ~15KB of backlogged traffic. x 2 (sender, receiver) Multiply by two (both the sender's reception of acks and the receiver's reception of data) and you get ~30KB. This is awefully close to the default 32.5KB window size that iperf uses. Other then window sizing issues I can think of no rational reason why throughput would be lower. Can you? And, in fact, when I do the same tests on DragonFly and play with the interrupt throttle rate I get nearly the results I expect. * Shuttle Athlon 64 3200+ box, EM card in 32 bit PCI slot * 2 machines connected through a GiGE switch * All other hw.em0 delays set to 0 on both sides * throttle settings set on both sides * -w option set on iperf client AND server for 63.5KB window * software interrupt throttling has been turned off for these tests throttle result result freq (32.5KB win) (63.5KB win) (default) -------- -------------- ----------- maxrate 481 MBit/s 533 MBit/s (not sure what's going on here) 120000 518 MBit/s 558 MBit/s (not sure what's going on here) 100000 613 MBit/s 667 MBit/s (not sure what's going on here) 70000 679 MBit/s 691 MBit/s 60000 668 MBit/s 694 MBit/s 50000 678 MBit/s 684 MBit/s 40000 694 MBit/s 696 MBit/s 30000 694 MBit/s 696 MBit/s 20000 698 MBit/s 703 MBit/s 10000 707 MBit/s 716 MBit/s 9000 708 MBit/s 716 MBit/s 8000 710 MBit/s 717 MBit/s <--- drop off pt 32.5KB win 7000 683 MBit/s 716 MBit/s 6000 680 MBit/s 720 MBit/s 5000 652 MBit/s 718 MBit/s <--- drop off pt 63.5KB win 4000 555 Mbit/s 695 MBit/s 3000 522 MBit/s 533 MBit/s <--- GiGE throttling likely 2000 449 MBit/s 384 MBit/s (256 ring descriptors = 1000 260 MBit/s 193 MBit/s 2500 hz minimum) Unless you are in a situation where you need to route small packets flying around a cluster where low latency is important, it doesn't really make any sense to turn off interrupt throttling. It might make sense to change the default from 8000 to 10000 to handle typical default TCP window sizes (at least in a LAN situation), but it certainly should not be turned off. I got some weird results when I increased the frequency past 100KHz, and when I turned throttling off entirely. I'm not sure why. Maybe setting the ITR register to 0 is a bad idea. If I set it to 1 (i.e. 3906250 Hz) then I get 625 MBit/s. Setting the ITR to 1 (i.e. 256ns delay) should amount to the same thing as setting it to 0 but it doesn't. Very odd. The maximum interrupt rate as reported by systat is only ~46000 ints/sec so all the values above 50KHz should read about the same... and they do until we hit around 100Khz (10uS delay). Then everything goes to hell in a handbasket. Conclusion: 10000 hz would probably be a better default then 8000 hz. -Matt Matthew Dillon <dillon_at_backplane.com>Received on Sun Nov 21 2004 - 23:38:12 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:22 UTC