On Wed, Dec 07, 2011 at 09:58:31PM +0200, Daniel Kalchev wrote: > > On Dec 7, 2011, at 8:08 PM, Luigi Rizzo wrote: > > > Summary: > > > > - with default interrupt mitigation, the fastest configuration > > is with checksums enabled on both sender and receiver, lro > > enabled on the receiver. This gets about 8.0 Gbit/s > > I do not observe this. With defaults: > ... Sorry, forgot to mention that the above is with TSO DISABLED (which is not the default). TSO seems to have a very bad interaction with HWCSUM and non-zero mitigation. also remember that hw.ixgbe.max_interrupt_rate has only effect at module load -- i.e. you set it with the bootloader, or with kenv before loading the module. Please retry the measurements disabling tso (on both sides, but it really matters only on the sender). Also, LRO requires HWCSUM. cheers luigi > # nuttcp -t -T 5 -w 128 -v 10.2.101.11 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.053 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1857.4978 MB in 5.02 real seconds = 378856.02 KB/sec = 3103.5885 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 29720 I/O calls, msec/call = 0.17, calls/sec = 5919.63 > nuttcp-t: 0.0user 2.5sys 0:05real 52% 115i+1544d 630maxrss 0+2pf 107264+1csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.12 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1857.4978 MB in 5.15 real seconds = 369617.39 KB/sec = 3027.9057 Mbps > nuttcp-r: 543991 I/O calls, msec/call = 0.01, calls/sec = 105709.95 > nuttcp-r: 0.1user 4.1sys 0:05real 83% 110i+1482d 618maxrss 0+15pf 158432+0csw > > On receiver: > > ifconfig ix1 lro > > # nuttcp -t -T 5 -w 128 -v 10.2.101.11 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.068 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1673.3125 MB in 5.02 real seconds = 341312.36 KB/sec = 2796.0308 Mbps > nuttcp-t: host-retrans = 1 > nuttcp-t: 26773 I/O calls, msec/call = 0.19, calls/sec = 5333.01 > nuttcp-t: 0.0user 1.0sys 0:05real 21% 113i+1518d 630maxrss 0+2pf 12772+1csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.12 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1673.3125 MB in 5.15 real seconds = 332975.19 KB/sec = 2727.7327 Mbps > nuttcp-r: 106268 I/O calls, msec/call = 0.05, calls/sec = 20650.82 > nuttcp-r: 0.0user 1.3sys 0:05real 28% 101i+1354d 618maxrss 0+15pf 64567+0csw > > On sender: > > ifconfig ix1 lro > > (now both receiver and sender have LRO enabled) > > # nuttcp -t -T 5 -w 128 -v 10.2.101.11 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.063 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1611.7805 MB in 5.02 real seconds = 328716.18 KB/sec = 2692.8430 Mbps > nuttcp-t: host-retrans = 1 > nuttcp-t: 25789 I/O calls, msec/call = 0.20, calls/sec = 5136.29 > nuttcp-t: 0.0user 1.0sys 0:05real 21% 109i+1465d 630maxrss 0+2pf 12697+0csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.12 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1611.7805 MB in 5.15 real seconds = 320694.82 KB/sec = 2627.1319 Mbps > nuttcp-r: 104346 I/O calls, msec/call = 0.05, calls/sec = 20275.05 > nuttcp-r: 0.0user 1.3sys 0:05real 27% 113i+1516d 618maxrss 0+15pf 63510+0csw > > remove LRO from receiver (only sender has LRO): > > # nuttcp -t -T 5 -w 128 -v 10.2.101.11 > nuttcp-t: v6.1.2: socket > nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11 > nuttcp-t: time limit = 5.00 seconds > nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.065 ms > nuttcp-t: send window size = 131768, receive window size = 66608 > nuttcp-t: 1884.8702 MB in 5.02 real seconds = 384464.57 KB/sec = 3149.5338 Mbps > nuttcp-t: host-retrans = 0 > nuttcp-t: 30158 I/O calls, msec/call = 0.17, calls/sec = 6007.27 > nuttcp-t: 0.0user 2.7sys 0:05real 55% 104i+1403d 630maxrss 0+2pf 106046+0csw > > nuttcp-r: v6.1.2: socket > nuttcp-r: buflen=65536, nstream=1, port=5001 tcp > nuttcp-r: accept from 10.2.101.12 > nuttcp-r: send window size = 33304, receive window size = 131768 > nuttcp-r: 1884.8702 MB in 5.15 real seconds = 375093.52 KB/sec = 3072.7661 Mbps > nuttcp-r: 540237 I/O calls, msec/call = 0.01, calls/sec = 104988.68 > nuttcp-r: 0.1user 4.2sys 0:05real 84% 110i+1483d 618maxrss 0+15pf 156340+0csw > > Strange enough, setting hw.ixgbe.max_interrupt_rate=0 does not have any effect.. > > Daniel >Received on Wed Dec 07 2011 - 19:07:35 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:21 UTC