Re: quick summary results with ixgbe (was Re: datapoints on 10G throughput with TCP ?

On Dec 7, 2011, at 8:08 PM, Luigi Rizzo wrote:

> Summary:
> 
> - with default interrupt mitigation, the fastest configuration
>  is with checksums enabled on both sender and receiver, lro
>  enabled on the receiver. This gets about 8.0 Gbit/s

I do not observe this. With defaults:

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.053 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1857.4978 MB in 5.02 real seconds = 378856.02 KB/sec = 3103.5885 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 29720 I/O calls, msec/call = 0.17, calls/sec = 5919.63
nuttcp-t: 0.0user 2.5sys 0:05real 52% 115i+1544d 630maxrss 0+2pf 107264+1csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1857.4978 MB in 5.15 real seconds = 369617.39 KB/sec = 3027.9057 Mbps
nuttcp-r: 543991 I/O calls, msec/call = 0.01, calls/sec = 105709.95
nuttcp-r: 0.1user 4.1sys 0:05real 83% 110i+1482d 618maxrss 0+15pf 158432+0csw

On receiver:

ifconfig ix1 lro

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.068 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1673.3125 MB in 5.02 real seconds = 341312.36 KB/sec = 2796.0308 Mbps
nuttcp-t: host-retrans = 1
nuttcp-t: 26773 I/O calls, msec/call = 0.19, calls/sec = 5333.01
nuttcp-t: 0.0user 1.0sys 0:05real 21% 113i+1518d 630maxrss 0+2pf 12772+1csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1673.3125 MB in 5.15 real seconds = 332975.19 KB/sec = 2727.7327 Mbps
nuttcp-r: 106268 I/O calls, msec/call = 0.05, calls/sec = 20650.82
nuttcp-r: 0.0user 1.3sys 0:05real 28% 101i+1354d 618maxrss 0+15pf 64567+0csw

On sender:

ifconfig ix1 lro

(now both receiver and sender have LRO enabled)

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.063 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1611.7805 MB in 5.02 real seconds = 328716.18 KB/sec = 2692.8430 Mbps
nuttcp-t: host-retrans = 1
nuttcp-t: 25789 I/O calls, msec/call = 0.20, calls/sec = 5136.29
nuttcp-t: 0.0user 1.0sys 0:05real 21% 109i+1465d 630maxrss 0+2pf 12697+0csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1611.7805 MB in 5.15 real seconds = 320694.82 KB/sec = 2627.1319 Mbps
nuttcp-r: 104346 I/O calls, msec/call = 0.05, calls/sec = 20275.05
nuttcp-r: 0.0user 1.3sys 0:05real 27% 113i+1516d 618maxrss 0+15pf 63510+0csw

remove LRO from receiver (only sender has LRO):

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.065 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1884.8702 MB in 5.02 real seconds = 384464.57 KB/sec = 3149.5338 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 30158 I/O calls, msec/call = 0.17, calls/sec = 6007.27
nuttcp-t: 0.0user 2.7sys 0:05real 55% 104i+1403d 630maxrss 0+2pf 106046+0csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1884.8702 MB in 5.15 real seconds = 375093.52 KB/sec = 3072.7661 Mbps
nuttcp-r: 540237 I/O calls, msec/call = 0.01, calls/sec = 104988.68
nuttcp-r: 0.1user 4.2sys 0:05real 84% 110i+1483d 618maxrss 0+15pf 156340+0csw

Strange enough, setting hw.ixgbe.max_interrupt_rate=0 does not have any effect..

Daniel