Re: datapoints on 10G throughput with TCP ?

Some tests with updated FreeBSD to 8-stable as of today, compared with 
the previous run


On 06.12.11 13:18, Daniel Kalchev wrote:
>
> FreeBSD 8.2-STABLE #0: Wed Sep 28 11:23:59 EEST 2011
> CPU: Intel(R) Xeon(R) CPU           E5620  _at_ 2.40GHz (2403.58-MHz 
> K8-class CPU)
> real memory  = 51539607552 (49152 MB)
> blade 1:
>
> # nuttcp -S
> # nuttcp -t -T 5 -w 128 -v localhost
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.044 ms
> nuttcp-t: send window size = 143360, receive window size = 71680
> nuttcp-t: 8959.8750 MB in 5.02 real seconds = 1827635.67 KB/sec = 
> 14971.9914 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 143358 I/O calls, msec/call = 0.04, calls/sec = 28556.81
> nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 602maxrss 0+5pf 16+46csw
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 127.0.0.1
> nuttcp-r: send window size = 43008, receive window size = 143360
> nuttcp-r: 8959.8750 MB in 5.17 real seconds = 1773171.07 KB/sec = 
> 14525.8174 Mbps
> nuttcp-r: 219708 I/O calls, msec/call = 0.02, calls/sec = 42461.43
> nuttcp-r: 0.0user 3.8sys 0:05real 76% 105i+1407d 614maxrss 1+17pf 
> 95059+22csw

New results:

FreeBSD 8.2-STABLE #1: Tue Dec  6 13:51:01 EET 2011


# nuttcp -t -T 5 -w 128 -v localhost
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.030 ms
nuttcp-t: send window size = 143360, receive window size = 71680
nuttcp-t: 12748.0625 MB in 5.02 real seconds = 2599947.38 KB/sec = 
21298.7689 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 203969 I/O calls, msec/call = 0.03, calls/sec = 40624.18
nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+2pf 1+82csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 127.0.0.1
nuttcp-r: send window size = 43008, receive window size = 143360
nuttcp-r: 12748.0625 MB in 5.15 real seconds = 2536511.81 KB/sec = 
20779.1048 Mbps
nuttcp-r: 297000 I/O calls, msec/call = 0.02, calls/sec = 57709.75
nuttcp-r: 0.1user 4.0sys 0:05real 81% 109i+1469d 626maxrss 0+15pf 
121136+34csw

Noticeable improvement.


>
> blade 2:
>
> # nuttcp -t -T 5 -w 128 -v 10.2.101.12
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.12
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.12 with mss=1448, RTT=0.059 ms
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 1340.6469 MB in 5.02 real seconds = 273449.90 KB/sec = 
> 2240.1016 Mbps
> nuttcp-t: host-retrans = 171
> nuttcp-t: 21451 I/O calls, msec/call = 0.24, calls/sec = 4272.78
> nuttcp-t: 0.0user 1.9sys 0:05real 39% 120i+1610d 600maxrss 2+3pf 
> 75658+0csw
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.11
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 1340.6469 MB in 5.17 real seconds = 265292.92 KB/sec = 
> 2173.2796 Mbps
> nuttcp-r: 408764 I/O calls, msec/call = 0.01, calls/sec = 78992.15
> nuttcp-r: 0.0user 3.3sys 0:05real 64% 105i+1413d 620maxrss 0+15pf 
> 105104+102csw

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.055 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1964.8640 MB in 5.02 real seconds = 400757.59 KB/sec = 
3283.0062 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 31438 I/O calls, msec/call = 0.16, calls/sec = 6261.87
nuttcp-t: 0.0user 2.7sys 0:05real 55% 112i+1501d 1124maxrss 1+2pf 
111165+112csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1964.8640 MB in 5.15 real seconds = 390972.20 KB/sec = 
3202.8442 Mbps
nuttcp-r: 560718 I/O calls, msec/call = 0.01, calls/sec = 108957.70
nuttcp-r: 0.1user 4.2sys 0:05real 84% 111i+1494d 626maxrss 0+15pf 
151930+16csw

Again, improvement.


>
>
> Another pari of blades:
>
> FreeBSD 8.2-STABLE #0: Tue Aug  9 12:37:55 EEST 2011
> CPU: AMD Opteron(tm) Processor 6134 (2300.04-MHz K8-class CPU)
> real memory  = 68719476736 (65536 MB)
>
> first blade:
>
> # nuttcp -S
> # nuttcp -t -T 5 -w 128 -v localhost
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.090 ms
> nuttcp-t: send window size = 143360, receive window size = 71680
> nuttcp-t: 2695.0625 MB in 5.00 real seconds = 551756.90 KB/sec = 
> 4519.9925 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 43121 I/O calls, msec/call = 0.12, calls/sec = 8621.20
> nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+4pf 2+71csw
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 127.0.0.1
> nuttcp-r: send window size = 43008, receive window size = 143360
> nuttcp-r: 2695.0625 MB in 5.14 real seconds = 536509.66 KB/sec = 
> 4395.0871 Mbps
> nuttcp-r: 43126 I/O calls, msec/call = 0.12, calls/sec = 8383.94
> nuttcp-r: 0.0user 3.1sys 0:05real 61% 94i+1264d 624maxrss 1+16pf 
> 43019+0csw

New result:

FreeBSD 8.2-STABLE #1: Tue Dec  6 15:06:22 EET 2011


# nuttcp -t -T 5 -w 128 -v localhost
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.094 ms
nuttcp-t: send window size = 143360, receive window size = 71680
nuttcp-t: 2659.1250 MB in 5.00 real seconds = 544493.51 KB/sec = 
4460.4909 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 42546 I/O calls, msec/call = 0.12, calls/sec = 8507.71
nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 614maxrss 0+3pf 1+71csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 127.0.0.1
nuttcp-r: send window size = 43008, receive window size = 143360
nuttcp-r: 2659.1250 MB in 5.14 real seconds = 529484.10 KB/sec = 
4337.5338 Mbps
nuttcp-r: 42560 I/O calls, msec/call = 0.12, calls/sec = 8275.91
nuttcp-r: 0.0user 3.1sys 0:05real 62% 103i+1391d 618maxrss 0+16pf 42454+0csw

Same result.. something is wrong with this Opteron system.

>
> second blade:
>
> # nuttcp -t -T 5 -w 128 -v 10.2.101.13
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.164 ms
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 1290.3750 MB in 5.00 real seconds = 264173.96 KB/sec = 
> 2164.1131 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 20646 I/O calls, msec/call = 0.25, calls/sec = 4127.72
> nuttcp-t: 0.0user 3.8sys 0:05real 77% 96i+1299d 616maxrss 0+3pf 
> 27389+0csw
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.14
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 1290.3750 MB in 5.14 real seconds = 256835.92 KB/sec = 
> 2103.9998 Mbps
> nuttcp-r: 85668 I/O calls, msec/call = 0.06, calls/sec = 16651.70
> nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1437d 624maxrss 0+15pf 
> 11848+0csw
>

# nuttcp -t -T 5 -w 128 -v 10.2.101.13
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.130 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1317.6875 MB in 5.00 real seconds = 269767.01 KB/sec = 
2209.9313 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 21083 I/O calls, msec/call = 0.24, calls/sec = 4215.11
nuttcp-t: 0.0user 3.7sys 0:05real 75% 105i+1409d 616maxrss 0+3pf 39629+11csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.14
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1317.6875 MB in 5.14 real seconds = 262306.61 KB/sec = 
2148.8157 Mbps
nuttcp-r: 85980 I/O calls, msec/call = 0.06, calls/sec = 16714.53
nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1446d 618maxrss 0+15pf 10783+0csw

Same,but.. again interrupt storm:

interrupt storm detected on "irq269:"; throttling interrupt source

Also interesting, Intel (server) - AMD (client)

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.087 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1301.1125 MB in 5.00 real seconds = 266388.35 KB/sec = 
2182.2534 Mbps
nuttcp-t: host-retrans = 1
nuttcp-t: 20818 I/O calls, msec/call = 0.25, calls/sec = 4162.36
nuttcp-t: 0.0user 4.3sys 0:05real 87% 106i+1430d 612maxrss 0+3pf 88509+11csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.13
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1301.1125 MB in 5.14 real seconds = 259062.19 KB/sec = 
2122.2374 Mbps
nuttcp-r: 537176 I/O calls, msec/call = 0.01, calls/sec = 104449.37
nuttcp-r: 0.0user 3.2sys 0:05real 64% 122i+1643d 626maxrss 0+15pf 
222593+0csw

AMD (server) - Intel (client)

  # nuttcp -t -T 5 -w 128 -v 10.2.101.14
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.14
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.14 with mss=1448, RTT=0.099 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 750.8897 MB in 5.02 real seconds = 153161.09 KB/sec = 
1254.6957 Mbps
nuttcp-t: host-retrans = 25
nuttcp-t: 12015 I/O calls, msec/call = 0.43, calls/sec = 2393.29
nuttcp-t: 0.0user 0.9sys 0:05real 19% 112i+1507d 618maxrss 0+2pf 97668+4csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.11
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 750.8897 MB in 5.15 real seconds = 149424.38 KB/sec = 
1224.0845 Mbps
nuttcp-r: 213009 I/O calls, msec/call = 0.02, calls/sec = 41394.56
nuttcp-r: 0.0user 4.6sys 0:05real 91% 107i+1445d 618maxrss 0+16pf 18273+0csw

It seems performance measurements are more dependent on the server 
(nuttcp -S) machine.
We will have to rule out the interrupt storms first of course, any advice?

Daniel