Re: datapoints on 10G throughput with TCP ?

Set the storm threshold to 0, that will disable it, its going to throttle
your performance
when it happens.

Jack


On Tue, Dec 6, 2011 at 6:24 AM, Daniel Kalchev <daniel_at_digsys.bg> wrote:

> Some tests with updated FreeBSD to 8-stable as of today, compared with the
> previous run
>
>
>
> On 06.12.11 13:18, Daniel Kalchev wrote:
>
>>
>> FreeBSD 8.2-STABLE #0: Wed Sep 28 11:23:59 EEST 2011
>> CPU: Intel(R) Xeon(R) CPU           E5620  _at_ 2.40GHz (2403.58-MHz
>> K8-class CPU)
>> real memory  = 51539607552 (49152 MB)
>> blade 1:
>>
>> # nuttcp -S
>> # nuttcp -t -T 5 -w 128 -v localhost
>> nuttcp-t: v6.1.2: socket
>> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
>> nuttcp-t: time limit = 5.00 seconds
>> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.044 ms
>> nuttcp-t: send window size = 143360, receive window size = 71680
>> nuttcp-t: 8959.8750 MB in 5.02 real seconds = 1827635.67 KB/sec =
>> 14971.9914 Mbps
>> nuttcp-t: host-retrans = 0
>> nuttcp-t: 143358 I/O calls, msec/call = 0.04, calls/sec = 28556.81
>> nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 602maxrss 0+5pf 16+46csw
>>
>> nuttcp-r: v6.1.2: socket
>> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
>> nuttcp-r: accept from 127.0.0.1
>> nuttcp-r: send window size = 43008, receive window size = 143360
>> nuttcp-r: 8959.8750 MB in 5.17 real seconds = 1773171.07 KB/sec =
>> 14525.8174 Mbps
>> nuttcp-r: 219708 I/O calls, msec/call = 0.02, calls/sec = 42461.43
>> nuttcp-r: 0.0user 3.8sys 0:05real 76% 105i+1407d 614maxrss 1+17pf
>> 95059+22csw
>>
>
> New results:
>
> FreeBSD 8.2-STABLE #1: Tue Dec  6 13:51:01 EET 2011
>
>
>
> # nuttcp -t -T 5 -w 128 -v localhost
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.030 ms
>
> nuttcp-t: send window size = 143360, receive window size = 71680
> nuttcp-t: 12748.0625 MB in 5.02 real seconds = 2599947.38 KB/sec =
> 21298.7689 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 203969 I/O calls, msec/call = 0.03, calls/sec = 40624.18
> nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+2pf 1+82csw
>
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 127.0.0.1
> nuttcp-r: send window size = 43008, receive window size = 143360
> nuttcp-r: 12748.0625 MB in 5.15 real seconds = 2536511.81 KB/sec =
> 20779.1048 Mbps
> nuttcp-r: 297000 I/O calls, msec/call = 0.02, calls/sec = 57709.75
> nuttcp-r: 0.1user 4.0sys 0:05real 81% 109i+1469d 626maxrss 0+15pf
> 121136+34csw
>
> Noticeable improvement.
>
>
>
>
>> blade 2:
>>
>> # nuttcp -t -T 5 -w 128 -v 10.2.101.12
>> nuttcp-t: v6.1.2: socket
>> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.12
>> nuttcp-t: time limit = 5.00 seconds
>> nuttcp-t: connect to 10.2.101.12 with mss=1448, RTT=0.059 ms
>> nuttcp-t: send window size = 131768, receive window size = 66608
>> nuttcp-t: 1340.6469 MB in 5.02 real seconds = 273449.90 KB/sec =
>> 2240.1016 Mbps
>> nuttcp-t: host-retrans = 171
>> nuttcp-t: 21451 I/O calls, msec/call = 0.24, calls/sec = 4272.78
>> nuttcp-t: 0.0user 1.9sys 0:05real 39% 120i+1610d 600maxrss 2+3pf
>> 75658+0csw
>>
>> nuttcp-r: v6.1.2: socket
>> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
>> nuttcp-r: accept from 10.2.101.11
>> nuttcp-r: send window size = 33304, receive window size = 131768
>> nuttcp-r: 1340.6469 MB in 5.17 real seconds = 265292.92 KB/sec =
>> 2173.2796 Mbps
>> nuttcp-r: 408764 I/O calls, msec/call = 0.01, calls/sec = 78992.15
>> nuttcp-r: 0.0user 3.3sys 0:05real 64% 105i+1413d 620maxrss 0+15pf
>> 105104+102csw
>>
>
> # nuttcp -t -T 5 -w 128 -v 10.2.101.11
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
>
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.055 ms
>
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 1964.8640 MB in 5.02 real seconds = 400757.59 KB/sec = 3283.0062
> Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 31438 I/O calls, msec/call = 0.16, calls/sec = 6261.87
> nuttcp-t: 0.0user 2.7sys 0:05real 55% 112i+1501d 1124maxrss 1+2pf
> 111165+112csw
>
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.12
>
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 1964.8640 MB in 5.15 real seconds = 390972.20 KB/sec = 3202.8442
> Mbps
> nuttcp-r: 560718 I/O calls, msec/call = 0.01, calls/sec = 108957.70
> nuttcp-r: 0.1user 4.2sys 0:05real 84% 111i+1494d 626maxrss 0+15pf
> 151930+16csw
>
> Again, improvement.
>
>
>
>>
>> Another pari of blades:
>>
>> FreeBSD 8.2-STABLE #0: Tue Aug  9 12:37:55 EEST 2011
>> CPU: AMD Opteron(tm) Processor 6134 (2300.04-MHz K8-class CPU)
>> real memory  = 68719476736 (65536 MB)
>>
>> first blade:
>>
>> # nuttcp -S
>> # nuttcp -t -T 5 -w 128 -v localhost
>> nuttcp-t: v6.1.2: socket
>> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
>> nuttcp-t: time limit = 5.00 seconds
>> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.090 ms
>> nuttcp-t: send window size = 143360, receive window size = 71680
>> nuttcp-t: 2695.0625 MB in 5.00 real seconds = 551756.90 KB/sec =
>> 4519.9925 Mbps
>> nuttcp-t: host-retrans = 0
>> nuttcp-t: 43121 I/O calls, msec/call = 0.12, calls/sec = 8621.20
>> nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+4pf 2+71csw
>>
>> nuttcp-r: v6.1.2: socket
>> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
>> nuttcp-r: accept from 127.0.0.1
>> nuttcp-r: send window size = 43008, receive window size = 143360
>> nuttcp-r: 2695.0625 MB in 5.14 real seconds = 536509.66 KB/sec =
>> 4395.0871 Mbps
>> nuttcp-r: 43126 I/O calls, msec/call = 0.12, calls/sec = 8383.94
>> nuttcp-r: 0.0user 3.1sys 0:05real 61% 94i+1264d 624maxrss 1+16pf
>> 43019+0csw
>>
>
> New result:
>
> FreeBSD 8.2-STABLE #1: Tue Dec  6 15:06:22 EET 2011
>
>
>
> # nuttcp -t -T 5 -w 128 -v localhost
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.094 ms
>
> nuttcp-t: send window size = 143360, receive window size = 71680
> nuttcp-t: 2659.1250 MB in 5.00 real seconds = 544493.51 KB/sec = 4460.4909
> Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 42546 I/O calls, msec/call = 0.12, calls/sec = 8507.71
> nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 614maxrss 0+3pf 1+71csw
>
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 127.0.0.1
> nuttcp-r: send window size = 43008, receive window size = 143360
> nuttcp-r: 2659.1250 MB in 5.14 real seconds = 529484.10 KB/sec = 4337.5338
> Mbps
> nuttcp-r: 42560 I/O calls, msec/call = 0.12, calls/sec = 8275.91
> nuttcp-r: 0.0user 3.1sys 0:05real 62% 103i+1391d 618maxrss 0+16pf
> 42454+0csw
>
> Same result.. something is wrong with this Opteron system.
>
>
>
>> second blade:
>>
>> # nuttcp -t -T 5 -w 128 -v 10.2.101.13
>> nuttcp-t: v6.1.2: socket
>> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13
>> nuttcp-t: time limit = 5.00 seconds
>> nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.164 ms
>> nuttcp-t: send window size = 131768, receive window size = 66608
>> nuttcp-t: 1290.3750 MB in 5.00 real seconds = 264173.96 KB/sec =
>> 2164.1131 Mbps
>> nuttcp-t: host-retrans = 0
>> nuttcp-t: 20646 I/O calls, msec/call = 0.25, calls/sec = 4127.72
>> nuttcp-t: 0.0user 3.8sys 0:05real 77% 96i+1299d 616maxrss 0+3pf 27389+0csw
>>
>> nuttcp-r: v6.1.2: socket
>> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
>> nuttcp-r: accept from 10.2.101.14
>> nuttcp-r: send window size = 33304, receive window size = 131768
>> nuttcp-r: 1290.3750 MB in 5.14 real seconds = 256835.92 KB/sec =
>> 2103.9998 Mbps
>> nuttcp-r: 85668 I/O calls, msec/call = 0.06, calls/sec = 16651.70
>> nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1437d 624maxrss 0+15pf
>> 11848+0csw
>>
>>
> # nuttcp -t -T 5 -w 128 -v 10.2.101.13
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.130 ms
>
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 1317.6875 MB in 5.00 real seconds = 269767.01 KB/sec = 2209.9313
> Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 21083 I/O calls, msec/call = 0.24, calls/sec = 4215.11
> nuttcp-t: 0.0user 3.7sys 0:05real 75% 105i+1409d 616maxrss 0+3pf
> 39629+11csw
>
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.14
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 1317.6875 MB in 5.14 real seconds = 262306.61 KB/sec = 2148.8157
> Mbps
> nuttcp-r: 85980 I/O calls, msec/call = 0.06, calls/sec = 16714.53
> nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1446d 618maxrss 0+15pf
> 10783+0csw
>
> Same,but.. again interrupt storm:
>
> interrupt storm detected on "irq269:"; throttling interrupt source
>
> Also interesting, Intel (server) - AMD (client)
>
> # nuttcp -t -T 5 -w 128 -v 10.2.101.11
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.11
>
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.11 with mss=1448, RTT=0.087 ms
>
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 1301.1125 MB in 5.00 real seconds = 266388.35 KB/sec = 2182.2534
> Mbps
> nuttcp-t: host-retrans = 1
> nuttcp-t: 20818 I/O calls, msec/call = 0.25, calls/sec = 4162.36
> nuttcp-t: 0.0user 4.3sys 0:05real 87% 106i+1430d 612maxrss 0+3pf
> 88509+11csw
>
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.13
>
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 1301.1125 MB in 5.14 real seconds = 259062.19 KB/sec = 2122.2374
> Mbps
> nuttcp-r: 537176 I/O calls, msec/call = 0.01, calls/sec = 104449.37
> nuttcp-r: 0.0user 3.2sys 0:05real 64% 122i+1643d 626maxrss 0+15pf
> 222593+0csw
>
> AMD (server) - Intel (client)
>
>  # nuttcp -t -T 5 -w 128 -v 10.2.101.14
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.14
>
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.14 with mss=1448, RTT=0.099 ms
>
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 750.8897 MB in 5.02 real seconds = 153161.09 KB/sec = 1254.6957
> Mbps
> nuttcp-t: host-retrans = 25
> nuttcp-t: 12015 I/O calls, msec/call = 0.43, calls/sec = 2393.29
> nuttcp-t: 0.0user 0.9sys 0:05real 19% 112i+1507d 618maxrss 0+2pf 97668+4csw
>
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.11
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 750.8897 MB in 5.15 real seconds = 149424.38 KB/sec = 1224.0845
> Mbps
> nuttcp-r: 213009 I/O calls, msec/call = 0.02, calls/sec = 41394.56
> nuttcp-r: 0.0user 4.6sys 0:05real 91% 107i+1445d 618maxrss 0+16pf
> 18273+0csw
>
> It seems performance measurements are more dependent on the server (nuttcp
> -S) machine.
> We will have to rule out the interrupt storms first of course, any advice?
>
>
> Daniel
>
> ______________________________**_________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/**mailman/listinfo/freebsd-**current<http://lists.freebsd.org/mailman/listinfo/freebsd-current>
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_**
> freebsd.org <freebsd-current-unsubscribe_at_freebsd.org>"
>