Re: datapoints on 10G throughput with TCP ?

Here is what I get, with an existing install, no tuning other than 
kern.ipc.nmbclusters=512000

Pair of Supermicro blades:

FreeBSD 8.2-STABLE #0: Wed Sep 28 11:23:59 EEST 2011
CPU: Intel(R) Xeon(R) CPU           E5620  _at_ 2.40GHz (2403.58-MHz 
K8-class CPU)
real memory  = 51539607552 (49152 MB)
[...]
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10> 
port 0xdc00-0xdc1f mem 0xfbc00000-0xfbdfffff,0xfbbfc000-0xfbbfffff irq 
16 at device 0.0 on pci3
ix0: Using MSIX interrupts with 9 vectors
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: Ethernet address: xx:xx:xx:xx:xx:xx
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10> 
port 0xd880-0xd89f mem 0xfb800000-0xfb9fffff,0xfbbf8000-0xfbbfbfff irq 
17 at device 0.1 on pci3
ix1: Using MSIX interrupts with 9 vectors
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: Ethernet address: xx:xx:xx:xx:xx:xx
ix1: PCI Express Bus: Speed 5.0Gb/s Width x8


blade 1:

# nuttcp -S
# nuttcp -t -T 5 -w 128 -v localhost
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.044 ms
nuttcp-t: send window size = 143360, receive window size = 71680
nuttcp-t: 8959.8750 MB in 5.02 real seconds = 1827635.67 KB/sec = 
14971.9914 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 143358 I/O calls, msec/call = 0.04, calls/sec = 28556.81
nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 602maxrss 0+5pf 16+46csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 127.0.0.1
nuttcp-r: send window size = 43008, receive window size = 143360
nuttcp-r: 8959.8750 MB in 5.17 real seconds = 1773171.07 KB/sec = 
14525.8174 Mbps
nuttcp-r: 219708 I/O calls, msec/call = 0.02, calls/sec = 42461.43
nuttcp-r: 0.0user 3.8sys 0:05real 76% 105i+1407d 614maxrss 1+17pf 
95059+22csw

blade 2:

# nuttcp -t -T 5 -w 128 -v 10.2.101.12
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.12
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.12 with mss=1448, RTT=0.059 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1340.6469 MB in 5.02 real seconds = 273449.90 KB/sec = 
2240.1016 Mbps
nuttcp-t: host-retrans = 171
nuttcp-t: 21451 I/O calls, msec/call = 0.24, calls/sec = 4272.78
nuttcp-t: 0.0user 1.9sys 0:05real 39% 120i+1610d 600maxrss 2+3pf 75658+0csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.11
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1340.6469 MB in 5.17 real seconds = 265292.92 KB/sec = 
2173.2796 Mbps
nuttcp-r: 408764 I/O calls, msec/call = 0.01, calls/sec = 78992.15
nuttcp-r: 0.0user 3.3sys 0:05real 64% 105i+1413d 620maxrss 0+15pf 
105104+102csw


Another pari of blades:

FreeBSD 8.2-STABLE #0: Tue Aug  9 12:37:55 EEST 2011
CPU: AMD Opteron(tm) Processor 6134 (2300.04-MHz K8-class CPU)
real memory  = 68719476736 (65536 MB)
[...]
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10> 
port 0xe400-0xe41f mem 0xfe600000-0xfe7fffff,0xfe4fc000-0xfe4fffff irq 
19 at device 0.0 on pci3
ix0: Using MSIX interrupts with 9 vectors
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: [ITHREAD]
ix0: Ethernet address: xx:xx:xx:xx:xx:xx
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.3.10> 
port 0xe800-0xe81f mem 0xfea00000-0xfebfffff,0xfe8fc000-0xfe8fffff irq 
16 at device 0.1 on pci3
ix1: Using MSIX interrupts with 9 vectors
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: [ITHREAD]
ix1: Ethernet address: xx:xx:xx:xx:xx:xx
ix1: PCI Express Bus: Speed 5.0Gb/s Width x8

first blade:

# nuttcp -S
# nuttcp -t -T 5 -w 128 -v localhost
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.090 ms
nuttcp-t: send window size = 143360, receive window size = 71680
nuttcp-t: 2695.0625 MB in 5.00 real seconds = 551756.90 KB/sec = 
4519.9925 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 43121 I/O calls, msec/call = 0.12, calls/sec = 8621.20
nuttcp-t: 0.0user 4.9sys 0:05real 99% 106i+1428d 620maxrss 0+4pf 2+71csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 127.0.0.1
nuttcp-r: send window size = 43008, receive window size = 143360
nuttcp-r: 2695.0625 MB in 5.14 real seconds = 536509.66 KB/sec = 
4395.0871 Mbps
nuttcp-r: 43126 I/O calls, msec/call = 0.12, calls/sec = 8383.94
nuttcp-r: 0.0user 3.1sys 0:05real 61% 94i+1264d 624maxrss 1+16pf 43019+0csw

second blade:

# nuttcp -t -T 5 -w 128 -v 10.2.101.13
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.164 ms
nuttcp-t: send window size = 131768, receive window size = 66608
nuttcp-t: 1290.3750 MB in 5.00 real seconds = 264173.96 KB/sec = 
2164.1131 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 20646 I/O calls, msec/call = 0.25, calls/sec = 4127.72
nuttcp-t: 0.0user 3.8sys 0:05real 77% 96i+1299d 616maxrss 0+3pf 27389+0csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.2.101.14
nuttcp-r: send window size = 33304, receive window size = 131768
nuttcp-r: 1290.3750 MB in 5.14 real seconds = 256835.92 KB/sec = 
2103.9998 Mbps
nuttcp-r: 85668 I/O calls, msec/call = 0.06, calls/sec = 16651.70
nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1437d 624maxrss 0+15pf 11848+0csw


Not impresive... I am rebuilding now to -stable.

Daniel