datapoints on 10G throughput with TCP ?

Hi,
I am trying to establish the baseline performance for 10G throughput
over TCP, and would like to collect some data points.  As a testing
program i am using nuttcp from ports (as good as anything, i
guess -- it is reasonably flexible, and if you use it in
TCP with relatively large writes, the overhead for syscalls
and gettimeofday() shouldn't kill you).

I'd be very grateful if you could do the following test:

- have two machines connected by a 10G link
- on one run "nuttcp -S"
- on the other one run "nuttcp -t -T 5 -w 128 -v the.other.ip"

and send me a dump of the output, such as the one(s) at the end of
the message.

I am mostly interested in two configurations:
- one over loopback, which should tell how fast is the CPU+memory
  As an example, one of my machines does about 15 Gbit/s, and
  one of the faster ones does about 44 Gbit/s

- one over the wire using 1500 byte mss. Here it really matters
  how good is the handling of small MTUs.

As a data point, on my machines i get 2..3.5 Gbit/s on the
"slow" machine with a 1500 byte mtu and default card setting.
Clearing the interrupt mitigation register (so no mitigation)
brings the rate to 5-6 Gbit/s. Same hardware with linux does
about 8 Gbit/s. HEAD seems 10-20% slower than RELENG_8 though i
am not sure who is at fault.

The receive side is particularly critical - on FreeBSD
the receiver is woken up every two packets (do the math
below, between the number of rx calls and throughput and mss),
resulting in almost 200K activations per second, and despite
the fact that interrupt mitigation is set to a much lower
value (so incoming packets should be batched).
On linux, i see much fewer reads, presumably the process is
woken up only at the end of a burst.

	cheers
	luigi


------------ EXAMPLES OF OUTPUT ----------------------

> nuttcp -t -T 5 -w 128 -v  10.0.1.2
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.0.1.2
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 10.0.1.2 with mss=1460, RTT=0.103 ms
nuttcp-t: send window size = 131400, receive window size = 65700
nuttcp-t: 3095.0982 MB in 5.00 real seconds = 633785.85 KB/sec = 5191.9737 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 49522 I/O calls, msec/call = 0.10, calls/sec = 9902.99
nuttcp-t: 0.0user 2.7sys 0:05real 54% 100i+2639d 752maxrss 0+3pf 258876+6csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 10.0.1.4
nuttcp-r: send window size = 33580, receive window size = 131400
nuttcp-r: 3095.0982 MB in 5.17 real seconds = 613526.42 KB/sec = 5026.0084 Mbps
nuttcp-r: 1114794 I/O calls, msec/call = 0.00, calls/sec = 215801.03
nuttcp-r: 0.1user 3.5sys 0:05real 69% 112i+1104d 626maxrss 0+15pf 507653+188csw
>

> nuttcp -t -T 5 -w 128 -v localhost
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
nuttcp-t: time limit = 5.00 seconds
nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.051 ms
nuttcp-t: send window size = 143360, receive window size = 71680
nuttcp-t: 26963.4375 MB in 5.00 real seconds = 5521440.59 KB/sec = 45231.6413 Mbps
nuttcp-t: host-retrans = 0
nuttcp-t: 431415 I/O calls, msec/call = 0.01, calls/sec = 86272.51
nuttcp-t: 0.0user 4.6sys 0:05real 93% 102i+2681d 774maxrss 0+3pf 2510+1csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
nuttcp-r: accept from 127.0.0.1
nuttcp-r: send window size = 43008, receive window size = 143360
nuttcp-r: 26963.4375 MB in 5.20 real seconds = 5313135.74 KB/sec = 43525.2080 Mbps
nuttcp-r: 767807 I/O calls, msec/call = 0.01, calls/sec = 147750.09
nuttcp-r: 0.1user 3.9sys 0:05real 79% 98i+2570d 772maxrss 0+16pf 311014+8csw


on the server, run  "