Re: datapoints on 10G throughput with TCP ?

On Mon, Dec 05, 2011 at 08:27:03PM +0100, Luigi Rizzo wrote:

> Hi,
> I am trying to establish the baseline performance for 10G throughput
> over TCP, and would like to collect some data points.  As a testing
> program i am using nuttcp from ports (as good as anything, i
> guess -- it is reasonably flexible, and if you use it in
> TCP with relatively large writes, the overhead for syscalls
> and gettimeofday() shouldn't kill you).
> 
> I'd be very grateful if you could do the following test:
> 
> - have two machines connected by a 10G link
> - on one run "nuttcp -S"
> - on the other one run "nuttcp -t -T 5 -w 128 -v the.other.ip"
> 
> and send me a dump of the output, such as the one(s) at the end of
> the message.
> 
> I am mostly interested in two configurations:
> - one over loopback, which should tell how fast is the CPU+memory
>   As an example, one of my machines does about 15 Gbit/s, and
>   one of the faster ones does about 44 Gbit/s
> 
> - one over the wire using 1500 byte mss. Here it really matters
>   how good is the handling of small MTUs.
> 
> As a data point, on my machines i get 2..3.5 Gbit/s on the
> "slow" machine with a 1500 byte mtu and default card setting.
> Clearing the interrupt mitigation register (so no mitigation)
> brings the rate to 5-6 Gbit/s. Same hardware with linux does
> about 8 Gbit/s. HEAD seems 10-20% slower than RELENG_8 though i
> am not sure who is at fault.
> 
> The receive side is particularly critical - on FreeBSD
> the receiver is woken up every two packets (do the math
> below, between the number of rx calls and throughput and mss),
> resulting in almost 200K activations per second, and despite
> the fact that interrupt mitigation is set to a much lower
> value (so incoming packets should be batched).
> On linux, i see much fewer reads, presumably the process is
> woken up only at the end of a burst.

About relative performance FreeBSD and Linux I wrote in -performance_at_
at Jan'11 (Interrupt performance)

> 
> ------------ EXAMPLES OF OUTPUT ----------------------
> 
> > nuttcp -t -T 5 -w 128 -v  10.0.1.2
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.0.1.2
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.0.1.2 with mss=1460, RTT=0.103 ms
> nuttcp-t: send window size = 131400, receive window size = 65700
> nuttcp-t: 3095.0982 MB in 5.00 real seconds = 633785.85 KB/sec = 5191.9737 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 49522 I/O calls, msec/call = 0.10, calls/sec = 9902.99
> nuttcp-t: 0.0user 2.7sys 0:05real 54% 100i+2639d 752maxrss 0+3pf 258876+6csw
> 
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.0.1.4
> nuttcp-r: send window size = 33580, receive window size = 131400
> nuttcp-r: 3095.0982 MB in 5.17 real seconds = 613526.42 KB/sec = 5026.0084 Mbps
> nuttcp-r: 1114794 I/O calls, msec/call = 0.00, calls/sec = 215801.03
> nuttcp-r: 0.1user 3.5sys 0:05real 69% 112i+1104d 626maxrss 0+15pf 507653+188csw
> >
> 
> > nuttcp -t -T 5 -w 128 -v localhost
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> localhost
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 127.0.0.1 with mss=14336, RTT=0.051 ms
> nuttcp-t: send window size = 143360, receive window size = 71680
> nuttcp-t: 26963.4375 MB in 5.00 real seconds = 5521440.59 KB/sec = 45231.6413 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 431415 I/O calls, msec/call = 0.01, calls/sec = 86272.51
> nuttcp-t: 0.0user 4.6sys 0:05real 93% 102i+2681d 774maxrss 0+3pf 2510+1csw
> 
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 127.0.0.1
> nuttcp-r: send window size = 43008, receive window size = 143360
> nuttcp-r: 26963.4375 MB in 5.20 real seconds = 5313135.74 KB/sec = 43525.2080 Mbps
> nuttcp-r: 767807 I/O calls, msec/call = 0.01, calls/sec = 147750.09
> nuttcp-r: 0.1user 3.9sys 0:05real 79% 98i+2570d 772maxrss 0+16pf 311014+8csw
> 
> 
> on the server, run  "
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"