Re: datapoints on 10G throughput with TCP ?

On 06.12.11 13:18, Daniel Kalchev wrote:
> [...]
> second blade:
>
> # nuttcp -t -T 5 -w 128 -v 10.2.101.13
> nuttcp-t: v6.1.2: socket
> nuttcp-t: buflen=65536, nstream=1, port=5001 tcp -> 10.2.101.13
> nuttcp-t: time limit = 5.00 seconds
> nuttcp-t: connect to 10.2.101.13 with mss=1448, RTT=0.164 ms
> nuttcp-t: send window size = 131768, receive window size = 66608
> nuttcp-t: 1290.3750 MB in 5.00 real seconds = 264173.96 KB/sec = 
> 2164.1131 Mbps
> nuttcp-t: host-retrans = 0
> nuttcp-t: 20646 I/O calls, msec/call = 0.25, calls/sec = 4127.72
> nuttcp-t: 0.0user 3.8sys 0:05real 77% 96i+1299d 616maxrss 0+3pf 
> 27389+0csw
>
> nuttcp-r: v6.1.2: socket
> nuttcp-r: buflen=65536, nstream=1, port=5001 tcp
> nuttcp-r: accept from 10.2.101.14
> nuttcp-r: send window size = 33304, receive window size = 131768
> nuttcp-r: 1290.3750 MB in 5.14 real seconds = 256835.92 KB/sec = 
> 2103.9998 Mbps
> nuttcp-r: 85668 I/O calls, msec/call = 0.06, calls/sec = 16651.70
> nuttcp-r: 0.0user 4.8sys 0:05real 94% 107i+1437d 624maxrss 0+15pf 
> 11848+0csw
>
>
> Not impresive... I am rebuilding now to -stable.
>
> Daniel

I also noticed interrupt storms happening while this was running on the 
second pair of blades:

interrupt storm detected on "irq272:"; throttling interrupt source
interrupt storm detected on "irq272:"; throttling interrupt source
interrupt storm detected on "irq272:"; throttling interrupt source
interrupt storm detected on "irq270:"; throttling interrupt source
interrupt storm detected on "irq270:"; throttling interrupt source
interrupt storm detected on "irq270:"; throttling interrupt source

some stats

# sysctl -a dev.ix.1
dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 
2.3.10
dev.ix.1.%driver: ix
dev.ix.1.%location: slot=0 function=1
dev.ix.1.%pnpinfo: vendor=0x8086 device=0x10fc subvendor=0xffff 
subdevice=0xffff class=0x020000
dev.ix.1.%parent: pci3
dev.ix.1.flow_control: 3
dev.ix.1.advertise_gig: 0
dev.ix.1.enable_aim: 1
dev.ix.1.rx_processing_limit: 128
dev.ix.1.dropped: 0
dev.ix.1.mbuf_defrag_failed: 0
dev.ix.1.no_tx_dma_setup: 0
dev.ix.1.watchdog_events: 0
dev.ix.1.tso_tx: 1193460
dev.ix.1.link_irq: 1
dev.ix.1.queue0.interrupt_rate: 1000000
dev.ix.1.queue0.txd_head: 45
dev.ix.1.queue0.txd_tail: 45
dev.ix.1.queue0.no_desc_avail: 0
dev.ix.1.queue0.tx_packets: 23
dev.ix.1.queue0.rxd_head: 16
dev.ix.1.queue0.rxd_tail: 15
dev.ix.1.queue0.rx_packets: 16
dev.ix.1.queue0.rx_bytes: 2029
dev.ix.1.queue0.lro_queued: 0
dev.ix.1.queue0.lro_flushed: 0
dev.ix.1.queue1.interrupt_rate: 62500
dev.ix.1.queue1.txd_head: 0
dev.ix.1.queue1.txd_tail: 0
dev.ix.1.queue1.no_desc_avail: 0
dev.ix.1.queue1.tx_packets: 0
dev.ix.1.queue1.rxd_head: 0
dev.ix.1.queue1.rxd_tail: 2047
dev.ix.1.queue1.rx_packets: 0
dev.ix.1.queue1.rx_bytes: 0
dev.ix.1.queue1.lro_queued: 0
dev.ix.1.queue1.lro_flushed: 0
dev.ix.1.queue2.interrupt_rate: 200000
dev.ix.1.queue2.txd_head: 545
dev.ix.1.queue2.txd_tail: 545
dev.ix.1.queue2.no_desc_avail: 0
dev.ix.1.queue2.tx_packets: 331690
dev.ix.1.queue2.rxd_head: 1099
dev.ix.1.queue2.rxd_tail: 1098
dev.ix.1.queue2.rx_packets: 498763
dev.ix.1.queue2.rx_bytes: 32954702
dev.ix.1.queue2.lro_queued: 0
dev.ix.1.queue2.lro_flushed: 0
dev.ix.1.queue3.interrupt_rate: 62500
dev.ix.1.queue3.txd_head: 0
dev.ix.1.queue3.txd_tail: 0
dev.ix.1.queue3.no_desc_avail: 0
dev.ix.1.queue3.tx_packets: 0
dev.ix.1.queue3.rxd_head: 0
dev.ix.1.queue3.rxd_tail: 2047
dev.ix.1.queue3.rx_packets: 0
dev.ix.1.queue3.rx_bytes: 0
dev.ix.1.queue3.lro_queued: 0
dev.ix.1.queue3.lro_flushed: 0
dev.ix.1.queue4.interrupt_rate: 1000000
dev.ix.1.queue4.txd_head: 13
dev.ix.1.queue4.txd_tail: 13
dev.ix.1.queue4.no_desc_avail: 0
dev.ix.1.queue4.tx_packets: 6
dev.ix.1.queue4.rxd_head: 6
dev.ix.1.queue4.rxd_tail: 5
dev.ix.1.queue4.rx_packets: 6
dev.ix.1.queue4.rx_bytes: 899
dev.ix.1.queue4.lro_queued: 0
dev.ix.1.queue4.lro_flushed: 0
dev.ix.1.queue5.interrupt_rate: 200000
dev.ix.1.queue5.txd_head: 982
dev.ix.1.queue5.txd_tail: 982
dev.ix.1.queue5.no_desc_avail: 0
dev.ix.1.queue5.tx_packets: 302592
dev.ix.1.queue5.rxd_head: 956
dev.ix.1.queue5.rxd_tail: 955
dev.ix.1.queue5.rx_packets: 474044
dev.ix.1.queue5.rx_bytes: 31319840
dev.ix.1.queue5.lro_queued: 0
dev.ix.1.queue5.lro_flushed: 0
dev.ix.1.queue6.interrupt_rate: 200000
dev.ix.1.queue6.txd_head: 1902
dev.ix.1.queue6.txd_tail: 1902
dev.ix.1.queue6.no_desc_avail: 0
dev.ix.1.queue6.tx_packets: 184922
dev.ix.1.queue6.rxd_head: 1410
dev.ix.1.queue6.rxd_tail: 1409
dev.ix.1.queue6.rx_packets: 402818
dev.ix.1.queue6.rx_bytes: 27759640
dev.ix.1.queue6.lro_queued: 0
dev.ix.1.queue6.lro_flushed: 0
dev.ix.1.queue7.interrupt_rate: 200000
dev.ix.1.queue7.txd_head: 660
dev.ix.1.queue7.txd_tail: 660
dev.ix.1.queue7.no_desc_avail: 0
dev.ix.1.queue7.tx_packets: 378078
dev.ix.1.queue7.rxd_head: 885
dev.ix.1.queue7.rxd_tail: 884
dev.ix.1.queue7.rx_packets: 705397
dev.ix.1.queue7.rx_bytes: 46572290
dev.ix.1.queue7.lro_queued: 0
dev.ix.1.queue7.lro_flushed: 0
dev.ix.1.mac_stats.crc_errs: 0
dev.ix.1.mac_stats.ill_errs: 0
dev.ix.1.mac_stats.byte_errs: 0
dev.ix.1.mac_stats.short_discards: 0
dev.ix.1.mac_stats.local_faults: 3
dev.ix.1.mac_stats.remote_faults: 1
dev.ix.1.mac_stats.rec_len_errs: 0
dev.ix.1.mac_stats.link_xon_txd: 0
dev.ix.1.mac_stats.link_xon_rcvd: 0
dev.ix.1.mac_stats.link_xoff_txd: 0
dev.ix.1.mac_stats.link_xoff_rcvd: 0
dev.ix.1.mac_stats.total_octets_rcvd: 146983578
dev.ix.1.mac_stats.good_octets_rcvd: 146933576
dev.ix.1.mac_stats.total_pkts_rcvd: 2081451
dev.ix.1.mac_stats.good_pkts_rcvd: 2081044
dev.ix.1.mac_stats.mcast_pkts_rcvd: 0
dev.ix.1.mac_stats.bcast_pkts_rcvd: 2
dev.ix.1.mac_stats.rx_frames_64: 4
dev.ix.1.mac_stats.rx_frames_65_127: 2081036
dev.ix.1.mac_stats.rx_frames_128_255: 0
dev.ix.1.mac_stats.rx_frames_256_511: 0
dev.ix.1.mac_stats.rx_frames_512_1023: 4
dev.ix.1.mac_stats.rx_frames_1024_1522: 0
dev.ix.1.mac_stats.recv_undersized: 0
dev.ix.1.mac_stats.recv_fragmented: 0
dev.ix.1.mac_stats.recv_oversized: 0
dev.ix.1.mac_stats.recv_jabberd: 0
dev.ix.1.mac_stats.management_pkts_rcvd: 0
dev.ix.1.mac_stats.management_pkts_drpd: 0
dev.ix.1.mac_stats.checksum_errs: 0
dev.ix.1.mac_stats.good_octets_txd: 4752616248
dev.ix.1.mac_stats.total_pkts_txd: 3302683
dev.ix.1.mac_stats.good_pkts_txd: 3302683
dev.ix.1.mac_stats.bcast_pkts_txd: 3
dev.ix.1.mac_stats.mcast_pkts_txd: 0
dev.ix.1.mac_stats.management_pkts_txd: 0
dev.ix.1.mac_stats.tx_frames_64: 3
dev.ix.1.mac_stats.tx_frames_65_127: 9203
dev.ix.1.mac_stats.tx_frames_128_255: 31588
dev.ix.1.mac_stats.tx_frames_256_511: 50179
dev.ix.1.mac_stats.tx_frames_512_1023: 153285
dev.ix.1.mac_stats.tx_frames_1024_1522: 3058425
dev.ix.1.mac_stats.fc_crc: 0
dev.ix.1.mac_stats.fc_last: 0
dev.ix.1.mac_stats.fc_drpd: 0
dev.ix.1.mac_stats.fc_pkts_rcvd: 0
dev.ix.1.mac_stats.fc_pkts_txd: 0
dev.ix.1.mac_stats.fc_dword_rcvd: 0
dev.ix.1.mac_stats.fc_dword_txd: 0

Maybe this controller needs some tuning?

Daniel