Re: igb broken? Unexplained weirdness with intel 82576 nics on a supermicro board.

From: joe <joe_at_hostedcontent.com>
Date: Sat, 08 May 2010 05:28:07 -0400
On 05/08/2010 05:08 AM, joe wrote:
> I have 3 boxes, each with two nics. One nic for the private network 
> and one for the public network.
> The private network is all on the same vlan. All 6 nics are on the 
> same switch. All connections are 1000tx Full Duplex.
>
> I will call the servers Box A, Box B, and Box C.
>
> When i FTP data between Box A & B i get abou 25MB/sec.
> When i FTP data from Box C to Box A or B, i get about 20MB/sec.
> When i FTP data from Box A to C i get 10MB/sec
> When i FTP data from Box B to C i get 200KB/sec...
>
> Can anyone suggest why i might only be getting 200KB when transfering 
> data from Box B to C but not when transferring data from Box A to C?
>
> I tried installing ubuntu to see if the problem was still there and it 
> is not. I can FTP data from Box A or B to box C at 10MB/sec, still not 
> the gigabit speeds i should be seeing but not the 200KB/sec  i am 
> getting with freebsd.
>
> I logged into my switch and it reports that the ports are 1000tx full 
> duplex, the same as what freebsd is reporting. The switch & freebsd 
> also report no errors.
>
> I really dont know what to do. Nothing anywhere reports showing a 
> problem but obviously there is a problem somewhere. I've tried drivers 
> 1.9.5, 1.8.4, and 1.7.3. If i use 1.8.4 or 1.7.3 i get 50KB/sec 
> transferring data from box B whereas if i use 1.9.5 i get 200-300KB 
> sec. I should be getting like 50MB/sec ;(
>
> Any help would be grateful. Or, maybe someone could put me in touch 
> with the person responsible for the intel nic drivers and i can work 
> with them on resolving this issue?
>
> Thanks in advance.
>
>
> Name    Mtu Network       Address              Ipkts Ierrs Idrop    
> Opkts Oerrs  Coll
> igb0   1500 <Link#1>      00:30:48:9f:11:04    25853     0     0    
> 18340     0     0
> igb0   1500 216.105.91.14 216.105.91.145       19040     -     -    
> 18343     -     -
> igb1*  1500 <Link#2>      00:30:48:9f:11:05        0     0     
> 0        0     0     0
> lo0   16384 <Link#3>                              72     0     0       
> 72     0     0
> lo0   16384 127.0.0.0/8   127.0.0.1               72     -     -       
> 72     -     -
>
> Sysctl info for igb0/1
> dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.8.4
> dev.igb.0.%driver: igb
> dev.igb.0.%location: slot=0 function=0
> dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10c9 subvendor=0x15d9 
> subdevice=0x0100 class=0x020000
> dev.igb.0.%parent: pci1
> dev.igb.0.debug: -1
> dev.igb.0.stats: -1
> dev.igb.0.flow_control: 3
> dev.igb.0.enable_aim: 1
> dev.igb.0.low_latency: 128
> dev.igb.0.ave_latency: 450
> dev.igb.0.bulk_latency: 1200
> dev.igb.0.rx_processing_limit: 100
>
> Debug info for igb0
> May  8 09:17:57 debbie kernel: igb0: TX(1) Packets sent = 1295
> May  8 09:17:57 debbie kernel: igb0: Queue(2) tdh = 106, tdt = 106
> May  8 09:17:57 debbie kernel: igb0: TX(2) no descriptors avail event = 0
> May  8 09:17:57 debbie kernel: igb0: TX(2) MSIX IRQ Handled = 5353
> May  8 09:17:57 debbie kernel: igb0: TX(2) Packets sent = 5450
> May  8 09:17:57 debbie kernel: igb0: Queue(3) tdh = 1687, tdt = 1687
> May  8 09:17:57 debbie kernel: igb0: TX(3) no descriptors avail event = 0
> May  8 09:17:57 debbie kernel: igb0: TX(3) MSIX IRQ Handled = 1335
> May  8 09:17:57 debbie kernel: igb0: TX(3) Packets sent = 1354
> May  8 09:17:57 debbie kernel: igb0: Queue(0) rdh = 425, rdt = 424
> May  8 09:17:57 debbie kernel: igb0: RX(0) Packets received = 16809
> May  8 09:17:57 debbie kernel: igb0: RX(0) Split Packets = 12262
> May  8 09:17:57 debbie kernel: igb0: RX(0) Byte count = 25894129
> May  8 09:17:57 debbie kernel: igb0: RX(0) MSIX IRQ Handled = 43428
> May  8 09:17:57 debbie kernel: igb0: RX(0) LRO Queued= 13601
> May  8 09:17:57 debbie kernel: igb0: RX(0) LRO Flushed= 9966
> May  8 09:17:57 debbie kernel: igb0: Queue(1) rdh = 1700, rdt = 1699
> May  8 09:17:57 debbie kernel: igb0: RX(1) Packets received = 1700
> May  8 09:17:57 debbie kernel: igb0: RX(1) Split Packets = 937
> May  8 09:17:57 debbie kernel: igb0: RX(1) Byte count = 1392691
> May  8 09:17:57 debbie kernel: igb0: RX(1) MSIX IRQ Handled = 31888
> May  8 09:17:57 debbie kernel: igb0: RX(1) LRO Queued= 1362
> May  8 09:17:57 debbie kernel: igb0: RX(1) LRO Flushed= 1351
> May  8 09:17:57 debbie kernel: igb0: Queue(2) rdh = 33, rdt = 32
> May  8 09:17:57 debbie kernel: igb0: RX(2) Packets received = 6177
> May  8 09:17:57 debbie kernel: igb0: RX(2) Split Packets = 1442
> May  8 09:17:57 debbie kernel: igb0: RX(2) Byte count = 2518498
> May  8 09:17:57 debbie kernel: igb0: RX(2) MSIX IRQ Handled = 36258
> May  8 09:17:57 debbie kernel: igb0: RX(2) LRO Queued= 5794
> May  8 09:17:57 debbie kernel: igb0: RX(2) LRO Flushed= 5673
> May  8 09:17:57 debbie kernel: igb0: Queue(3) rdh = 1418, rdt = 1417
> May  8 09:17:57 debbie kernel: igb0: RX(3) Packets received = 1418
> May  8 09:17:57 debbie kernel: igb0: RX(3) Split Packets = 939
> May  8 09:17:57 debbie kernel: igb0: RX(3) Byte count = 1700710
> May  8 09:17:57 debbie kernel: igb0: RX(3) MSIX IRQ Handled = 31399
> May  8 09:17:57 debbie kernel: igb0: RX(3) LRO Queued= 1102
> May  8 09:17:57 debbie kernel: igb0: RX(3) LRO Flushed= 882
> May  8 09:17:57 debbie kernel: igb0: LINK MSIX IRQ Handled = 2
> May  8 09:17:57 debbie kernel: igb0: Mbuf defrag failed = 0
> May  8 09:17:57 debbie kernel: igb0: Std mbuf header failed = 0
> May  8 09:17:57 debbie kernel: igb0: Std mbuf packet failed = 0
> May  8 09:17:57 debbie kernel: igb0: Driver dropped packets = 0
> May  8 09:17:57 debbie kernel: igb0: Driver tx dma failure in xmit = 0
>
>
> sysctl status info:
> May  8 09:19:08 debbie kernel: igb0: Excessive collisions = 0
> May  8 09:19:08 debbie kernel: igb0: Sequence errors = 0
> May  8 09:19:08 debbie kernel: igb0: Defer count = 0
> May  8 09:19:08 debbie kernel: igb0: Missed Packets = 0
> May  8 09:19:08 debbie kernel: igb0: Receive No Buffers = 0
> May  8 09:19:08 debbie kernel: igb0: Receive Length Errors = 0
> May  8 09:19:08 debbie kernel: igb0: Receive errors = 0
> May  8 09:19:08 debbie kernel: igb0: Crc errors = 0
> May  8 09:19:08 debbie kernel: igb0: Alignment errors = 0
> May  8 09:19:08 debbie kernel: igb0: Collision/Carrier extension 
> errors = 0
> May  8 09:19:08 debbie kernel: igb0: RX overruns = 0
> May  8 09:19:08 debbie kernel: igb0: watchdog timeouts = 0
> May  8 09:19:08 debbie kernel: igb0: XON Rcvd = 17
> May  8 09:19:08 debbie kernel: igb0: XON Xmtd = 0
> May  8 09:19:08 debbie kernel: igb0: XOFF Rcvd = 17
> May  8 09:19:08 debbie kernel: igb0: XOFF Xmtd = 0
> May  8 09:19:08 debbie kernel: igb0: Good Packets Rcvd = 26159
> May  8 09:19:08 debbie kernel: igb0: Good Packets Xmtd = 21564
> May  8 09:19:08 debbie kernel: igb0: TSO Contexts Xmtd = 1687
> May  8 09:19:08 debbie kernel: igb0: TSO Contexts Failed = 0
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to 
> "freebsd-current-unsubscribe_at_freebsd.org"

Follow UP:
FTP to Server A from Server C: 243187792 bytes sent in 00:04 (53.59 MB/s)
FTP to Server C from Server A: 243187792 bytes received in 00:21 (10.72 
MB/s)
FTP to Server B from Server C: 243187792 bytes sent in 00:06 (36.26 MB/s)
FTP to Server C from Server B: 705176 bytes received in 00:17 (38.91 
KB/s) (i aborted)


HDD Info for Server C:

debbie# dd if=/dev/ad6 of=/dev/null bs=1m count=300
300+0 records in
300+0 records out
314572800 bytes transferred in 5.163595 secs (60921277 bytes/sec)

/dev/ad6
         512             # sectorsize
         300069052416    # mediasize in bytes (279G)
         586072368       # mediasize in sectors
         0               # stripesize
         0               # stripeoffset
         581421          # Cylinders according to firmware.
         16              # Heads according to firmware.
         63              # Sectors according to firmware.
         5NF1MQTR        # Disk ident.
Received on Sat May 08 2010 - 08:02:01 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:03 UTC