Re: igb broken? Unexplained weirdness with intel 82576 nics on a supermicro board.

On 05/08/2010 02:21 PM, Jack Vogel wrote:
> The cable, its a simple thing but make SURE you try that, a slightly
> damaged one can do weird things and its quick to check, don't overlook
> it.
>
> Jack
>
>
> On Sat, May 8, 2010 at 10:22 AM, joe <joe_at_hostedcontent.com
> <mailto:joe_at_hostedcontent.com>> wrote:
>
>     On 05/08/2010 01:53 PM, Jack Vogel wrote:
>
>         I still am not clear on this system, how many ports are on it,
>         and its
>         an 82576?
>         Sounds to me like you've proven its not on the box if you can do
>         fine
>         when its
>         on its own. So change ports in the switch, as I said, change
>         cables, must be
>         something in that environment.
>
>         Jack
>
>
>         On Sat, May 8, 2010 at 10:04 AM, joe <joe_at_hostedcontent.com
>         <mailto:joe_at_hostedcontent.com>
>         <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>>>
>         wrote:
>
>             On 05/08/2010 01:31 PM, Jack Vogel wrote:
>
>                 Looks like something to do with system C, you might
>         isolate it,
>                 and try
>                 a back
>                 to back connection with its NICs, change cables, look at
>         BIOS
>                 settings,
>                 change
>                 the slot the nic is in... All just off the top of my head.
>
>                 Jack
>
>
>                 On Sat, May 8, 2010 at 9:41 AM, joe
>         <joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>
>         <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>>
>         <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>
>         <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>>>>
>
>                 wrote:
>
>                     On 05/08/2010 11:17 AM, Ian FREISLICH wrote:
>
>                         joe wrote:
>
>                             On 05/08/2010 06:55 AM, Ian FREISLICH wrote:
>
>                                 joe wrote:
>
>                                            I have just tried your
>         suggeston and
>                 it has
>                                     no effect for me ;(
>
>
>                                 Do you have another brand of NIC that
>         you can
>                 try?  At
>                                 least that
>                                 will isolate whether it's igb(4) or
>         something else.
>
>
>                             I will grab a new nic today and try...my
>         options are
>                 limited
>                             though.
>                             Here are the nics i can get my hands on
>
>                             TP-LINK TL-TG3468, 10/100/1000Mbps PCIe Adapter
>                 (supported
>                             by fbsd?)
>
>
>                         Based on the RTL8168B chip.  Should be supported
>         by the
>                 re(4)
>                         driver.
>
>                             Intel (EXPI9301CT) Gigabit CT Desktop
>         Adapter (yet
>                 another
>                             intel nic)
>
>
>                         i82574L chip.  Should be supported by the em(4)
>         driver.
>                   I have had
>                         good performance in the past with this driver
>         and less than
>                         satisfactory performance with the igb(4) driver.
>
>                         That may not be your problem though.  Before you
>         go out
>                 and buy,
>                         have a look at the amount of interrupt time your
>         slow
>                 machine spends
>                         in 'top' or 'systat -vm'.  systat will also show the
>                 interrupt rate
>                         for each driver, perhaps it's not doing
>         interrupt moderation
>                         properly.
>                         This will manifest as more than about a 1000 per
>         second.
>                   There are
>                         loader tunables for the driver to increase the
>         number of
>                 transfer
>                         descriptors and to tune interrupt moderation.
>
>                         You could try running trafshow (port) on the
>         interface while
>                         performing the transfer.  Perhaps promiscuous
>         mode will
>                 turn off
>                         some hardware feature that will improve things.
>           It may
>                 however
>                         break hardware vlanning as it does on my 82575GB
>         4 port
>                 igb card.
>
>                         Ian
>
>                         --
>                         Ian Freislich
>
>
>                     I bought those two cards anyways, im in a rush to
>         figure out
>                 this
>                     problem. That being said i am still encountering the
>         exact same
>                     problem regardless on which network card i am
>         running. I am at a
>                     complete loss. I am about to try a raid card to see
>         if the
>                 problem
>                     might lay within the onboard sata ports. I did pull the
>                 server and
>                     brought it home so that i can test more things quicker.
>
>                     I am going to try using a raid card instead of the
>         onboard sata
>                     ports and see if i still encounter the same problem.
>         I would
>                 love
>                     any suggestions you may have on where to go from here to
>                 figure out
>                     where the problem might be.
>
>                     joe
>
>                     _______________________________________________
>         freebsd-current_at_freebsd.org <mailto:freebsd-current_at_freebsd.org>
>         <mailto:freebsd-current_at_freebsd.org
>         <mailto:freebsd-current_at_freebsd.org>>
>         <mailto:freebsd-current_at_freebsd.org
>         <mailto:freebsd-current_at_freebsd.org>
>         <mailto:freebsd-current_at_freebsd.org
>         <mailto:freebsd-current_at_freebsd.org>>>
>
>                     mailing list
>         http://lists.freebsd.org/mailman/listinfo/freebsd-current
>                     To unsubscribe, send any mail to
>         "freebsd-current-unsubscribe_at_freebsd.org
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org>
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org>>
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org>
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org
>         <mailto:freebsd-current-unsubscribe_at_freebsd.org>>>"
>
>
>
>             I think it might have something to so with the nics /
>         switch, and
>             their features. I brought the box home, plugged into my gb
>         switch,
>             and i am able to FTP data to the server at around 35MB/sec.
>
>             I dont know what would cause this other than some sort of
>         issue with
>             the the 3 different types of nics and the switch i am using.
>
>             Any suggestions?
>
>
>
>     There are two embedded intel 82576 nics on this motherboard. I do
>     believe i have proven it is not the box itself as it is capable of
>     high incoming throughput. I have other servers on the switch which
>     do 55MB/sec without issues. I believe it is a combination of this
>     server and/or the nics i have and the switch i am using. It's the
>     only logical explanation if i get the desired throughput on my home
>     switch but not on the switch that is collocated. I will try updating
>     the firmware of the switch tonight as well as bringing the switch i
>     use at home with me.
>
>

Thanks for the suggestion Jack! Replacing the cables and trying 
different ports on the switch was the first thing i did when first 
encountered this problem.  One thing i did notice was that the switch 
has jumbo frames disabled while the cards had it enabled. I tried 
disabling it on the cards but it didnt seem to have an impact. I am 
hoping a firmware upgrade to the switch will resolve the problem and if 
not then i'll just replace the switch at the DC with the switch i have 
at home...

Thank you (and everyone else) for all their help with this matter!

Regards,

Joe