Re: igb broken? Unexplained weirdness with intel 82576 nics on a supermicro board.

From: Jack Vogel <jfvogel_at_gmail.com>
Date: Sun, 9 May 2010 10:14:35 -0700
On Sun, May 9, 2010 at 2:54 AM, joe <joe_at_hostedcontent.com> wrote:

> On 05/08/2010 02:21 PM, Jack Vogel wrote:
>
>> The cable, its a simple thing but make SURE you try that, a slightly
>> damaged one can do weird things and its quick to check, don't overlook
>> it.
>>
>> Jack
>>
>>
>> On Sat, May 8, 2010 at 10:22 AM, joe <joe_at_hostedcontent.com
>> <mailto:joe_at_hostedcontent.com>> wrote:
>>
>>    On 05/08/2010 01:53 PM, Jack Vogel wrote:
>>
>>        I still am not clear on this system, how many ports are on it,
>>        and its
>>        an 82576?
>>        Sounds to me like you've proven its not on the box if you can do
>>        fine
>>        when its
>>        on its own. So change ports in the switch, as I said, change
>>        cables, must be
>>        something in that environment.
>>
>>        Jack
>>
>>
>>        On Sat, May 8, 2010 at 10:04 AM, joe <joe_at_hostedcontent.com
>>        <mailto:joe_at_hostedcontent.com>
>>        <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>>>
>>        wrote:
>>
>>            On 05/08/2010 01:31 PM, Jack Vogel wrote:
>>
>>                Looks like something to do with system C, you might
>>        isolate it,
>>                and try
>>                a back
>>                to back connection with its NICs, change cables, look at
>>        BIOS
>>                settings,
>>                change
>>                the slot the nic is in... All just off the top of my head.
>>
>>                Jack
>>
>>
>>                On Sat, May 8, 2010 at 9:41 AM, joe
>>        <joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>
>>        <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>>
>>        <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>
>>        <mailto:joe_at_hostedcontent.com <mailto:joe_at_hostedcontent.com>>>>
>>
>>                wrote:
>>
>>                    On 05/08/2010 11:17 AM, Ian FREISLICH wrote:
>>
>>                        joe wrote:
>>
>>                            On 05/08/2010 06:55 AM, Ian FREISLICH wrote:
>>
>>                                joe wrote:
>>
>>                                           I have just tried your
>>        suggeston and
>>                it has
>>                                    no effect for me ;(
>>
>>
>>                                Do you have another brand of NIC that
>>        you can
>>                try?  At
>>                                least that
>>                                will isolate whether it's igb(4) or
>>        something else.
>>
>>
>>                            I will grab a new nic today and try...my
>>        options are
>>                limited
>>                            though.
>>                            Here are the nics i can get my hands on
>>
>>                            TP-LINK TL-TG3468, 10/100/1000Mbps PCIe Adapter
>>                (supported
>>                            by fbsd?)
>>
>>
>>                        Based on the RTL8168B chip.  Should be supported
>>        by the
>>                re(4)
>>                        driver.
>>
>>                            Intel (EXPI9301CT) Gigabit CT Desktop
>>        Adapter (yet
>>                another
>>                            intel nic)
>>
>>
>>                        i82574L chip.  Should be supported by the em(4)
>>        driver.
>>                  I have had
>>                        good performance in the past with this driver
>>        and less than
>>                        satisfactory performance with the igb(4) driver.
>>
>>                        That may not be your problem though.  Before you
>>        go out
>>                and buy,
>>                        have a look at the amount of interrupt time your
>>        slow
>>                machine spends
>>                        in 'top' or 'systat -vm'.  systat will also show
>> the
>>                interrupt rate
>>                        for each driver, perhaps it's not doing
>>        interrupt moderation
>>                        properly.
>>                        This will manifest as more than about a 1000 per
>>        second.
>>                  There are
>>                        loader tunables for the driver to increase the
>>        number of
>>                transfer
>>                        descriptors and to tune interrupt moderation.
>>
>>                        You could try running trafshow (port) on the
>>        interface while
>>                        performing the transfer.  Perhaps promiscuous
>>        mode will
>>                turn off
>>                        some hardware feature that will improve things.
>>          It may
>>                however
>>                        break hardware vlanning as it does on my 82575GB
>>        4 port
>>                igb card.
>>
>>                        Ian
>>
>>                        --
>>                        Ian Freislich
>>
>>
>>                    I bought those two cards anyways, im in a rush to
>>        figure out
>>                this
>>                    problem. That being said i am still encountering the
>>        exact same
>>                    problem regardless on which network card i am
>>        running. I am at a
>>                    complete loss. I am about to try a raid card to see
>>        if the
>>                problem
>>                    might lay within the onboard sata ports. I did pull the
>>                server and
>>                    brought it home so that i can test more things quicker.
>>
>>                    I am going to try using a raid card instead of the
>>        onboard sata
>>                    ports and see if i still encounter the same problem.
>>        I would
>>                love
>>                    any suggestions you may have on where to go from here
>> to
>>                figure out
>>                    where the problem might be.
>>
>>                    joe
>>
>>                    _______________________________________________
>>        freebsd-current_at_freebsd.org <mailto:freebsd-current_at_freebsd.org>
>>        <mailto:freebsd-current_at_freebsd.org
>>        <mailto:freebsd-current_at_freebsd.org>>
>>        <mailto:freebsd-current_at_freebsd.org
>>        <mailto:freebsd-current_at_freebsd.org>
>>        <mailto:freebsd-current_at_freebsd.org
>>        <mailto:freebsd-current_at_freebsd.org>>>
>>
>>                    mailing list
>>        http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>                    To unsubscribe, send any mail to
>>        "freebsd-current-unsubscribe_at_freebsd.org
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org>
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org>>
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org>
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org
>>        <mailto:freebsd-current-unsubscribe_at_freebsd.org>>>"
>>
>>
>>
>>            I think it might have something to so with the nics /
>>        switch, and
>>            their features. I brought the box home, plugged into my gb
>>        switch,
>>            and i am able to FTP data to the server at around 35MB/sec.
>>
>>            I dont know what would cause this other than some sort of
>>        issue with
>>            the the 3 different types of nics and the switch i am using.
>>
>>            Any suggestions?
>>
>>
>>
>>    There are two embedded intel 82576 nics on this motherboard. I do
>>    believe i have proven it is not the box itself as it is capable of
>>    high incoming throughput. I have other servers on the switch which
>>    do 55MB/sec without issues. I believe it is a combination of this
>>    server and/or the nics i have and the switch i am using. It's the
>>    only logical explanation if i get the desired throughput on my home
>>    switch but not on the switch that is collocated. I will try updating
>>    the firmware of the switch tonight as well as bringing the switch i
>>    use at home with me.
>>
>>
>>
> Here is a follow-up just incase anyone ever encounters this problem again.
> I updated the firmware on the switch, made sure jumbo packets were enabled,
> and the switch was restarted. I'm now seeing the throughput i had expected
> on this box!
>
> 897755008 bytes received in 00:11 (72.74 MB/s)
>
> Once again, thank you all for the help!
>
> Joe
>

Glad the issue is resolved Joe.

Jack
Received on Sun May 09 2010 - 15:14:36 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:03 UTC