Re: Call for stge(4) testers

On Tue, Jul 11, 2006 at 05:15:32AM +0000, Bill Paul wrote:
 > > On Sat, Jul 08, 2006 at 09:24:23PM -0700, Matthew Jacob wrote:
 > >  > Jason THorpe did one for NetBSD.
 > >  > 
 > >  > I did one for Solaris. Nice chip. Too bad it wasn't more popular.
 > >  > 
 > > 
 > > I'm satisfied with the performance and easy interface for the chip.
 > > But the lack of Tx interrupt moderation and extra accesses to a
 > > status register to check which kind of Tx errors were occurred are
 > > serious fault. Since the the error condition uses the same status
 > > bit in interrupt status register it's hard to distingush failures
 > > from sucess without extra register accesses.
 > > 
 > > BTW, I still have no clue how jumbo frame work on this chip. The
 > > documentation from wpaul didn't say any special things regarding
 > > jumbo frame. For instance the doc. said it supports jumbo frame
 > > up to 9022 but it seems the NIC can work with larger size.
 > > Even windows driver has a larger size than this, so I'm curious
 > > important part for jumbo frame were not released?
 > 
 > You have failed to specify in what way jumbo frame support doesn't work.
 > In your previous mail, you only said there was a "jumbo frame problem"
 > but did not elaborate further. What exactly _is_ the problem? Does it
 > not transmit jumbo frames correctly? Does it not receive them correctly?
 > Does C'thulu emerge from inside the machine and devour your soul?
 > (If so, this is a sure sign you left the 'devour soul' bit in the
 > control register turned on. I *hate* it when that happens.)
 > 

Actually jumbo frame support works but it takes a very long time
(30~50sec) to work correctly after selecting an MTU larger than
1500. During the time I could see the the NIC sends the jumbo frame
with tcpdump on sender side but the frame doesn't arrive at
destination. Sometimes the NIC generated 'Tx underrun' and sometimes
it didn't generate any errors. However in the long run it did work
after about a minuate without any action from user. I even changed
STGE_TxStartThresh to use a full frame size but it seemed that it had
no effects. Anyway, STGE_TxStartThresh is not used at all if stge(4)
enabled checksum offload capability.

 > The documentation refers to a 'MaxFrameSize' register (offset 0x86,
 > 16 bits access) which controls the point at which the receiver will
 > consider an inbound frame to be a giant. The default is 0x5ea (1514
 > bytes). I think you'll need to increase this to 9014 to receive 9K
 > jumbo frames (or 9018 for VLAN-tagged jumbo frames). It looks like
 > the register maxes out at 0x3FFF (16K), which is probably why the
 > Windows driver claimes to support frames larger than 9K. Note that
 > for this to work, you'll have to set up the RX descriptors with
 > enough buffer space to hold 9K of packet data. I haven't looked at
 > the driver code, but you should be able to set up the fragment array
 > in each RX descriptor to hold 5 mbufs with 2K clusters. You'll be
 > wasting a bit of space, but not necessarily as much as if you were
 > to allocate a single 9K buffer per descriptor.
 > 

I'm afraid the hardware doesn't support a single 9K buffer per
decriptor. It seems that the hardware can support up to 4k buffer
per descriptor so I chained received frames to a single jumbo
frame.

 > I don't see anything that limits TX frame size, though you might want
 > to look around and see if there's a TX threshold setting and try to
 > increase it if you see underruns.
 > 

Linux driver mentioned that Tx size for a single frame should be
less than 11008. I can't find that limitation from the chip
documentation though.

 > You need to establish a proper test setup to diagnose the problem. Plug
 > the NIC directly into another known good gigE card that does jumbo frames,
 > use ifconfig to set the MTU on both ports to 9000, monitor both sides of
 > the link with tcpdump/ethereal/whatever, then do "ping -s 8192 <dest IP>"
 > from both ends and see what happens.
 > 

That's exactly what I did it on my tests. From your detailed
explanation on jumbo frame internals and guide I've reread source
code related to jumbo frame and managed to find a bug in my code.
It was my programming error in selecting an MTU from ioctl handler.
However I don't understand how the hardware can send/receive larger
MTUs than 9022. If you program the hardware to receive 10K it
eventually work in a one or two minute.

Thanks for the hint and help.
-- 
Regards,
Pyun YongHyeon