Re: vge traffic problem

On Mon, Jan 11, 2010 at 05:22:49PM -0800, David Ehrmann wrote:
> Pyun YongHyeon wrote:
> >>I ran the test, again, and had tcpdump capture the packets (on the host 
> >>with the vge interface).
> >>
> >>tcpdump -i vge0 -w dump.cap -K -s 0 host 10.0.1.2
> >>
> >>When I opened it up in Wireshark, it's reporting that the outgoing UDP 
> >>checksums are incorrect; they're always 0x1ae3.  That said, maybe the 
> >>    
> >
> >Because bpf(4) see the frame before controller inserts checksum
> >value it always reports incorrect checksum. It's normal for TX
> >checksum offload capable controller.
> >In order to verify the hardware assisted checksum you should check
> >that on receiver side. If the checksum was invalid, receiver
> >dropped them. See "netstat -s | grep bad" on receiver host.
> >
> >  
> I did this one between two similar FreeBSD systems (8.0 and 8.0 RC1, I 
> think).  There wasn't an error in netstat -s on the receiver, but I ran 
> tcpdump, anyway.  The checksums were good, but iperf ended with "read 
> failed: connection refused."  On the non-vge box, once the vge box is 
> done sending, it sends four identical UDP packets to the vge host, I 
> think to ask it for sending statistics.  Instead of statistics, it got 
> back an ICMP packet saying "port unreachable."  This happens with a 
> freebsd client, but not a linux client (I didn't confirm the ICMP 
> message with tcpdump, though).  All iperf versions are the same (2.0.4), 
> but the linux one doesn't have any threading (which shouldn't be an 
> issue when I'm not using that option).

It seems iperf on FreeBSD was broken. It incorrectly generates
huge-packet with IP length 0 so other host disconnected the
TCP connection. Not sure it could be related with threading though.
Use netperf instead, it would be more reliable than iperf.

> >>checksums are done in hardware AFTER tcpdump sees them.
> >>
> >>I set net.inet.udp.checksum to 0.  The bad checksums are gone, but I 
> >>still see dropped packets.
> >>
> >>    
> >
> >This doesn't disable TX checksum offload of driver. Instead use
> >#ifconfig vge0 -txcsum
> >
> >  
> I'm still losing packets, but now it's interesting.  The other FreeBSD 
> machine (a VMware VM on a fairly fast machine with another gigabit 
> connection) loses just a few, now, and sometimes none.  The linux box 
> (200mhz, runs OpenWrt, 100mbps ethernet) still loses more than 1% of 
> packets.  I cut iperf back to 100kbps.  The results were much better (0 
> packets lost), but this did happen one time:
> 

It's normal see some dropped frames under high network load. And
you can't compare gigabit controller to fast ethernet controller.

> [  3] WARNING: did not receive ack of last datagram after 10 tries.
> 
> Maybe this is related to "port unreachable," and maybe to my nfs 
> problem.  The drops seem to be related to iperf not being able to do 
> much processing at 200 mhz (which is a little surprising; I was only 
> asking for 1Mbps).
> >>It's on the motherboard, probably wired directly to a PCI-E connection, 
> >>but yes, it is a VT6130.
> >>    
> >
> >Show me "pciconf -lcv" output.
> >  
> vge0_at_pci0:3:0:0:        class=0x020000 card=0x01101106 chip=0x31191106 
> rev=0x82 hdr=0x00
>    vendor     = 'VIA Technologies Inc'
>    device     = ''Velocity' Gigabit Ethernet Controllers 
> (VT6120/VT6121/VT6122)'
>    class      = network
>    subclass   = ethernet
>    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
>    cap 10[90] = PCI-Express 1 endpoint max data 128(128) link x1(x1)
>    cap 05[c0] = MSI supports 1 message, 64 bit, vector masks
> 
> It might not be vge, after all, and might just be its speed...but that 
> shouldn't cause a tcp nfs issue.

I have exact the same revision of the hardware and I don't have
encountered your issue here. Instead of measuring performance
number with broken iperf, check whether you still get
"Connection reset by peer" message with csup(1) when you use vge(4)
interface. If you still see the message, please send me tcpdump
capture in private.