Re: Help needed: TCP Wizards (was 8.0-RC1 NFS client timeout issue)

On Thu, 5 Nov 2009, Rui Paulo wrote:

>> 
>> Now, here's where someone may be able to help?
>> 
>> Grep'ng around, I found 4 places where the TCP stack called ip_output()
>> (one in each of tcp_output.c, tcp_subr.c, tcp_timewait.c and 
>> tcp_syncache.c) and I put a printf like this just before them:
>> 	if (flags & TH_RST)
>> 		printf("sent a reset\n");
>>
>> 	(The exact format varies for each, because of where the TCP
>>        header flags are and have different printf messages.)
>> 
>> Now, the weird part is, that when the extraneous RST is sent to the
>> server, I don't get any printf. (I do get a few of these, but at other
>> times for what appear to be legitimate RSTs.)
>> 
>> I can't see anywhere else that the TCP stack would send an RST and, so,
>> I'm stuck w.r.t. figuring out what is sending them?
>> 
Ok, if you found the previous posts entertaining, you might enjoy this:-)

Along with the printfs before all the ip_output() calls, I added calls
inside ip_output() and, eventually, even calls in front of every 
if_output(). Never got anything that indicated an RST was being sent.
(I only saw what I expected, which was an ACK reply being sent.)

BUT, at almost exactly the same time, there were the FreeBSD8-CURRENT
client->server RST packets on the server's snoop trace.

Hmm, did a tcpdump in the client and, yes, the same packets were there.

To keep it simple, I had done the dinosaur thing and plugged both the
client and server into an old, dumb 10baseT hub, so that I could easily
watch everything. (I also had an uplink cable to the net port in the
wall, so I could move kernels around from the machine I usually build
them on.)

I was at the point where I couldn't conceivably figure out where the
FreeBSD-CURRENT client was generating these RSTs. So guess what...
--> it wasn't

I unplugged the uplink cable and, no more RSTs. I've been testing for
long enough now, that I am 99% certain they were being injected. Since
the from address and even the MAC address is correct, I can only assume
that it was the Cisco switch that was doing the injecting. (How else
could a packet come in from the Cisco switch with the MAC address of
the FreeBSD-CURRENT client machine?)

It was usually triggered by a server reboot. After the server reboot,
the server does send an RST to the client. This seems legit, but might
be what makes Cisco think that "bad things" are happening? (I have no
access to info about the switches or their configuration, although the
campus standard is for these ports to be used by a single desktop machine
only and not a switch or hub.)

So, it seems that FreeBSD8-CURRENT reconnects fine now, so long as
nothing is injecting RSTs into the newly created connection.

Well, I'm not sure I found this fun, but hopefully others are
entertained:-), rick