Re: Help needed: TCP Wizards (was 8.0-RC1 NFS client timeout issue)

On 9 Nov 2009, at 23:04, Rick Macklem wrote:

>
>
> On Thu, 5 Nov 2009, Rui Paulo wrote:
>
>>> Now, here's where someone may be able to help?
>>> Grep'ng around, I found 4 places where the TCP stack called  
>>> ip_output()
>>> (one in each of tcp_output.c, tcp_subr.c, tcp_timewait.c and  
>>> tcp_syncache.c) and I put a printf like this just before them:
>>> 	if (flags & TH_RST)
>>> 		printf("sent a reset\n");
>>>
>>> 	(The exact format varies for each, because of where the TCP
>>>       header flags are and have different printf messages.)
>>> Now, the weird part is, that when the extraneous RST is sent to the
>>> server, I don't get any printf. (I do get a few of these, but at  
>>> other
>>> times for what appear to be legitimate RSTs.)
>>> I can't see anywhere else that the TCP stack would send an RST  
>>> and, so,
>>> I'm stuck w.r.t. figuring out what is sending them?
> Ok, if you found the previous posts entertaining, you might enjoy  
> this:-)
>
> Along with the printfs before all the ip_output() calls, I added calls
> inside ip_output() and, eventually, even calls in front of every  
> if_output(). Never got anything that indicated an RST was being sent.
> (I only saw what I expected, which was an ACK reply being sent.)
>
> BUT, at almost exactly the same time, there were the FreeBSD8-CURRENT
> client->server RST packets on the server's snoop trace.
>
> Hmm, did a tcpdump in the client and, yes, the same packets were  
> there.
>
> To keep it simple, I had done the dinosaur thing and plugged both the
> client and server into an old, dumb 10baseT hub, so that I could  
> easily
> watch everything. (I also had an uplink cable to the net port in the
> wall, so I could move kernels around from the machine I usually build
> them on.)
>
> I was at the point where I couldn't conceivably figure out where the
> FreeBSD-CURRENT client was generating these RSTs. So guess what...
> --> it wasn't
>
> I unplugged the uplink cable and, no more RSTs. I've been testing for
> long enough now, that I am 99% certain they were being injected. Since
> the from address and even the MAC address is correct, I can only  
> assume
> that it was the Cisco switch that was doing the injecting. (How else
> could a packet come in from the Cisco switch with the MAC address of
> the FreeBSD-CURRENT client machine?)
>
> It was usually triggered by a server reboot. After the server reboot,
> the server does send an RST to the client. This seems legit, but might
> be what makes Cisco think that "bad things" are happening? (I have no
> access to info about the switches or their configuration, although the
> campus standard is for these ports to be used by a single desktop  
> machine
> only and not a switch or hub.)
>
> So, it seems that FreeBSD8-CURRENT reconnects fine now, so long as
> nothing is injecting RSTs into the newly created connection.
>
> Well, I'm not sure I found this fun, but hopefully others are
> entertained:-), rick
>

We had some issues at work where the Cisco intrusion detection thing  
was resetting connections bogusly. Do you have IDS enabled on your  
Cisco gear?