Re: CSUP failure

On Mon, Nov 17, 2008 at 04:36:16PM -0500, jT wrote:
> On Mon, Nov 17, 2008 at 4:30 PM, Jeremy Chadwick <koitsu_at_freebsd.org> wrote:
> > On Mon, Nov 17, 2008 at 04:14:39PM -0500, jT wrote:
> >> All / Jeremy / Maxime,
> >>
> >> [root_at_bigmac ~]# csup -L 2 current-supfile
> >> Parsing supfile "current-supfile"
> >> Connecting to cvsup3.freebsd.org
> >> Connected to 128.31.0.28
> >> Server software version: SNAP_16_1h
> >> Invalid server reply to AUTHMD5
> >>
> >> [root_at_bigmac ~]# cvsup -g -L 2 -h cvsup2.us.freebsd.org current-supfile
> >> Parsing supfile "current-supfile"
> >> Connecting to cvsup2.us.freebsd.org
> >> Connected to cvsup2.us.freebsd.org
> >> Server software version: SNAP_16_1h
> >> Premature EOF from server
> >> Will retry at 16:14:29
> >>
> >> [root_at_bigmac ~]# cvsup -g -L 2 -h cvsup11.us.freebsd.org current-supfile
> >> Parsing supfile "current-supfile"
> >> Connecting to cvsup11.us.freebsd.org
> >> Connected to cvsup11.us.freebsd.org
> >> Server software version: SNAP_16_1h
> >> Premature EOF from server
> >> Will retry at 16:14:56
> >
> > Okay so the problem is not specific to the csup binary.  The errors
> > returned are different, but I'm willing to bet they're the result of the
> > same problem.
> >
> > Are you using a firewall (ipfw or pf) on this box?  Is there a firewall
> > upstream from you?  A NAT box that's severely broken?  This could cause
> > what you're seeing.
> 
> I am a CS undergrad at union college ITS is  notorious for doing silly
> things here from time to time, so anything is possible.

Okay, but you didn't answer my other question: are you using ipfw or pf
on your box?

> > truss -f -a -s 8192 -o truss.out csup -h someserver current-supfile
> >
> > Then please provide the contents of truss.out (they may be very long),
> > or put the file up on the web somewhere.  Do not include it as a MIME
> > attachment.  Do not copy/paste the contents either.
> 
> truss.out can be found here hope this is the format that you were looking for.
> 
> http://dpaste.com/91570/

And here we see the problem:

33849: socket(PF_INET,SOCK_STREAM,6)		 = 4 (0x4)
33849: connect(4,{ AF_INET 130.94.149.166:5999 },16) = 0 (0x0)
33849: fstat(1,{ mode=crw--w---- ,inode=111,size=0,blksize=4096 }) = 0 (0x0)
33849: ioctl(1,TIOCGETA,0xbfbfdc60)		 = 0 (0x0)
33849: write(1,"Connected to 130.94.149.166\n",28) = 28 (0x1c)
33849: read(4,"OK 17 0 SNAP_16_1h CVSup server ready\n",1023) = 38 (0x26)
33849: write(4,"PROTO 17 0 CSUP_1_0\n",20)	 = 20 (0x14)
33849: read(4,"PROTO 17 0\n",1023)		 = 11 (0xb)
33849: __sysctl(0xbfbfe8bc,0x2,0xbfbfe998,0xbfbfe8d4,0x0,0x0) = 0 (0x0)
33849: getlogin(0x28318c50,0x11,0xbfbfe8c8,0x283149e0,0xbfbfe998,0x805df20) = 0 (0x0)
33849: write(4,"USER bbs bigmac.union.edu\n",26) = 26 (0x1a)
33849: read(4,0x8115400,1023)			 ERR#54 'Connection reset by peer'
33849: write(2,"Invalid server reply to AUTHMD5\n",32) = 32 (0x20)

File descriptor 4 is the TCP socket connected to 130.94.149.166:5999.

We can see that after the write() is performed (with the string "USER
bbs bigmac.union.edu"), the following read() receives errno 54, which
is connection reset by peer.  The csup code is expecting to get back
data from the remote end, but its being severed in-between.

The USER string appears to be "USER <username> <machinename>", where
<username> is your actual username (even if you'd used sudo, it still
figures it out).

Here's what mine does:

40933: getlogin(0x60cf8f80,0x11,0x7fffffffe8d0,0x60bcc0bc,0x0,0x7fffffffe888) = 0 (0x0)
40933: write(4,"USER jdc icarus.home.lan\n",25)  = 25 (0x19)
40933: read(4,"AUTHMD5 . .\n",1023)              = 12 (0xc)
40933: write(4,"AUTHMD5 . . .\n",14)             = 14 (0xe)
40933: read(4,"OK .\n",1023)                     = 5 (0x5)
40933: write(4,"ATTR 6\n0\nfe7\n1e1\nff1\nff1\n9\n.\n",29) = 29 (0x1d)

So as you can see, something between you and the cvsup servers is
severing the connection before AUTHMD5 string can be sent to you.

I would love to blame your college for this (chances are that's what it
is; administrative ACLs that are intentionally injecting TCP RST back to
you as a form of network control -- it's very common, especially at
colleges where students are continually doing things they should not be
doing), but I don't have "hard evidence" that points you to them.

If you're not using ipfw or pf on your box, then it's likely your
college which is doing this.  The only solution is to talk to your
network administrators to get a hole punched to allow TCP port 5999,
which they very likely will not do (I worked at Oregon State University
for many years, CS/IT often resembles a dictatorship).  You could use
SSH port tunnelling (assuming you have a shell account somewhere on the
Internet) and siphon your csup through that.

Either way, I'd start by talking to your college network admins.  You
can point them to this thread.  They'll at least be able to confirm if
what you're seeing is the result of some TCP RST injection hardware on
their network.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |