Re: diskless client can not be loaded (works at r224471)

Boris Samorodov wrote:
> Hi!
> 
> I use the same kernel for both host and diskless stations.
> The FreeBSD-9 r224471 kernel at the host system works fine,
> but the r224870 kernel oes not work. The file /pxeboot seems to be
> loaded but then the host system does not answer to client's nfs
> questions.
> 
> Here are the relevan parts of tcpdump logs at the host system (kaka is
> the diskless station and 192.168.111.254 is the host system):
> ----- r224471 kernel (works) -----
> kaka.4 > 192.168.111.254.nfs: 100 lookup fh 1237,78874/9609216 "boot"
> 16:08:28.325938 IP (tos 0x0, ttl 64, id 2002, offset 0, flags [none],
> proto UDP (17), length 264)
> 192.168.111.254.nfs > kaka.4: reply ok 236 lookup fh
> 1237,78874/9609217 DIR 755 ids 0/0 sz 1024 nlink 8 rdev 192/38338761
> fsid 7e fileid 92a001 a/m/ctime 1313410061.000000 1313410058.000000
> 1313410058.000000 post dattr: DIR 755 ids 0/0 sz 512 nlink 19 rdev
> 192/38338760 fsid 7e fileid 92a000 a/m/ctime 1313410040.000000
> 1313393850.000000 1313393850.000000
> 16:08:28.326130 IP (tos 0x0, ttl 20, id 197, offset 0, flags [none],
> proto UDP (17), length 140)
> kaka.5 > 192.168.111.254.nfs: 112 lookup fh 1237,78874/9609217
> "boot.4th.split"
> 16:08:28.326180 IP (tos 0x0, ttl 64, id 2003, offset 0, flags [none],
> proto UDP (17), length 144)
> 192.168.111.254.nfs > kaka.5: reply ok 116 lookup ERROR: No such file
> or directory post dattr: DIR 755 ids 0/0 sz 1024 nlink 8 rdev
> 192/38338761 fsid 7e fileid 92a001 a/m/ctime 1313410061.000000
> 1313410058.000000 1313410058.000000
> 16:08:28.326420 IP (tos 0x0, ttl 20, id 198, offset 0, flags [none],
> proto UDP (17), length 104)
> kaka.exp2 > 192.168.111.254.entrust-aams: [udp sum ok] UDP, length 76
> 16:08:28.326555 IP (tos 0x0, ttl 64, id 2004, offset 0, flags [none],
> proto UDP (17), length 96)
> 192.168.111.254.entrust-aams > kaka.exp2: [bad udp cksum 55c4!] UDP,
> length 68
> 16:08:28.326717 IP (tos 0x0, ttl 20, id 199, offset 0, flags [none],
> proto UDP (17), length 128)
> kaka.7 > 192.168.111.254.nfs: 100 lookup fh 1237,78874/9609216 "boot"
> 16:08:28.326770 IP (tos 0x0, ttl 64, id 2005, offset 0, flags [none],
> proto UDP (17), length 264)
> 192.168.111.254.nfs > kaka.7: reply ok 236 lookup fh
> 1237,78874/9609217 DIR 755 ids 0/0 sz 1024 nlink 8 rdev 192/38338761
> fsid 7e fileid 92a001 a/m/ctime 1313410061.000000 1313410058.000000
> 1313410058.000000 post dattr: DIR 755 ids 0/0 sz 512 nlink 19 rdev
> 192/38338760 fsid 7e fileid 92a000 a/m/ctime 1313410040.000000
> 1313393850.000000 1313393850.000000
> 16:08:28.326965 IP (tos 0x0, ttl 20, id 200, offset 0, flags [none],
> proto UDP (17), length 144)
> kaka.8 > 192.168.111.254.nfs: 116 lookup fh 1237,78874/9609217
> "boot.4th.gz.split"
> ----- r224870 kernel (no answer) -----
> 15:21:45.825058 IP (tos 0x0, ttl 20, id 196, offset 0, flags [none],
> proto UDP (17), length 128)
> kaka.4 > 192.168.111.254.nfs: 100 lookup fh 1237,78874/9609216 "boot"
> 15:21:47.281800 IP (tos 0x0, ttl 20, id 197, offset 0, flags [none],
> proto UDP (17), length 128)
> kaka.4 > 192.168.111.254.nfs: 100 lookup fh 1237,78874/9609216 "boot"
> 15:21:51.282086 IP (tos 0x0, ttl 20, id 198, offset 0, flags [none],
> proto UDP (17), length 128)
> kaka.4 > 192.168.111.254.nfs: 100 lookup fh 1237,78874/9609216 "boot"
> -----
> 
> Some additional info about the host:
> -----
> % uname -a
> FreeBSD h30.sp.ipt.ru 9.0-BETA1 FreeBSD 9.0-BETA1 #29 r224870: Mon Aug
> 15 03:12:15 MSK 2011 bsam_at_h30.sp.ipt.ru:/usr/obj/usr/src/sys/BB i386
> 
> % ifconfig re0
> re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
> ether 00:1f:d0:91:e4:96
> inet 192.168.16.30 netmask 0xffffff00 broadcast 192.168.16.255
> inet 192.168.100.99 netmask 0xffffff00 broadcast 192.168.100.255
> inet 192.168.111.254 netmask 0xffffff00 broadcast 192.168.111.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> 
> % sudo tail /var/log/xferlog
> Aug 15 16:20:29 h30 tftpd[2653]: Full access allowedin
> /etc/hosts.allow
> Aug 15 16:20:29 h30 tftpd[2653]: Filename: '/pxeboot'
> Aug 15 16:20:29 h30 tftpd[2653]: Mode: 'octet'
> Aug 15 16:20:29 h30 tftpd[2653]: Got ERROR packet: TFTP Aborted
> Aug 15 16:20:29 h30 tftpd[2655]: Full access allowedin
> /etc/hosts.allow
> Aug 15 16:20:29 h30 tftpd[2655]: Filename: '/pxeboot'
> Aug 15 16:20:29 h30 tftpd[2655]: Mode: 'octet'
> Aug 15 16:20:29 h30 tftpd[2655]: 192.168.111.11: read request for
> /pxeboot: success
> Aug 15 16:20:29 h30 tftpd[2655]: Sent 268288 bytes in 0 seconds
> 
> % showmount -e
> /home 192.168.111.11 192.168.111.12
> /usr/local 192.168.111.11 192.168.111.12
> /diskless 192.168.111.11 192.168.111.12
> 
> % showmount -a
> All mount points on localhost:
> kaka:/diskless
> 
> % sockstat -4l
> USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
> www httpd 2537 3 tcp4 6 *:80 *:*
> root Xorg 2388 3 tcp4 *:6000 *:*
> www httpd 2386 3 tcp4 6 *:80 *:*
> www httpd 2385 3 tcp4 6 *:80 *:*
> www httpd 2384 3 tcp4 6 *:80 *:*
> www httpd 2383 3 tcp4 6 *:80 *:*
> www httpd 2382 3 tcp4 6 *:80 *:*
> root xdm 2381 3 udp4 192.168.111.254:177 *:*
> root inetd 2340 5 udp4 *:69 *:*
> root sendmail 2297 3 tcp4 127.0.0.1:25 *:*
> root sshd 2285 4 tcp4 *:22 *:*
> root httpd 2261 3 tcp4 6 *:80 *:*
> root cupsd 2177 6 tcp4 *:631 *:*
> root cupsd 2177 8 udp4 *:631 *:*
> mysql mysqld 2049 10 tcp4 *:3306 *:*
> dhcpd dhcpd 1967 7 udp4 *:67 *:*
> root ntpd 1837 20 udp4 *:123 *:*
> root ntpd 1837 22 udp4 192.168.16.30:123 *:*
> root ntpd 1837 23 udp4 192.168.100.99:123 *:*
> root ntpd 1837 24 udp4 192.168.111.254:123 *:*
> root ntpd 1837 27 udp4 127.0.0.1:123 *:*
> root nfsd 1763 5 tcp4 *:2049 *:*
> root mountd 1754 7 udp4 *:927 *:*
> root mountd 1754 8 tcp4 *:927 *:*
> root rpcbind 1676 9 udp4 *:111 *:*
> root rpcbind 1676 10 udp4 *:935 *:*
> root rpcbind 1676 11 tcp4 *:111 *:*
> root syslogd 1646 7 udp4 *:514 *:*
> -----
> 
> The kernel:
> -----
> include GENERIC
> ident BB
> 
> nooptions INVARIANTS
> nooptions INVARIANT_SUPPORT
> nooptions WITNESS
> nooptions WITNESS_SKIPSPIN
> 
> options INCLUDE_CONFIG_FILE
> 
> options SHMMAXPGS=65536
> options SEMMNI=40
> options SEMMNS=240
> options SEMUME=40
> options SEMMNU=120
> 
> options IPFIREWALL
> options IPFIREWALL_VERBOSE
> options IPFIREWALL_VERBOSE_LIMIT=100
> options IPFIREWALL_DEFAULT_TO_ACCEPT
> 
> # for zoneminder
> options SHMMNI=384
> options SHMSEG=256
> -----
> 
> Any help is appreciated. Thanks!
> 
By inspection, it appears that r224778 would break the nfsd server.
However, it's not obvious to me if the breakage could have resulted in
the failure you are seeing? (It would result in new socket connections
failing, but I would have thought that would be TCP specific?)

The only other 2 post-r224471 commits to the nfsd server are specific
to either rename or read so it seems unlikely they would affect a lookup.

Please try this one line patch:
--- nfs_nfsdport.c.sav	2011-08-15 18:13:22.000000000 -0400
+++ nfs_nfsdport.c	2011-08-15 18:14:01.000000000 -0400
_at__at_ -3036,7 +3036,6 _at__at_ nfssvc_nfsd(struct thread *td, struct nf
 		 */
 		if ((error = fget(td, sockarg.sock, CAP_SOCK_ALL, &fp)) != 0)
 			goto out;
-			return (error);
 		if (fp->f_type != DTYPE_SOCKET) {
 			fdrop(fp, td);
 			error = EPERM;

and let us know if it helps.

If this doesn't fix the problem, please do a:
"ps axHl" on the server after the diskless client has failed to boot,
to see what the nfsd threads are up to.
Also, if you have another running system, try an nfs mount using
"-o udp" to see if that works, since the pxeboot is using NFS over UDP.

Please let us know how it goes, rick