Re: GEOM hangover disables NFS

On Wed, Nov 12, 2008 at 03:59:03PM -0800, Steve Kargl wrote:
> On three nodes in my cluster (nodes n17, n18, and n19), I had
> GEOM use /dev/ad4s1e for tests with gmirror and ggated/ggatec.
> I found that GEOM was insufficient for my needs and decided 
> to return the 3 partitions to NFS-exported partitions.  It seems
> that once GEOM touches a partition, the partition can no longer
> be used by NFS.
>
> I'll illustrute the problem with n17:/dev/ad4s1e.  In what follows,
> n10 is the master node.  Both n10 and n17 have brand new worlds
> and kernels from about 45 minutes ago.
> 
> n10:kargl[203] ssh n17
> n17:kargl[201] df
> Filesystem        1M-blocks  Used  Avail Capacity  Mounted on
> /dev/ad4s1a             247   104    123    46%    /
> devfs                     0     0      0   100%    /dev
> /dev/ad4s1e          222780     0 204958     0%    /data
> /dev/ad4s1d            3962   182   3463     5%    /usr
> n10:/home            193947 92855  85576    52%    /home
> n10:/usr/local        19832 10494   7750    58%    /usr/local
> 
> n17:kargl[202] tail -1 /etc/exports
> /data   -alldirs        node10 node21
> 
> The above is after a 'newfs -U /dev/ad4s1e' and a reboot.
> 
> n10:root[244] ls / | grep -E ^n
> n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/
> 
> n10:root[245] mount_nfs -o tcp n17:/data /n17
> n10:root[246] mount -v | grep n17
> n17:/data on /n17 (nfs, fsid 0eff000303000000)
> n10:root[247] ls /n17
> ls: /n17: Input/output error
> n10:root[248] ls / | grep -E ^n
> ls: n17: Input/output error
> n11/ n12/ n13/ n14/ n15/ n16/ n18/ n19/ n20/ n21/
> 
> n10:root[251] umount /n17
> n10:root[252] ls / | grep -E ^n
> n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/
> 
> So, how does one exorcise GEOM from /dev/ad4s1e?

All disks and partitions are always represented as GEOM devices.  This is
the only way to access storage so I'm pretty sure you don't actually
want to get rid of GEOM. :)

The output of "sysctl -b kern.geom.conftxt" would show if there were
bits that were being picked up by a stray geom consumer.  A dmesg from
the problem nodes might also be helpful in determining what's wrong.
The best guess would be left over state, probably at the end of the
volumes.  If local access to the file system actually works, NFS really
shouldn't care what's been done to the disk below the file system.

-- Brooks