Re: GEOM hangover disables NFS

Steve Kargl wrote:
> On three nodes in my cluster (nodes n17, n18, and n19), I had
> GEOM use /dev/ad4s1e for tests with gmirror and ggated/ggatec.
> I found that GEOM was insufficient for my needs and decided 
> to return the 3 partitions to NFS-exported partitions.  It seems
> that once GEOM touches a partition, the partition can no longer
> be used by NFS.
> 
> I'll illustrute the problem with n17:/dev/ad4s1e.  In what follows,
> n10 is the master node.  Both n10 and n17 have brand new worlds
> and kernels from about 45 minutes ago.
> 
> n10:kargl[203] ssh n17
> n17:kargl[201] df
> Filesystem        1M-blocks  Used  Avail Capacity  Mounted on
> /dev/ad4s1a             247   104    123    46%    /
> devfs                     0     0      0   100%    /dev
> /dev/ad4s1e          222780     0 204958     0%    /data
> /dev/ad4s1d            3962   182   3463     5%    /usr
> n10:/home            193947 92855  85576    52%    /home
> n10:/usr/local        19832 10494   7750    58%    /usr/local
> 
> n17:kargl[202] tail -1 /etc/exports
> /data   -alldirs        node10 node21
> 
> The above is after a 'newfs -U /dev/ad4s1e' and a reboot.
> 
> n10:root[244] ls / | grep -E ^n
> n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/
> 
> n10:root[245] mount_nfs -o tcp n17:/data /n17
> n10:root[246] mount -v | grep n17
> n17:/data on /n17 (nfs, fsid 0eff000303000000)
> n10:root[247] ls /n17
> ls: /n17: Input/output error
> n10:root[248] ls / | grep -E ^n
> ls: n17: Input/output error
> n11/ n12/ n13/ n14/ n15/ n16/ n18/ n19/ n20/ n21/
> 
> n10:root[251] umount /n17
> n10:root[252] ls / | grep -E ^n
> n11/ n12/ n13/ n14/ n15/ n16/ n17/ n18/ n19/ n20/ n21/
> 
> So, how does one exorcise GEOM from /dev/ad4s1e?

Your message is very confusing. Here's some information that could help you:

* "GEOM" is just a name for "system of handling disk-like devices in
FreeBSD". If you "remove GEOM" (which would probably be hard to do and
would involve heavily modifying the kernel source) you will have no
access to your disk drives and other similar devices.

* Using ggated/ggatec you can either:
   - mount the file system exactly once as rw and none as ro (doesn't
matter if the file system is on local node or exported, the total is
important)
   - mount the file system as ro any number of times and none as rw.
(In other words, only one machine can have the file system mounted for
reading and writing. Once a file system has been mounted as rw, no other
machines can do anything with it. This has nothing to do with NFS as NFS
operates on top of a mounted file system and inherits its rules.)

* You can remove gmirror metadata on devices with "gmirror clean".
ggated and ggatec don't store metadata on devices.