Re: r253070 and "disappearing" zpool

[turning this into a public discussion with Glen's permission]

on 10/07/2013 21:05 Glen Barber said the following:
> Hi,
> 
> My setup is like this:
> 
>     root_at_nucleus:/usr/src # zpool list
>     NAME     SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
>     zboot0  9.94G   379M  9.57G     3%  1.00x  ONLINE  -
>     zroot0   159G   113G  46.2G    70%  1.00x  ONLINE  -
> 
>     root_at_nucleus:/usr/src # zpool get bootfs
>     NAME    PROPERTY  VALUE   SOURCE
>     zboot0  bootfs    -       default
>     zroot0  bootfs    -       default
> 
>     root_at_nucleus:/usr/src # zfs list zboot0
>     NAME     USED  AVAIL  REFER  MOUNTPOINT
>     zboot0   379M  9.41G   281M  /bootdir
> 
>     root_at_nucleus:/usr/src # zfs list zroot0
>     NAME     USED  AVAIL  REFER  MOUNTPOINT
>     zroot0   113G  43.7G   147M  /
> 
> 'zroot0' is a GELI-backed pool, so I have this to fix the boot process:
> 
>     root_at_nucleus:/usr/src # ll /boot
>     lrwxr-xr-x  1 root  wheel  12 Aug 25  2012 /boot_at_ -> bootdir/boot
> 
> I upgraded from head/ on July 1 to r253159, and when I rebooted the
> system, I could correctly boot from the "/bootdir/boot".  Once I enter
> the GELI passphrase, "/" (from zroot0) is mounted.  Normally, everything
> would be okay at that point, but since the upgrade, "/bootdir/boot"
> disappears because the zboot0 pool is not imported as it was before.
> 
> Any thoughts?

I think that this setup (on ZFS level) is quite untypical, although not
impossible on FreeBSD (and perhaps only FreeBSD).
It's untypical because you have separate boot pool (where loader, loader.conf
and kernel are taken from) and root pool (where "/" is mounted from).

There is this "magic" zpool.cache file that essentially tells what pools should
be automatically imported.  On FreeBSD this file lives in /boot/zfs directory
while originally (in Solaris and its descendants) its home is /etc/zfs.

Until recently FreeBSD could use only zpool.cache from a boot pool and, in fact,
if a root pool was different from a boot pool the root pool had to be in
zpool.cache.
I changed things a little bit and now a root pool does not have to be in
zpool.cache.  Also, now zpool.cache is taken from the root
pool, or to be more precise from a root filesystem (whatever happens to be
/boot/zfs/zpool.cache after "/" is mounted).  I am considering if perhaps now we
should move zpool.cache back to /etc/zfs/.

So, I see three ways of resolving the problem that my changes caused for your
configuration.

1.  [the easiest] Put zpool.cache loading instructions that used to be in
defaults/loader.conf into your loader.conf.  This way everything should work as
before -- zpool.cache would be loaded from your boot pool.

2. Somehow (I don't want to go into any technical details here) arrange that
your root pool has /boot/zfs/zpool.cache that describes your boot pool.  This is
probably hard given that your /boot is a symlink at the moment.  This probably
would be easier to achieve if zpool.cache lived in /etc/zfs.

3. [my favorite]  Remove an artificial difference between your boot and root
pools, so that they are a single root+boot pool (as zfs gods intended).  As far
as I understand your setup, you use GELI to protect some sensitive data.
Apparently your kernel is not sensitive data, so I wonder if your /bin/sh or
/sbin/init are really sensitive either.
So perhaps you can arrange your unencrypted pool to hold all of the base system
(boot + root) and put all your truly sensitive filesystems (like e.g. /home or
/var/data or /opt/xyz) onto your encrypted pool.
ZFS is really flexible, you can use mountpoint and canmount properties to place
your filesystems from same or different pools into whatever file namespace
hierarchy you desire.  Remember that your filesystem hierarchy in the mount
namespace does not always have to be the same as your ZFS dataset hierarchy.

I hope that this makes sense to you.
If you have any additional questions, please do not hesitate.

P.S.
ZFS/FreeBSD boot process is extremely flexible.  For example zfsboot can take
zfsloader from pool1/fsA, zfsloader can boot kernel from pool2/fsB and kernel
can mount / from pool3/fsC.  Of these 3 filesystems from where should
zpool.cache be taken?
My firm opinion is that it should be taken from / (pool3/fsC in the example
above).  Because it is the root filesystem that defines what a system is going
to do ultimately: what daemons are started, with what configurations, etc.
And thus it should also determine what pools to auto-import.
We can say that zpool.cache is analogous to /etc/fstab in this respect.

So I understand that my change causes a problem for a setup like yours, but I
believe that the change is correct.

-- 
Andriy Gapon