FreeBSD/powerpc64 (POWER9) with ZFS on /

ZFS is by default not available for partitioning in bsdinstall. We can, however, use manual partitioning in shell to make ZFS partitions.

But what if we want to do ZFS on /? This is currently not supported out-of-the-box. The reason is that Petitboot, used on POWER9 to boot kernel, doesn't support loading modules (at least when booting FreeBSD). ZFS is by default distributed only as a module, to overcome licensing issues.

There are two ways to boot ZFS on root with Petitboot:
1. We can use initramfs, which will load necessary modules. It's said to be supported on FreeBSD/powerpc64, but no one knows how to create one for FreeBSD :)
2. We can actually compile a custom kernel with ZFS in it (not as a module). This the approach that I use. Unfortunately, I can't redistribute this kernel, because of ZFS's license. I created a custom installation iso, which uses my kernel with ZFS included.

Here are the steps to do it (on an installed powerpc64 system, but should be possible on other architectures after redefining TARGET and TARGET_ARCH):
1. cd /usr/src/sys/powerpc/conf
2. cat << EOF > ZFS
include GENERIC64
options ZFS
EOF
3. cd ../../../
3. make buildworld buildkernel KERNCONF=ZFS
4. cd release
5. make release

When it's done, there will be iso files in /usr/obj/usr/src/powerpc.powerpc64/release.

You can use it to install ZFS on root.

Note that you need a FAT32 partition to mount on /boot.
This partition needs to have /boot/etc/kboot.conf file, with a line like:
FreeBSD='/kernel/kernel kernelname=/boot/kernel/kernel vfs.root.mountfrom=zfs:zroot/ROOT/default'

You also need the appriopriate /etc/fstab:
/dev/nda0p1     /boot           msdosfs rw      2       2

Other than that, howto at https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot#Create_Partitions (from point 2., omit 1.) is totally ok to follow.

Thanks to mmacy for informing me about this little trick with including ZFS to kernel.

-- 
 _________________________________________ 
/ It has been said that man is a rational \
| animal. All my life I have been         |
| searching for evidence which could      |
| support this.                           |
|                                         |
\ -- Bertrand Russell                     /
 ----------------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||