Re: ZFSROOT UEFI boot

On 24/01/2016 12:53, Tomoaki AOKI wrote:
> Unfortunately, this (and its committed successor and original for UFS)
> fails to boot in some situation, like below. OTOH, gptzfsboot (and
> maybe gptboot for UFS, too) is OK.
>
> When I select Disk1 from UEFI firmware, bootx64.efi in Disk1 EFI
> partition is used and it searches /boot/loader.efi from Disk2 (in ZFS,
> if none, in UFS) only.
> And when I select Disk2, bootx64.efi in Disk2 EFI partition is used and
> it searches /boot/loader.efi from Disk1 only.
>
>
> In fact, this is a long-standing and living problem.
> At past, USB memstick with head memstick.img (UEFI enabled, but
> without root-on-ZFS support) booted fine, but after I added UFS2
> partition in internal disk, the USB memstick didn't boot anymore.
> It searches /boot/loader.efi from internal UFS and fails as it was
> blank (only newfs'ed) at that time. Another USB memstick with stable/10
> memstick.img is still fine, as it's still ancient BIOS based.
>
> Possibly, it's not a fault of boot1.efi but caused by differense in
> implementation of UEFI firmware. If that's it, different boot1.efi
> would be needed for each implementation.
>
> A bit more details of tests are as below. Not all combinations are
> covered, but would be sufficient to determine above conclusion.
>
>
> Common configurations for all tests:
>    *Each disk has one EFI partition (p1), one freebsd-boot partition
>     (p2), one swap partition (p3), one UFS partition (p4), and one
>     ZFS pool (p5) with this order.
>
>    *Each partition has different GEOM label.
>
>    *In each disk, FreeBSD is installed as root on ZFS. No other OS.
>
>    *stable/10 (r294614) is installed in Disk1.
>
>    *head (r294567) is installed in Disk2.
>
>    *ZFS-enabled boot1.efi (head r294567) is used as bootx64.efi.
>
>
> Set 1: Boot from Disk1 (select it in UEFI firmware).
>         In all tests, /boot/loader.efi in Disk1 (both UFS and ZFS)
>         are NOT searched at all.
Could you clarify what you mean by this?

When looking performing the scan boot1 uses the following coding:
* "+" = partition probe success (potential boot partition)
* "." = partition probe unsupported (valid partition not detected)
* "x" = partition probe error (unexpected error)
>   1-1) Both UFS and ZFS has no /boot/loader.efi
>          -> Fail to boot. Fall back to boot1 prompt.
This is expected
>   1-2) Disk2 UFS only has /boot/loader.efi, whole /boot of Disk2 ZFS
>        is copied to UFS.
>          -> head in Disk2 boots fine.
What do you mean by "whole /boot of Disk2 ZFS is copied to UFS"?
>   1-3) Same as 1-2, except its /boot/loader.efi is overwritten by the
>        one of stable/10.
>          -> head in Disk2 boots fine, as loader.efi loads kernel from
>             /boot/kernel/kernel in UFS and kernel with zfs.ko can mount
>             root on ZFS specified by vfs.root.mountfrom.
>
>   1-4) Disk2 UFS only has /boot/loader.efi, whole /boot of Disk1 ZFS
>        is copied to UFS and its /boot/loader.efi is overwritten by
>        the one of head.
>          -> stable/10 in Disk1 ZFS boots fine.
>
>   1-5) Disk2 ZFS only has /boot/loader.efi.
>          -> head in Disk2 ZFS boots fine.
>
>   1-6) Both UFS and ZFS in Disk2 has /boot/loader.efi.
>        (Mix of 1-4 and 1-5)
>          -> head in Disk2 ZFS boots fine.
>
>
> Set 2: Boot from Disk2 (select it in UEFI firmware).
>         In all tests, /boot/loader.efi in Disk2 (both UFS and ZFS)
>         are NOT searched at all.
>
>   2-1) Both UFS and ZFS has no /boot/loader.efi
>          -> Fail to boot. Fall back to boot1 prompt.
>             ZFS pool in Disk2 is shown before one in Disk1.
>
>   2-2) Disk1 UFS only has /boot/loader.efi, whole /boot of Disk2 ZFS
>        is copied to UFS.
>          -> head in Disk2 ZFS boots fine.
>
>   2-3) Disk1 UFS only has /boot/loader.efi, whole /boot of Disk1 ZFS
>        is copied to UFS.
>          -> stable/10 in Disk1 ZFS boots fine, as loader.efi loads
>             kernel from /boot/kernel/kernel in UFS and kernel with zfs.ko
>             can mount root on ZFS specified by vfs.root.mountfrom.
>
>   2-4) Disk1 UFS only has /boot/loader.efi, whole /boot of Disk1 ZFS
>        is copied to UFS and its /boot/loader.efi is overwritten by
>        the one of head.
>          -> stable/10 in Disk1 ZFS boots fine.
>
>   2-5) Disk1 ZFS only has /boot/loader.efi of stable/10 itself.
>          -> Fail to boot. Fall back to boot1 prompt.
>             ZFS pool in Disk2 is shown before one in Disk1.
>
>   2-6) Disk1 ZFS only has /boot/loader.efi of head.
>          -> stable/10 in Disk1 ZFS boots fine.
>
>   2-7) Both UFS and ZFS in Disk1 has /boot/loader.efi of head.
>        (Mix of 2-2 and 2-6)
>          -> stable/10 in Disk1 ZFS boots fine.
>
>   2-8) UFS has /boot/loader.efi of head (head kernel copied), but ZFS
>        has /boot/loader.efi of stable/10 itself. (Mix of 2-2 and 2-5)
>          -> Same as 2-5. Fail to boot. Fall back to boot1 prompt.
>             ZFS pool in Disk2 is shown before one in Disk1.
>
> Set 3: Disk2 is removed. (Disk1 only environment)
>
>    3-1) ZFS only has /boot/loader.efi of head.
>          -> stable/10 in Disk1 ZFS boots fine.
>
>    3-2) Same as 2-2 without Disk2.
>          -> Fail to boot. Fall back to loader prompt.
>             (Of course. Specified root device doesn't exists.)
>
>    3-3) Same as 2-4 without Disk2.
>          -> stable/10 in Disk1 ZFS boots fine.
>
>    3-4) Both UFS and ZFS have /boot/loader.efi of head.
>          -> stable/10 in Disk1 ZFS boots fine.
>
You're tests here are quite hard to follow as each one refers to other 
details. It would be good to get:
1. gpart show for each test
2. information on which partitions have valid head /boot/loader.efi, 
don't mix and match /boot/boot1.efifat and /boot/loader.efi from 
different versions.
3. The output / exact error you see.

To be clear where boot1 is read from should have no impact on which 
partition is booted, boo1 will pick the first partition it finds that it 
believes is bootable and will boot from it, if that fails then it will 
try the next etc.

Be aware that ZFS is always used in preference to UFS and that first 
partition (according to the iteration order of EFI) will be used to read 
the loader.efi from.

     Regards
     Steve