Re: ZFSROOT UEFI boot

From: Tomoaki AOKI <junchoon_at_dec.sakura.ne.jp>
Date: Fri, 29 Jan 2016 00:03:44 +0900
On Sun, 24 Jan 2016 18:39:08 +0000
Steven Hartland <killing_at_multiplay.co.uk> wrote:

> On 24/01/2016 12:53, Tomoaki AOKI wrote:
> > Unfortunately, this (and its committed successor and original for UFS)
> > fails to boot in some situation, like below. OTOH, gptzfsboot (and
> > maybe gptboot for UFS, too) is OK.
> >
> > When I select Disk1 from UEFI firmware, bootx64.efi in Disk1 EFI
> > partition is used and it searches /boot/loader.efi from Disk2 (in ZFS,
> > if none, in UFS) only.
> > And when I select Disk2, bootx64.efi in Disk2 EFI partition is used and
> > it searches /boot/loader.efi from Disk1 only.
> >
> >
> > In fact, this is a long-standing and living problem.
> > At past, USB memstick with head memstick.img (UEFI enabled, but
> > without root-on-ZFS support) booted fine, but after I added UFS2
> > partition in internal disk, the USB memstick didn't boot anymore.
> > It searches /boot/loader.efi from internal UFS and fails as it was
> > blank (only newfs'ed) at that time. Another USB memstick with stable/10
> > memstick.img is still fine, as it's still ancient BIOS based.
> >
> > Possibly, it's not a fault of boot1.efi but caused by differense in
> > implementation of UEFI firmware. If that's it, different boot1.efi
> > would be needed for each implementation.
> >
> > A bit more details of tests are as below. Not all combinations are
> > covered, but would be sufficient to determine above conclusion.
> >
> >
> > Common configurations for all tests:
> >    *Each disk has one EFI partition (p1), one freebsd-boot partition
> >     (p2), one swap partition (p3), one UFS partition (p4), and one
> >     ZFS pool (p5) with this order.
> >
> >    *Each partition has different GEOM label.
> >
> >    *In each disk, FreeBSD is installed as root on ZFS. No other OS.
> >
> >    *stable/10 (r294614) is installed in Disk1.
> >
> >    *head (r294567) is installed in Disk2.
> >
> >    *ZFS-enabled boot1.efi (head r294567) is used as bootx64.efi.
> >
> >
> > Set 1: Boot from Disk1 (select it in UEFI firmware).
> >         In all tests, /boot/loader.efi in Disk1 (both UFS and ZFS)
> >         are NOT searched at all.
> Could you clarify what you mean by this?
> 
> When looking performing the scan boot1 uses the following coding:
> * "+" = partition probe success (potential boot partition)
> * "." = partition probe unsupported (valid partition not detected)
> * "x" = partition probe error (unexpected error)
> >   1-1) Both UFS and ZFS has no /boot/loader.efi
> >          -> Fail to boot. Fall back to boot1 prompt.
> This is expected

I know.:-) Just done to see how boot1 errors are shown. 2-2, too.


> >   1-2) Disk2 UFS only has /boot/loader.efi, whole /boot of Disk2 ZFS
> >        is copied to UFS.
> >          -> head in Disk2 boots fine.
> What do you mean by "whole /boot of Disk2 ZFS is copied to UFS"?

Like below.

#mount /dev/ada1p4 /mnt
#cp -a /boot /mnt
#umount /mnt

Where /dev/ada[01]p4 is the UFS partition.

> >   1-3) Same as 1-2, except its /boot/loader.efi is overwritten by the
> >        one of stable/10.
> >          -> head in Disk2 boots fine, as loader.efi loads kernel from
> >             /boot/kernel/kernel in UFS and kernel with zfs.ko can mount
> >             root on ZFS specified by vfs.root.mountfrom.
> >
> >   1-4) Disk2 UFS only has /boot/loader.efi, whole /boot of Disk1 ZFS
> >        is copied to UFS and its /boot/loader.efi is overwritten by
> >        the one of head.
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> >   1-5) Disk2 ZFS only has /boot/loader.efi.
> >          -> head in Disk2 ZFS boots fine.
> >
> >   1-6) Both UFS and ZFS in Disk2 has /boot/loader.efi.
> >        (Mix of 1-4 and 1-5)
> >          -> head in Disk2 ZFS boots fine.
> >
> >
> > Set 2: Boot from Disk2 (select it in UEFI firmware).
> >         In all tests, /boot/loader.efi in Disk2 (both UFS and ZFS)
> >         are NOT searched at all.
> >
> >   2-1) Both UFS and ZFS has no /boot/loader.efi
> >          -> Fail to boot. Fall back to boot1 prompt.
> >             ZFS pool in Disk2 is shown before one in Disk1.
> >
> >   2-2) Disk1 UFS only has /boot/loader.efi, whole /boot of Disk2 ZFS
> >        is copied to UFS.
> >          -> head in Disk2 ZFS boots fine.
> >
> >   2-3) Disk1 UFS only has /boot/loader.efi, whole /boot of Disk1 ZFS
> >        is copied to UFS.
> >          -> stable/10 in Disk1 ZFS boots fine, as loader.efi loads
> >             kernel from /boot/kernel/kernel in UFS and kernel with zfs.ko
> >             can mount root on ZFS specified by vfs.root.mountfrom.
> >
> >   2-4) Disk1 UFS only has /boot/loader.efi, whole /boot of Disk1 ZFS
> >        is copied to UFS and its /boot/loader.efi is overwritten by
> >        the one of head.
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> >   2-5) Disk1 ZFS only has /boot/loader.efi of stable/10 itself.
> >          -> Fail to boot. Fall back to boot1 prompt.
> >             ZFS pool in Disk2 is shown before one in Disk1.
> >
> >   2-6) Disk1 ZFS only has /boot/loader.efi of head.
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> >   2-7) Both UFS and ZFS in Disk1 has /boot/loader.efi of head.
> >        (Mix of 2-2 and 2-6)
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> >   2-8) UFS has /boot/loader.efi of head (head kernel copied), but ZFS
> >        has /boot/loader.efi of stable/10 itself. (Mix of 2-2 and 2-5)
> >          -> Same as 2-5. Fail to boot. Fall back to boot1 prompt.
> >             ZFS pool in Disk2 is shown before one in Disk1.
> >
> > Set 3: Disk2 is removed. (Disk1 only environment)
> >
> >    3-1) ZFS only has /boot/loader.efi of head.
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> >    3-2) Same as 2-2 without Disk2.
> >          -> Fail to boot. Fall back to loader prompt.
> >             (Of course. Specified root device doesn't exists.)
> >
> >    3-3) Same as 2-4 without Disk2.
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> >    3-4) Both UFS and ZFS have /boot/loader.efi of head.
> >          -> stable/10 in Disk1 ZFS boots fine.
> >
> You're tests here are quite hard to follow as each one refers to other 
> details. It would be good to get:
> 1. gpart show for each test

It's exactly the same throughout all tests, as it's on single hardware.

ada0 is stable/10 installation, my main working environment.

ada1 is head installation, test environment mainly for this case or
alike that VMs don't work well.

da0 is head installation, dd'ed with memstick.img.


a) Without USB memstick:

=>        34  2000409197  ada0  GPT  (954G)
          34        2014        - free -  (1.0M)
        2048      409600     1  efi  (200M)
      411648        1024     2  freebsd-boot  (512K)
      412672        1024        - free -  (512K)
      413696    20971520     3  freebsd-swap  (10G)
    21385216    10485760     4  freebsd-ufs  (5.0G)
    31870976  1968537600     5  freebsd-zfs  (939G)
  2000408576         655        - free -  (328K)

=>       34  488397101  ada1  GPT  (233G)
         34       2014        - free -  (1.0M)
       2048     204800     1  efi  (100M)
     206848       1024     2  freebsd-boot  (512K)
     207872       1024        - free -  (512K)
     208896   20971520     3  freebsd-swap  (10G)
   21180416   16777216     4  freebsd-ufs  (8.0G)
   37957632  450439168     5  freebsd-zfs  (215G)
  488396800        335        - free -  (168K)

=>       34  488397101  diskid/DISK-S21MNXAG821947P  GPT  (233G)
         34       2014                               - free -  (1.0M)
       2048     204800                            1  efi  (100M)
     206848       1024                            2  freebsd-boot
(512K) 207872       1024                               - free -  (512K)
     208896   20971520                            3  freebsd-swap  (10G)
   21180416   16777216                            4  freebsd-ufs  (8.0G)
   37957632  450439168                            5  freebsd-zfs  (215G)
  488396800        335                               - free -  (168K)


b) With USB memstick (USB portion only, as others are the same above.):

=>      3  1617794  da0  GPT  (1.9G) [CORRUPT]
        3     1600    1  efi  (800K)
     1603       34    2  freebsd-boot  (17K)
     1637  1614112    3  freebsd-ufs  (788M)
  1615749     2048    4  freebsd-swap  (1.0M)


> 2. information on which partitions have valid head /boot/loader.efi, 
> don't mix and match /boot/boot1.efifat and /boot/loader.efi from 
> different versions.

I don't think digging deeper in each test is important.
What I wanted to determine thorouout these tests is "Where is the bad
point to get selected disk isn't booted?".

Now I determined that latest loader.efi and kernel looks OK, OTOH,
boot1.efi or UEFI firmware has the cause of problem.


> 3. The output / exact error you see.

As I don't have serial console, I took photos and typed up.
So there can be typo in it.
In all cases, /boot/loader.efi in all partitions are renamed to let
boot1 panic.

A) Disk1 is selected, no USB memstick inserted:

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Probing 12 block devices........++...++ done
    ZFS found the following pools: zsysS02 zsysS01
    UFS found 2 partitions
Failed to lookup /boot/loader.efi on pool zsysS02 (2)
ZFS failed to load /boot/loader.efi (2)
panic: No bootable partitions found!


B) Disk1 is selected, one USB memstick (memstick.img of head) inserted:

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Probing 12 block devices........+....++...++ done
    ZFS found the following pools: zsysS02 zsysS01
    UFS found 3 partitions
Failed to lookup /boot/loader.efi on pool zsysS02 (2)
ZFS failed to load /boot/loader.efi (2)
panic: No bootable partitions found!


C) Disk2 is selected, no USB memstick inserted:

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Probing 12 block devices........++...++ done
    ZFS found the following pools: zsysS01 zsysS02
    UFS found 2 partitions
Failed to lookup /boot/loader.efi on pool zsysS01 (2)
ZFS failed to load /boot/loader.efi (2)
panic: No bootable partitions found!


D) Disk2 is selected, one USB memstick (memstick.img of head) inserted:

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Probing 12 block devices........++...+....++ done
    ZFS found the following pools: zsysS01 zsysS02
    UFS found 3 partitions
Failed to lookup /boot/loader.efi on pool zsysS01 (2)
ZFS failed to load /boot/loader.efi (2)
panic: No bootable partitions found!


E) USB memstick is selected:

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Probing 12 block devices........++...++...+. done
    ZFS found the following pools: zsysS01 zsysS02
    UFS found 3 partitions
Failed to lookup /boot/loader.efi on pool zsysS01 (2)
ZFS failed to load /boot/loader.efi (2)
panic: No bootable partitions found!


> To be clear where boot1 is read from should have no impact on which 
> partition is booted, boo1 will pick the first partition it finds that it 
> believes is bootable and will boot from it, if that fails then it will 
> try the next etc.

It's exactly the NO GOOD point. The disk where boot1 is read from
should be where loader.efi and loader.conf are first read.

In rescue drive case, including installation memstick, I believe boot
from selected disk is mandatory feature of boot1.efi, unless some boot
management feature before loader.efi is loaded (like boot0ext did) is
provided and enabled.

If one need to fix crashed installation on first disk, but
its /boot/loader.efi alone is healthy while kernels and *.4th files are
crashed, boot1.efi in emergency memstick will load the /boot/loader.efi
and boot should fail.
But if boot1.efi in emergency memstick is promised to
load /boot/loader.efi in itself first, boot should be fine and fix
can be proceeded without opening the box. 


> Be aware that ZFS is always used in preference to UFS and that first 
> partition (according to the iteration order of EFI) will be used to read 
> the loader.efi from.

I now understood it through my tests. :-)

If there's the functionality to select drive/partition to boot like
boot0 has (had before? ), I (and anyone suffering like me) would be
happy. If it would never be planned (possibly secure boot related),
loading loader.efi from the same disk with boot1.efi would become more
and more important, at least for installation media.

Regards.

>      Regards
>      Steve
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
> 


-- 
Tomoaki AOKI    junchoon_at_dec.sakura.ne.jp
Received on Thu Jan 28 2016 - 14:03:55 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:02 UTC