Re: r346025: ZFS filesystems do not mount anymore

> On Apr 9, 2019, at 18:29, Michael Butler <imb_at_protected-networks.net> wrote:
> 
>> On 2019-04-09 09:30, O. Hartmann wrote:
>> On Tue, 9 Apr 2019 10:42:47 +0200 (CEST)
>> Trond Endrestøl <Trond.Endrestol_at_fagskolen.gjovik.no> wrote:
>> 
>>>> On Tue, 9 Apr 2019 10:06+0200, Trond Endrestøl wrote:
>>>> 
>>>>> On Mon, 8 Apr 2019 19:58+0200, O. Hartmann wrote:
>>>>> 
>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>> Hash: SHA256
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> after a bunch of updates this weekend, mounting ZFS filesystems on
>>>>> CURRENT r346025 doesn't work anymore at boot time when ZFS is
>>>>> built-in-kernel. zfs_enable="YES" is set in /etc/rc.conf.
>>>>> 
>>>>> After the system has booted, mounting all ZFS filesystems via "zfs mount
>>>>> -a" operates as expected and all filesystems are available as usual.  
>>>> 
>>>> I blame r346017:
>>>> 
>>>> Trying to mount root from zfs:zroot/ROOT/20190409-r346017 []
>>>> Setting hostuuid: 7b624d1c-4bc3-e2cb-4ac2-6487a4b4774c.
>>>> Setting hostid: 0xf9071336.
>>>> interface zfsctrl.1 already present in the KLD 'kernel'!
>>>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>>>> kldload: an error occurred while loading module zfs. Please check dmesg(8)
>>>> for more details. /etc/rc: WARNING: Unable to load kernel module zfs
>>>> Starting file system checks:
>>>> Mounting local filesystems:.
>>>> interface zfsctrl.1 already present in the KLD 'kernel'!
>>>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>>>> kldload: an error occurred while loading module zfs. Please check dmesg(8)
>>>> for more details. /etc/rc: WARNING: Unable to load kernel module zfs
>>>> interface zfsctrl.1 already present in the KLD 'kernel'!
>>>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>>>> kldload: an error occurred while loading module zfs. Please check dmesg(8)
>>>> for more details. /etc/rc: WARNING: Unable to load kernel module zfs
>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat
>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>> 
>>>> [...]
>>>> 
>>>> Creating and/or trimming log files.
>>>> Starting syslogd.
>>>> Apr  9 09:48:32 <kern.emerg> freebsd-head-zfs syslogd: /var/log/security:
>>>> No such file or directory Setting date via ntp.
>>>> 9 Apr 09:48:38 ntpdate[1073]: step time server 2001:W:X:Y::Z offset
>>>> -0.992370 sec No core dumps found.
>>>> Clearing /tmp (X related).
>>>> Updating motd:.
>>>> Mounting late filesystems:mount: /usr/compat/linux: No such file or
>>>> directory mount: /usr/compat/linux: No such file or directory
>>>> mount: /usr/compat/linux: No such file or directory
>>>> mount: /usr/compat/linux: No such file or directory
>>>> .
>>>> Mounting /etc/fstab filesystems failed,  startup aborted
>>>> ERROR: ABORTING BOOT (sending SIGTERM to parent)!
>>>> Enter full pathname of shell or RETURN for /bin/sh:
>>>> root_at_freebsd-head-zfs:/ # zfs mount -av
>>>> root_at_freebsd-head-zfs:/ # mount -al
>>>> root_at_freebsd-head-zfs:/ # exit
>>>> Setting hostuuid: 7b624d1c-4bc3-e2cb-4ac2-6487a4b4774c.
>>>> Setting hostid: 0xf9071336.
>>>> interface zfsctrl.1 already present in the KLD 'kernel'!
>>>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>>>> kldload: an error occurred while loading module zfs. Please check dmesg(8)
>>>> for more details. /etc/rc: WARNING: Unable to load kernel module zfs
>>>> Fast boot: skipping disk checks.
>>>> Mounting local filesystems:.
>>>> interface zfsctrl.1 already present in the KLD 'kernel'!
>>>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>>>> kldload: an error occurred while loading module zfs. Please check dmesg(8)
>>>> for more details. /etc/rc: WARNING: Unable to load kernel module zfs
>>>> interface zfsctrl.1 already present in the KLD 'kernel'!
>>>> linker_load_file: /boot/kernel/zfs.ko - unsupported file type
>>>> kldload: an error occurred while loading module zfs. Please check dmesg(8)
>>>> for more details. /etc/rc: WARNING: Unable to load kernel module zfs
>>>> ELF ldconfig
>>>> path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/perl5/5.28/mach/CORE
>>>> 32-bit compatibility ldconfig path: /usr/lib32
>>>> 
>>>> [...]  
>>> 
>>> At a later point, this happens:
>>> 
>>> Starting syslogd.
>>> usage: protect [-i] command
>>>       protect [-cdi] -g pgrp | -p pid
>>> Setting date via ntp.
>>> 
>>> It looks like syslogd is no longer protected against the OOM killer.
>>> 
>>>> Once I manually mount the remaining filesystems, multiuser boot 
>>>> proceeds as expected. My kernel has options ZFS, and the boot scripts 
>>>> doesn't account for this scenario.  
>>> 
>>> 
>> 
>> Hello,
>> 
>> thank you for investigating! I think you're correct. I was wondering how this
>> change could make it into the tree with this kind of impact.
>> 
>> Maybe filing a PR would be fine?
>> 
> 
> I just confirmed that the change at SVN r346017 after 'mergemaster -U'
> yields a (ZFS-based) system which can only be brought into normal
> service through:
> 
>    a single-user boot
>    mount -a
>    mount -u -o rw /
>    zfs mount -a
> 
> Reverting this change restores normal operation at start-up.
> 
> ZFS is also compiled into the kernel on my system,
> 
>    imb

CCing crees_at_, the author of r346017, for context. I will file a PR soon if one hasn’t already been filed.
-Enji