Re: ZFS i/o error in recent 12.0

From: Allan Jude <allanjude_at_freebsd.org>
Date: Mon, 19 Mar 2018 19:09:31 -0400
On 2018-03-19 19:00, KIRIYAMA Kazuhiko wrote:
> Hi,
> 
> I've been encountered suddenly death in ZFS full volume
> machine(r330434) about 10 days after installation[1]:
> 
> ZFS: i/o error - all block copies unavailable
> ZFS: can't read MOS of pool zroot
> gptzfsboot: failed to mount default pool zroot
> 
> FreeBSD/x86 boot
> ZFS: i/o error - all block copies unavailable
> ZFS: can't find dataset u
> Default: zroot/<0x0>:
> boot: 
> 
> Partition is bellow:
> 
>  # gpart show /dev/mfid0
> =>         40  31247564720  mfid0  GPT  (15T)
>            40       409600      1  efi  (200M)
>        409640         1024      2  freebsd-boot  (512K)
>        410664          984         - free -  (492K)
>        411648    268435456      3  freebsd-swap  (128G)
>     268847104  30978715648      4  freebsd-zfs  (14T)
>   31247562752         2008         - free -  (1.0M)
> 
> # 
> 
> But nothing had beed happend in old current ZFS full volume
> machine(r327038M). According to [2] the reason is boot/zfs/zpool.cache
> inconsistent. I've tried to cope with this by repairing
> /boot [3] from rescue bootable USB as follows:
> 
> # kldload zfs
> # zpool import 
>    pool: zroot
>      id: 17762298124265859537
>   state: ONLINE
>  action: The pool can be imported using its name or numeric identifier.
>  config:
> 
>         zroot       ONLINE
>           mfid0p4   ONLINE
> # zpool import -fR /mnt zroot
> # df -h
> Filesystem                  Size    Used   Avail Capacity  Mounted on
> /dev/da0p2                   14G    1.6G     11G    13%    /
> devfs                       1.0K    1.0K      0B   100%    /dev
> zroot/.dake                  14T     18M     14T     0%    /mnt/.dake
> zroot/ds                     14T     96K     14T     0%    /mnt/ds
> zroot/ds/backup              14T     88K     14T     0%    /mnt/ds/backup
> zroot/ds/backup/kazu.pis     14T     31G     14T     0%    /mnt/ds/backup/kazu.pis
> zroot/ds/distfiles           14T    7.9M     14T     0%    /mnt/ds/distfiles
> zroot/ds/obj                 14T     10G     14T     0%    /mnt/ds/obj
> zroot/ds/packages            14T    4.0M     14T     0%    /mnt/ds/packages
> zroot/ds/ports               14T    1.3G     14T     0%    /mnt/ds/ports
> zroot/ds/src                 14T    2.6G     14T     0%    /mnt/ds/src
> zroot/tmp                    14T     88K     14T     0%    /mnt/tmp
> zroot/usr/home               14T    136K     14T     0%    /mnt/usr/home
> zroot/usr/local              14T     10M     14T     0%    /mnt/usr/local
> zroot/var/audit              14T     88K     14T     0%    /mnt/var/audit
> zroot/var/crash              14T     88K     14T     0%    /mnt/var/crash
> zroot/var/log                14T    388K     14T     0%    /mnt/var/log
> zroot/var/mail               14T     92K     14T     0%    /mnt/var/mail
> zroot/var/ports              14T     11M     14T     0%    /mnt/var/ports
> zroot/var/tmp                14T    6.0M     14T     0%    /mnt/var/tmp
> zroot/vm                     14T    2.8G     14T     0%    /mnt/vm
> zroot/vm/tbedfc              14T    1.6G     14T     0%    /mnt/vm/tbedfc
> zroot                        14T     88K     14T     0%    /mnt/zroot
> # zfs list
> NAME                       USED  AVAIL  REFER  MOUNTPOINT
> zroot                     51.1G  13.9T    88K  /mnt/zroot
> zroot/.dake               18.3M  13.9T  18.3M  /mnt/.dake
> zroot/ROOT                1.71G  13.9T    88K  none
> zroot/ROOT/default        1.71G  13.9T  1.71G  /mnt/mnt
> zroot/ds                  45.0G  13.9T    96K  /mnt/ds
> zroot/ds/backup           30.8G  13.9T    88K  /mnt/ds/backup
> zroot/ds/backup/kazu.pis  30.8G  13.9T  30.8G  /mnt/ds/backup/kazu.pis
> zroot/ds/distfiles        7.88M  13.9T  7.88M  /mnt/ds/distfiles
> zroot/ds/obj              10.4G  13.9T  10.4G  /mnt/ds/obj
> zroot/ds/packages         4.02M  13.9T  4.02M  /mnt/ds/packages
> zroot/ds/ports            1.26G  13.9T  1.26G  /mnt/ds/ports
> zroot/ds/src              2.56G  13.9T  2.56G  /mnt/ds/src
> zroot/tmp                   88K  13.9T    88K  /mnt/tmp
> zroot/usr                 10.4M  13.9T    88K  /mnt/usr
> zroot/usr/home             136K  13.9T   136K  /mnt/usr/home
> zroot/usr/local           10.2M  13.9T  10.2M  /mnt/usr/local
> zroot/var                 17.4M  13.9T    88K  /mnt/var
> zroot/var/audit             88K  13.9T    88K  /mnt/var/audit
> zroot/var/crash             88K  13.9T    88K  /mnt/var/crash
> zroot/var/log              388K  13.9T   388K  /mnt/var/log
> zroot/var/mail              92K  13.9T    92K  /mnt/var/mail
> zroot/var/ports           10.7M  13.9T  10.7M  /mnt/var/ports
> zroot/var/tmp             5.98M  13.9T  5.98M  /mnt/var/tmp
> zroot/vm                  4.33G  13.9T  2.75G  /mnt/vm
> zroot/vm/tbedfc           1.58G  13.9T  1.58G  /mnt/vm/tbedfc
> # zfs mount zroot/ROOT/default
> # cd /mnt/mnt/
> # mv boot boot.bak
> # cp -RPp boot.bak boot
> # gpart show /dev/mfid0
> =>         40  31247564720  mfid0  GPT  (15T)
>            40       409600      1  efi  (200M)
>        409640         1024      2  freebsd-boot  (512K)
>        410664          984         - free -  (492K)
>        411648    268435456      3  freebsd-swap  (128G)
>     268847104  30978715648      4  freebsd-zfs  (14T)
>   31247562752         2008         - free -  (1.0M)
> 
> # gpart bootcode -b /mnt/mnt/boot/pmbr -p /boot/gptzfsboot -i 2 mfid0
> partcode written to mfid0p2
> bootcode written to mfid0
> # cd
> # zpool export zroot
> # 
> 
> But can not boot:
> 
> ZFS: i/o error - all block copies unavailable
> ZFS: can't read MOS of pool zroot
> gptzfsboot: failed to mount default pool zroot
> 
> FreeBSD/x86 boot
> 
> Any idea?
> 
> Best regards
> 
> [1] http://ds.truefc.org/~kiri/freebsd/current/zfs/messages
> [2] https://lists.freebsd.org/pipermail/freebsd-questions/2016-February/270505.html
> [3] https://forums.freebsd.org/threads/zfs-i-o-error-all-block-copies-unavailable-invalid-format.55227/#post-312830
> 
> ---
> KIRIYAMA Kazuhiko
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
> 

Since you were about to 'zpool import' on the USB stick, this suggests
the problem is with the boot blocks, not ZFS.

The early boot phase (gptzfsboot) does not read zpool.cache, since that
only lives ON the pool, and the pool has not been imported yet.

Maybe kevans_at_ or imp_at_ who have been working on the boot code have some
insight.

-- 
Allan Jude


Received on Mon Mar 19 2018 - 22:09:33 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:15 UTC