ZFS i/o error in recent 12.0

From: KIRIYAMA Kazuhiko <kiri_at_kx.openedu.org>
Date: Tue, 20 Mar 2018 08:00:04 +0900
Hi,

I've been encountered suddenly death in ZFS full volume
machine(r330434) about 10 days after installation[1]:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
gptzfsboot: failed to mount default pool zroot

FreeBSD/x86 boot
ZFS: i/o error - all block copies unavailable
ZFS: can't find dataset u
Default: zroot/<0x0>:
boot: 

Partition is bellow:

 # gpart show /dev/mfid0
=>         40  31247564720  mfid0  GPT  (15T)
           40       409600      1  efi  (200M)
       409640         1024      2  freebsd-boot  (512K)
       410664          984         - free -  (492K)
       411648    268435456      3  freebsd-swap  (128G)
    268847104  30978715648      4  freebsd-zfs  (14T)
  31247562752         2008         - free -  (1.0M)

# 

But nothing had beed happend in old current ZFS full volume
machine(r327038M). According to [2] the reason is boot/zfs/zpool.cache
inconsistent. I've tried to cope with this by repairing
/boot [3] from rescue bootable USB as follows:

# kldload zfs
# zpool import 
   pool: zroot
     id: 17762298124265859537
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        zroot       ONLINE
          mfid0p4   ONLINE
# zpool import -fR /mnt zroot
# df -h
Filesystem                  Size    Used   Avail Capacity  Mounted on
/dev/da0p2                   14G    1.6G     11G    13%    /
devfs                       1.0K    1.0K      0B   100%    /dev
zroot/.dake                  14T     18M     14T     0%    /mnt/.dake
zroot/ds                     14T     96K     14T     0%    /mnt/ds
zroot/ds/backup              14T     88K     14T     0%    /mnt/ds/backup
zroot/ds/backup/kazu.pis     14T     31G     14T     0%    /mnt/ds/backup/kazu.pis
zroot/ds/distfiles           14T    7.9M     14T     0%    /mnt/ds/distfiles
zroot/ds/obj                 14T     10G     14T     0%    /mnt/ds/obj
zroot/ds/packages            14T    4.0M     14T     0%    /mnt/ds/packages
zroot/ds/ports               14T    1.3G     14T     0%    /mnt/ds/ports
zroot/ds/src                 14T    2.6G     14T     0%    /mnt/ds/src
zroot/tmp                    14T     88K     14T     0%    /mnt/tmp
zroot/usr/home               14T    136K     14T     0%    /mnt/usr/home
zroot/usr/local              14T     10M     14T     0%    /mnt/usr/local
zroot/var/audit              14T     88K     14T     0%    /mnt/var/audit
zroot/var/crash              14T     88K     14T     0%    /mnt/var/crash
zroot/var/log                14T    388K     14T     0%    /mnt/var/log
zroot/var/mail               14T     92K     14T     0%    /mnt/var/mail
zroot/var/ports              14T     11M     14T     0%    /mnt/var/ports
zroot/var/tmp                14T    6.0M     14T     0%    /mnt/var/tmp
zroot/vm                     14T    2.8G     14T     0%    /mnt/vm
zroot/vm/tbedfc              14T    1.6G     14T     0%    /mnt/vm/tbedfc
zroot                        14T     88K     14T     0%    /mnt/zroot
# zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
zroot                     51.1G  13.9T    88K  /mnt/zroot
zroot/.dake               18.3M  13.9T  18.3M  /mnt/.dake
zroot/ROOT                1.71G  13.9T    88K  none
zroot/ROOT/default        1.71G  13.9T  1.71G  /mnt/mnt
zroot/ds                  45.0G  13.9T    96K  /mnt/ds
zroot/ds/backup           30.8G  13.9T    88K  /mnt/ds/backup
zroot/ds/backup/kazu.pis  30.8G  13.9T  30.8G  /mnt/ds/backup/kazu.pis
zroot/ds/distfiles        7.88M  13.9T  7.88M  /mnt/ds/distfiles
zroot/ds/obj              10.4G  13.9T  10.4G  /mnt/ds/obj
zroot/ds/packages         4.02M  13.9T  4.02M  /mnt/ds/packages
zroot/ds/ports            1.26G  13.9T  1.26G  /mnt/ds/ports
zroot/ds/src              2.56G  13.9T  2.56G  /mnt/ds/src
zroot/tmp                   88K  13.9T    88K  /mnt/tmp
zroot/usr                 10.4M  13.9T    88K  /mnt/usr
zroot/usr/home             136K  13.9T   136K  /mnt/usr/home
zroot/usr/local           10.2M  13.9T  10.2M  /mnt/usr/local
zroot/var                 17.4M  13.9T    88K  /mnt/var
zroot/var/audit             88K  13.9T    88K  /mnt/var/audit
zroot/var/crash             88K  13.9T    88K  /mnt/var/crash
zroot/var/log              388K  13.9T   388K  /mnt/var/log
zroot/var/mail              92K  13.9T    92K  /mnt/var/mail
zroot/var/ports           10.7M  13.9T  10.7M  /mnt/var/ports
zroot/var/tmp             5.98M  13.9T  5.98M  /mnt/var/tmp
zroot/vm                  4.33G  13.9T  2.75G  /mnt/vm
zroot/vm/tbedfc           1.58G  13.9T  1.58G  /mnt/vm/tbedfc
# zfs mount zroot/ROOT/default
# cd /mnt/mnt/
# mv boot boot.bak
# cp -RPp boot.bak boot
# gpart show /dev/mfid0
=>         40  31247564720  mfid0  GPT  (15T)
           40       409600      1  efi  (200M)
       409640         1024      2  freebsd-boot  (512K)
       410664          984         - free -  (492K)
       411648    268435456      3  freebsd-swap  (128G)
    268847104  30978715648      4  freebsd-zfs  (14T)
  31247562752         2008         - free -  (1.0M)

# gpart bootcode -b /mnt/mnt/boot/pmbr -p /boot/gptzfsboot -i 2 mfid0
partcode written to mfid0p2
bootcode written to mfid0
# cd
# zpool export zroot
# 

But can not boot:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
gptzfsboot: failed to mount default pool zroot

FreeBSD/x86 boot

Any idea?

Best regards

[1] http://ds.truefc.org/~kiri/freebsd/current/zfs/messages
[2] https://lists.freebsd.org/pipermail/freebsd-questions/2016-February/270505.html
[3] https://forums.freebsd.org/threads/zfs-i-o-error-all-block-copies-unavailable-invalid-format.55227/#post-312830

---
KIRIYAMA Kazuhiko
Received on Mon Mar 19 2018 - 22:00:09 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:15 UTC