gmirror and/or ufs got into an odd state after a panic, where fsck could not fix errors on the mirror device, but each member filesystem was fine. Even after destroying the mirror, fixing all three member file systems with fsck, and recreating the mirror with just a single member fsck found the same errors on the mirror device. I ended up newfs'ing the mirror and copying all data over from one of the old mirror members (and then reattaching the two other members), and all seems to be OK now, but it's still puzzling to me where those errors originated from. Blow by blow account below. Stefan I'm running my root fs (UFS with softupdates) on a three disk gmirror: # gmirror status Name Status Components mirror/diesel_root COMPLETE ad6p2 ad8p2 ad4p2 mirror/diesel_swap DEGRADED ad6p3 ad8p3 ad4p3 After a panic, fsck could not fix the root fs: Trying to mount root from ufs:/dev/mirror/diesel_root WARNING: / was not properly dismounted /: mount pending error: blocks 52 files 4 Entropy harvesting: interrupts ethernet point_to_point kickstart. /dev/mirror/diesel_root: CANNOT READ BLK: 8388576 /dev/mirror/diesel_root: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. Automatic file system check failed; help! ERRORJOul 11 03:10:47 init: /bin/sh on /etc/rc terminated abnormally, ng to single user mode Enter full pathname of shell or RETURN for /bin/sh: GEOM_MIRROR: Device diesel_root: rebuilding provider ad6p2 finished. GEOM_MIRROR: Device diesel_root: rebuilding provider ad4p2 finished. # fsck -y ** /dev/mirror/diesel_root ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes CANNOT READ BLK: 8388576 UNEXPECTED SOFT UPDATE INCONSISTENCY CONTINUE? yes THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607, ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=424146 OWNER=root MODE=140666 SIZE=0 MTIME=Jul 10 22:00 2009 CLEAR? yes ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks, 1.6% fragmentation) ***** FILE SYSTEM STILL DIRTY ***** ***** FILE SYSTEM WAS MODIFIED ***** ***** PL/: reload pending error: blocks 52 files 4 EASE RERUN FSCK ***** # fsck -y ** /dev/mirror/diesel_root ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes CANNOT READ BLK: 8388576 UNEXPECTED SOFT UPDATE INCONSISTENCY CONTINUE? yes THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607, ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 1a18582 files, 1357018 used, 672013 free (31549 frags, 80058 bloc6% fragmen/: reload pending error: blocks 52 files 4 tation) ***** FILE SYSTEM STILL DIRTY ***** ***** PLEASE RERUN FSCK ***** # dd if=/dev/mirror/diesel_root bs=1m of=/dev/null 4095+1 records in 4095+1 records out 4294966784 bytes transferred in 36.500572 secs (117668479 bytes/sec) However, I could read the whole mirror device with dd without problems. Also note the absence of any i/o errors. So I detached one of the mirrors and ran fsck on the raw partition: # gmirror remove diesel_root ad8p2 # GEOM_MIRROR: Device diesel_root: provider ad8p2 destroyed. # fsck -t ufs -y /dev/ad8p2 ** /dev/ad8p2 ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks, 1.6% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** Odd. So I decided to boot off that partition, and try and fix the filesystem by recreating the mirror based from a single, good partition: # gmirror remove diesel_root ad4p2 # gmirror remove diesel_root ad6p2 # gmirror remove diesel_root ad8p2 # fsck -t ufs -y /dev/ad4p2 # gmirror label diesel_root ad4p2 # fsck -t ufs -y /dev/mirror/diesel_root ** /dev/mirror/diesel_root ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes CANNOT READ BLK: 8388576 UNEXPECTED SOFT UPDATE INCONSISTENCY CONTINUE? yes THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607, ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks, 1.6% fragmentation) ***** FILE SYSTEM MARKED DIRTY ***** ***** PLEASE RERUN FSCK ***** # gmirror status Name Status Components mirror/diesel_swap COMPLETE ad4p3 ad6p3 ad8p3 mirror/diesel_root COMPLETE ad4p2 At which point I decided to newfs the mirror and copy all data over from one of the other (former) mirror members. That turned out to work without problems. -- Stefan Bethke <stb_at_lassitu.de> Fon +49 151 14070811Received on Sat Jul 11 2009 - 10:54:04 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:51 UTC