Unrecoverable UFS error but only with gmirror

From: Stefan Bethke <stb_at_lassitu.de>
Date: Sat, 11 Jul 2009 14:54:01 +0200
gmirror and/or ufs got into an odd state after a panic, where fsck  
could not fix errors on the mirror device, but each member filesystem  
was fine.  Even after destroying the mirror, fixing all three member  
file systems with fsck, and recreating the mirror with just a single  
member fsck found the same errors on the mirror device.

I ended up newfs'ing the mirror and copying all data over from one of  
the old mirror members (and then reattaching the two other members),  
and all seems to be OK now, but it's still puzzling to me where those  
errors originated from.

Blow by blow account below.

Stefan


I'm running my root fs (UFS with softupdates) on a three disk gmirror:

# gmirror status
               Name    Status  Components
mirror/diesel_root  COMPLETE  ad6p2
                               ad8p2
                               ad4p2
mirror/diesel_swap  DEGRADED  ad6p3
                               ad8p3
                               ad4p3

After a panic, fsck could not fix the root fs:

Trying to mount root from ufs:/dev/mirror/diesel_root
WARNING: / was not properly dismounted
/: mount pending error: blocks 52 files 4
Entropy harvesting: interrupts ethernet point_to_point kickstart.
/dev/mirror/diesel_root: CANNOT READ BLK: 8388576
/dev/mirror/diesel_root: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN  
fsck MANUALLY.
Automatic file system check failed; help!
ERRORJOul 11 03:10:47 init: /bin/sh on /etc/rc terminated abnormally,  
ng to single user mode
Enter full pathname of shell or RETURN for /bin/sh: GEOM_MIRROR:  
Device diesel_root: rebuilding provider ad6p2 finished.
GEOM_MIRROR: Device diesel_root: rebuilding provider ad4p2 finished.

# fsck -y
** /dev/mirror/diesel_root
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes

CANNOT READ BLK: 8388576
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607,
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=424146  OWNER=root MODE=140666
SIZE=0 MTIME=Jul 10 22:00 2009
CLEAR? yes

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes

SUMMARY INFORMATION BAD
SALVAGE? yes

BLK(S) MISSING IN BIT MAPS
SALVAGE? yes

118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks,  
1.6% fragmentation)

***** FILE SYSTEM STILL DIRTY *****

***** FILE SYSTEM WAS MODIFIED *****

***** PL/: reload pending error: blocks 52 files 4
EASE RERUN FSCK *****
# fsck -y
** /dev/mirror/diesel_root
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes

CANNOT READ BLK: 8388576
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607,
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1a18582 files, 1357018 used, 672013 free (31549 frags, 80058 bloc6%  
fragmen/: reload pending error: blocks 52 files 4
tation)

***** FILE SYSTEM STILL DIRTY *****

***** PLEASE RERUN FSCK *****
# dd if=/dev/mirror/diesel_root bs=1m of=/dev/null
4095+1 records in
4095+1 records out
4294966784 bytes transferred in 36.500572 secs (117668479 bytes/sec)


However, I could read the whole mirror device with dd without  
problems. Also note the absence of any i/o errors.


So I detached one of the mirrors and ran fsck on the raw partition:
# gmirror remove diesel_root ad8p2
# GEOM_MIRROR: Device diesel_root: provider ad8p2 destroyed.
# fsck -t ufs -y /dev/ad8p2
** /dev/ad8p2
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks,  
1.6% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****


Odd. So I decided to boot off that partition, and try and fix the  
filesystem by recreating the mirror based from a single, good partition:
# gmirror remove diesel_root ad4p2
# gmirror remove diesel_root ad6p2
# gmirror remove diesel_root ad8p2
# fsck -t ufs -y /dev/ad4p2
# gmirror label diesel_root ad4p2
# fsck -t ufs -y /dev/mirror/diesel_root
** /dev/mirror/diesel_root
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes

CANNOT READ BLK: 8388576
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 8388607,
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
118582 files, 1357018 used, 672013 free (31549 frags, 80058 blocks,  
1.6% fragmentation)

***** FILE SYSTEM MARKED DIRTY *****

***** PLEASE RERUN FSCK *****
# gmirror status
               Name    Status  Components
mirror/diesel_swap  COMPLETE  ad4p3
                               ad6p3
                               ad8p3
mirror/diesel_root  COMPLETE  ad4p2


At which point I decided to newfs the mirror and copy all data over  
from one of the other (former) mirror members.  That turned out to  
work without problems.


-- 
Stefan Bethke <stb_at_lassitu.de>   Fon +49 151 14070811
Received on Sat Jul 11 2009 - 10:54:04 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:51 UTC