Background info: One of the file system inconsistencies that is "expected" by soft updates and that can be repaired by background fsck is a link count that is too high. Historically, rmdir would fail on directories with a link count greater than 2 even if they were empty. This was changed a few months ago in ufs_vnops.c 1.269. It appears that there is some sort of interaction between soft updates and background fsck that results in the link count of the parent of one of these directories being double decremented, resulting in the file system being put into an invalid state. The following transcript shows the link count of a leaf directory (/mnt/tmp/a/b) being adjusted upwards, the file system being mounted, the leaf directory being removed, the file system unmounted, and then being successfully repaired by a foreground fsck. # fsck -fp /dev/da0s2a /dev/da0s2a: 374158 files, 2551020 used, 2894085 free (25677 frags, 358551 block s, 0.5% fragmentation) # mount /dev/da0s2a /mnt # mkdir /mnt/tmp/a /mnt/tmp/a/b # ls -lid /mnt/tmp/a /mnt/tmp/b ls: /mnt/tmp/b: No such file or directory 320920 drwxr-xr-x 3 root wheel 512 Sep 23 13:31 /mnt/tmp/a # ls -lid /mnt/tmp/a /mnt/tmp/a/b 320920 drwxr-xr-x 3 root wheel 512 Sep 23 13:31 /mnt/tmp/a 320921 drwxr-xr-x 2 root wheel 512 Sep 23 13:31 /mnt/tmp/a/b # umount /mnt # fsdb /dev/da0s2a ** /dev/da0s2a Editing file system `/dev/da0s2a' Last Mounted on /mnt current inode: directory I=2 MODE=40755 SIZE=512 MTIME=Sep 20 11:54:46 2005 [0 nsec] CTIME=Sep 20 11:54:46 2005 [0 nsec] ATIME=Sep 22 21:36:58 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245 fsdb (inum: 2)> inode 320921 current inode: directory I=320921 MODE=40755 SIZE=512 MTIME=Sep 23 13:31:30 2005 [0 nsec] CTIME=Sep 23 13:31:30 2005 [0 nsec] ATIME=Sep 23 13:31:30 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=2 FLAGS=0 BLKCNT=4 GEN=1292b1d5 fsdb (inum: 320921)> uplink inode 320921 link count now 3 fsdb (inum: 320921)> q ***** FILE SYSTEM STILL DIRTY ***** *** FILE SYSTEM MARKED DIRTY *** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE *** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload # mount /dev/da0s2a /mnt WARNING: /mnt was not properly dismounted # ls -lid /mnt/tmp/a /mnt/tmp/a/b 320920 drwxr-xr-x 3 root wheel 512 Sep 23 13:31 /mnt/tmp/a 320921 drwxr-xr-x 3 root wheel 512 Sep 23 13:31 /mnt/tmp/a/b # rmdir /mnt/tmp/a/b # umount /mnt # fsdb /dev/da0s2a ** /dev/da0s2a Editing file system `/dev/da0s2a' Last Mounted on /mnt current inode: directory I=2 MODE=40755 SIZE=512 MTIME=Sep 20 11:54:46 2005 [0 nsec] CTIME=Sep 20 11:54:46 2005 [0 nsec] ATIME=Sep 22 21:36:58 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245 fsdb (inum: 2)> inode 320920 current inode: directory I=320920 MODE=40755 SIZE=512 MTIME=Sep 23 13:33:56 2005 [0 nsec] CTIME=Sep 23 13:33:56 2005 [0 nsec] ATIME=Sep 23 13:31:30 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=3 FLAGS=0 BLKCNT=4 GEN=33d3740d fsdb (inum: 320920)> inode 320921 current inode: directory I=320921 MODE=40755 SIZE=0 MTIME=Sep 23 13:34:05 2005 [0 nsec] CTIME=Sep 23 13:34:05 2005 [0 nsec] ATIME=Sep 23 13:33:56 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=1 FLAGS=0 BLKCNT=0 GEN=1292b1d5 fsdb (inum: 320921)> q ***** FILE SYSTEM STILL DIRTY ***** *** FILE SYSTEM MARKED DIRTY *** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE *** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload # fsck -fp /dev/da0s2a /dev/da0s2a: LINK COUNT DIR I=320920 OWNER=root MODE=40755 /dev/da0s2a: SIZE=512 MTIME=Sep 23 13:33 2005 COUNT 3 SHOULD BE 2 (ADJUSTED) /dev/da0s2a: ZERO LENGTH DIR I=320921 OWNER=root MODE=40755 /dev/da0s2a: SIZE=0 MTIME=Sep 23 13:34 2005 (CLEARED) /dev/da0s2a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/da0s2a: SUMMARY INFORMATION BAD (SALVAGED) /dev/da0s2a: BLK(S) MISSING IN BIT MAPS (SALVAGED) /dev/da0s2a: 374159 files, 2551021 used, 2894084 free (25676 frags, 358551 block s, 0.5% fragmentation) # fsck -fp /dev/da0s2a /dev/da0s2a: 374159 files, 2551021 used, 2894084 free (25676 frags, 358551 block s, 0.5% fragmentation) The following transcript demonstrates what happens if a background fsck is run after the leaf directory is removed. What is interesting is that after the directory the leaf directory has been removed, the effective link count of the parent directory (displayed by ls) has been decremented from 3 to 2, whereas the on-disk link count shown by fsdb is still 3. The background fsck appears to detect the link count as 3, and executes the sysctl call to decrement the link count, causing both the effective and actual link counts to be decremented to 1. My suspicion is that the physical update of the parent directory's link count after the rmdir of the leaf directory has been deferred until the leaf directory's inode is zeroed, which turns out to be an indefinite wait because the inode doesn't get zeroed until fsck is run. # fsck -fp /dev/da0s2a /dev/da0s2a: 374158 files, 2551020 used, 2894085 free (25677 frags, 358551 block s, 0.5% fragmentation) # mount /dev/da0s2a /mnt # mkdir /mnt/tmp/a /mnt/tmp/a/b # ls -lid /mnt/tmp/a /mnt/tmp/a/b 320920 drwxr-xr-x 3 root wheel 512 Sep 23 13:58 /mnt/tmp/a 320921 drwxr-xr-x 2 root wheel 512 Sep 23 13:58 /mnt/tmp/a/b # umount /mnt # fsdb /dev/da0s2a ** /dev/da0s2a Editing file system `/dev/da0s2a' Last Mounted on /mnt current inode: directory I=2 MODE=40755 SIZE=512 MTIME=Sep 20 11:54:46 2005 [0 nsec] CTIME=Sep 20 11:54:46 2005 [0 nsec] ATIME=Sep 22 21:36:58 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245 fsdb (inum: 2)> inode 320921 current inode: directory I=320921 MODE=40755 SIZE=512 MTIME=Sep 23 13:58:46 2005 [0 nsec] CTIME=Sep 23 13:58:46 2005 [0 nsec] ATIME=Sep 23 13:58:46 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=2 FLAGS=0 BLKCNT=4 GEN=10aac654 fsdb (inum: 320921)> uplink inode 320921 link count now 3 fsdb (inum: 320921)> q ***** FILE SYSTEM STILL DIRTY ***** *** FILE SYSTEM MARKED DIRTY *** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE *** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload # mount /dev/da0s2a /mnt WARNING: /mnt was not properly dismounted # ls -lid /mnt/tmp/a /mnt/tmp/a/b 320920 drwxr-xr-x 3 root wheel 512 Sep 23 13:58 /mnt/tmp/a 320921 drwxr-xr-x 3 root wheel 512 Sep 23 13:58 /mnt/tmp/a/b # rmdir /mnt/tmp/a/b # sync # sleep 60 # ls -lid /mnt/tmp/a /mnt/tmp/a/b ls: /mnt/tmp/a/b: No such file or directory 320920 drwxr-xr-x 2 root wheel 512 Sep 23 14:00 /mnt/tmp/a # fsdb /dev/da0s2a ** /dev/da0s2a (NO WRITE) Editing file system `/dev/da0s2a' Last Mounted on /mnt current inode: directory I=2 MODE=40755 SIZE=512 MTIME=Sep 20 11:54:46 2005 [0 nsec] CTIME=Sep 20 11:54:46 2005 [0 nsec] ATIME=Sep 22 21:36:58 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245 fsdb (inum: 2)> inode 320920 current inode: directory I=320920 MODE=40755 SIZE=512 MTIME=Sep 23 14:00:00 2005 [0 nsec] CTIME=Sep 23 14:00:00 2005 [0 nsec] ATIME=Sep 23 13:58:46 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=3 FLAGS=0 BLKCNT=4 GEN=33d3740f fsdb (inum: 320920)> q SETTING DIRTY FLAG IN READ_ONLY MODE UNEXPECTED SOFT UPDATE INCONSISTENCY *** FILE SYSTEM MARKED DIRTY *** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE *** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload # fsck -B -p /dev/da0s2a /dev/da0s2a: LINK COUNT DIR I=320920 OWNER=root MODE=40755 /dev/da0s2a: SIZE=512 MTIME=Sep 23 14:00 2005 COUNT 3 SHOULD BE 2 (ADJUSTED) /dev/da0s2a: ZERO LENGTH DIR I=320921 OWNER=root MODE=40755 /dev/da0s2a: SIZE=0 MTIME=Sep 23 14:00 2005 (CLEARED) /dev/da0s2a: Reclaimed: 0 directories, 1 files, 0 fragments /dev/da0s2a: 374160 files, 2551021 used, 2894084 free (25676 frags, 358551 block s, 0.5% fragmentation) # ls -lid /mnt/tmp/a 320920 drwxr-xr-x 1 root wheel 512 Sep 23 14:00 /mnt/tmp/a # fsdb /dev/da0s2a ** /dev/da0s2a (NO WRITE) Editing file system `/dev/da0s2a' Last Mounted on /mnt current inode: directory I=2 MODE=40755 SIZE=512 MTIME=Sep 20 11:54:46 2005 [0 nsec] CTIME=Sep 20 11:54:46 2005 [0 nsec] ATIME=Sep 22 21:36:58 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=8 FLAGS=0 BLKCNT=4 GEN=53557245 fsdb (inum: 2)> inode 320920 current inode: directory I=320920 MODE=40755 SIZE=512 MTIME=Sep 23 14:00:00 2005 [0 nsec] CTIME=Sep 23 14:05:21 2005 [0 nsec] ATIME=Sep 23 13:58:46 2005 [0 nsec] OWNER=root GRP=wheel LINKCNT=1 FLAGS=0 BLKCNT=4 GEN=33d3740f fsdb (inum: 320920)> q SETTING DIRTY FLAG IN READ_ONLY MODE UNEXPECTED SOFT UPDATE INCONSISTENCY *** FILE SYSTEM MARKED DIRTY *** BE SURE TO RUN FSCK TO CLEAN UP ANY DAMAGE *** IF IT WAS MOUNTED, RE-MOUNT WITH -u -o reload # umount /mnt # fsck -fp /dev/da0s2a /dev/da0s2a: LINK COUNT DIR I=320920 OWNER=root MODE=40755 /dev/da0s2a: SIZE=512 MTIME=Sep 23 14:00 2005 COUNT 1 SHOULD BE 2 /dev/da0s2a: LINK COUNT INCREASING /dev/da0s2a: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.Received on Fri Sep 23 2005 - 19:47:16 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:44 UTC