Re: bgfsck strikes again

> Date: Mon, 8 Oct 2007 08:14:42 +1000
> From: Peter Jeremy <peterjeremy_at_optushome.com.au>
> Sender: owner-freebsd-current_at_freebsd.org
> 
> FreeBSD server.vk2pj.dyndns.org 7.0-CURRENT FreeBSD 7.0-CURRENT #9: Wed Oct  3 12:00:03 EST 2007     root_at_server.vk2pj.dyndns.org:/var/obj/k7/usr/src/sys/server  i386
> 
> A short power glitch last night reset my main server at 1929.  It had
> been reasonably idle and I wanted it back so I let it reboot normally
> and relied on bgfsck to cleanup.  bgfsck cleared 13 unreferenced files
> in /var and /usr and all seemed OK.
> 
> At 0247, it panic'd "panic: getblk: bp 0xcceb52cc not locked" with
> the following backtrace:
> #8  0xc055c945 in panic (fmt=0xc073040f "getblk: bp %p not locked") at /usr/src/sys/kern/kern_shutdown.c:547
> #9  0xc05c4843 in getblk (vp=0xc2c9ccc0, blkno=0x18b88dc0, size=0x4000, slpflag=Variable "slpflag" is not available.) at /usr/src/sys/kern/vfs_bio.c:2666
> #10 0xc05c4e24 in breadn (vp=0xc2c9ccc0, blkno=Unhandled dwarf expression opcode 0x93
> ) at /usr/src/sys/kern/vfs_bio.c:786
> #11 0xc05c4fac in bread (vp=0xc2c9ccc0, blkno=Unhandled dwarf expression opcode 0x93
> ) at /usr/src/sys/kern/vfs_bio.c:734
> #12 0xc0664d7a in ffs_update (vp=0xc31f3660, waitfor=0x0) at /usr/src/sys/ufs/ffs/ffs_inode.c:103
> #13 0xc06845ea in ufs_inactive (ap=0xd62a9adc) at /usr/src/sys/ufs/ufs/ufs_inode.c:159
> #14 0xc06f4d15 in VOP_INACTIVE_APV (vop=0xc07828a0, a=0xd62a9adc) at vnode_if.c:1513
> #15 0xc05d4401 in vinactive (vp=0xc31f3660, td=0xc2ffd210) at vnode_if.h:796
> #16 0xc05d7d66 in vput (vp=0xc31f3660) at /usr/src/sys/kern/vfs_subr.c:2197
> #17 0xc05e2c1b in vn_close (vp=0xc31f3660, flags=0xa, file_cred=0xc301a400, td=0xc2ffd210)
>     at /usr/src/sys/kern/vfs_vnops.c:295
> #18 0xc05e2d17 in vn_closefile (fp=0xc308e8b8, td=0xc2ffd210) at /usr/src/sys/kern/vfs_vnops.c:868
> #19 0xc05316d2 in fdrop (fp=0xc308e8b8, td=0xc2ffd210) at file.h:297
> #20 0xc0532c03 in closef (fp=0xc308e8b8, td=0xc2ffd210) at /usr/src/sys/kern/kern_descrip.c:1958
> #21 0xc0532f91 in kern_close (td=0xc2ffd210, fd=0x6) at /usr/src/sys/kern/kern_descrip.c:1054
> #22 0xc053302a in close (td=0xc2ffd210, uap=0xd62a9cfc) at /usr/src/sys/kern/kern_descrip.c:1006
> #23 0xc06dfffa in syscall (frame=0xd62a9d38) at /usr/src/sys/i386/i386/trap.c:1008
> #24 0xc06cdbb0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:196
> 
> This time, I manually fsck'd it and was greeted with several thousand
> errors.  Looking through them, it looks very much like UFS had just
> stopped garbage collecting unreferenced files so deleted files were
> just left lying around.
> 
> Is it worth trying to investigate this further or should I just put it
> down to "don't use bgfsck"?

Silly question, but is the disk doing write caching? If so, it is
possible that the data in the cache was lost, leaving inconsistency in
the metadata. It's not a trivial issue to deal with, especially on disks
that lie about whether data has actually been written to disk.
-- 
R. Kevin Oberman, Network Engineer
Energy Sciences Network (ESnet)
Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab)
E-mail: oberman_at_es.net			Phone: +1 510 486-8634
Key fingerprint:059B 2DDF 031C 9BA3 14A4  EADA 927D EBB3 987B 3751