Nasty non-recursive lockmgr panic on softdep only enabled UFS partition when filesystem full

From: Garrett Cooper <yanegomi_at_gmail.com>
Date: Tue, 3 May 2011 22:40:26 -0700
Hi Jeff and Dr. McKusick,
    Ran into this panic when /usr ran out of space doing a make
universe on amd64/r221219 (it took ~15 minutes for the panic to occur
after the filesystem ran out of space -- wasn't quite sure what it was
doing at the time):

pid 24486 (ld), uid 0 inumber 9993 on /usr: filesystem full
pid 24511 (config), uid 0 inumber 361082 on /usr: filesystem full
pid 24494 (make), uid 0 inumber 1886295 on /usr: filesystem full
panic: __lockmgr_args: recursing on non recursive lockmgr bufwait _at_
/usr/src/sys/ufs/ffs/ffs_softdep.c:11025
(kgdb) #0  doadump () at pcpu.h:224
#1  0xffffffff802af22c in db_fncall (dummy1=Variable "dummy1" is not available.
)
    at /usr/src/sys/ddb/db_command.c:548
#2  0xffffffff802af561 in db_command (last_cmdp=0xffffffff808f93c0,
cmd_table=Variable "cmd_table" is not available.

) at /usr/src/sys/ddb/db_command.c:445
#3  0xffffffff802af7a9 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:498
#4  0xffffffff802b1737 in db_trap (type=Variable "type" is not available.
) at /usr/src/sys/ddb/db_main.c:229
#5  0xffffffff803f7d48 in kdb_trap (type=3, code=0, tf=0xffffff834e4c8ef0)
    at /usr/src/sys/kern/subr_kdb.c:533
#6  0xffffffff80599da5 in trap (frame=0xffffff834e4c8ef0)
    at /usr/src/sys/amd64/amd64/trap.c:590
#7  0xffffffff80584ef3 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#8  0xffffffff803f7baf in kdb_enter (why=0xffffffff806178cf "panic",
    msg=0xa <Address 0xa out of bounds>) at cpufunc.h:63
#9  0xffffffff803c4b6f in panic (fmt=Variable "fmt" is not available.
)
    at /usr/src/sys/kern/kern_shutdown.c:584
#10 0xffffffff803af3ac in __lockmgr_args (lk=0x100, flags=0,
    ilk=0xfffffe00b95766c0, wmesg=Variable "wmesg" is not available.
) at /usr/src/sys/kern/kern_lock.c:720
#11 0xffffffff8054240b in softdep_sync_metadata (vp=0xfffffe017fe5d000)
    at lockmgr.h:97
#12 0xffffffff80548e90 in ffs_syncvnode (vp=0xfffffe017fe5d000,
waitfor=Variable "waitfor" is not available.
)
    at /usr/src/sys/ufs/ffs/ffs_vnops.c:331
#13 0xffffffff8053be23 in softdep_request_cleanup (fs=0xfffffe00086ef000,
    vp=0xfffffe00b95765a0, cred=Variable "cred" is not available.
) at /usr/src/sys/ufs/ffs/ffs_softdep.c:11392
#14 0xffffffff80523895 in ffs_realloccg (ip=0xfffffe00b9285bd0, lbprev=0,
    bprev=10092847, bpref=10723304, osize=2048, nsize=4096, flags=65536,
    cred=0xfffffe026e905a00, bpp=0xffffff834e4c95f0)
    at /usr/src/sys/ufs/ffs/ffs_alloc.c:423
#15 0xffffffff805266de in ffs_balloc_ufs2 (vp=0xfffffe00b95765a0,
startoffset=Variable "startoffset" is not available.

) at /usr/src/sys/ufs/ffs/ffs_balloc.c:699
#16 0xffffffff8054fbfb in ufs_direnter (dvp=0xfffffe00b95765a0,
    tvp=0xfffffe00701c4000, dirp=0xffffff834e4c97b0, cnp=Variable
"cnp" is not available.
)
    at /usr/src/sys/ufs/ufs/ufs_lookup.c:910
#17 0xffffffff80557af8 in ufs_mkdir (ap=0xffffff834e4c9a90)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:1961
#18 0xffffffff805d666b in VOP_MKDIR_APV (vop=0xffffffff808c2a40,
    a=0xffffff834e4c9a90) at vnode_if.c:1534
#19 0xffffffff80457eb8 in kern_mkdirat (td=0xfffffe0149df4000, fd=-100,
    path=0x6096e0 <Address 0x6096e0 out of bounds>, segflg=Variable
"segflg" is not available.
) at vnode_if.h:665
#20 0xffffffff80404cd1 in syscallenter (td=0xfffffe0149df4000,
    sa=0xffffff834e4c9bb0) at /usr/src/sys/kern/subr_trap.c:344
#21 0xffffffff8059996e in syscall (frame=0xffffff834e4c9c50)
    at /usr/src/sys/amd64/amd64/trap.c:910
#22 0xffffffff805851bd in Xfast_syscall ()
    at /usr/src/sys/amd64/amd64/exception.S:384
#23 0x0000000800b3798c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
$ sudo tunefs -p /usr
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       disabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  2048
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

    Let me know what other commands you would like for me to run in kgdb.
Thanks,
-Garrett
Received on Wed May 04 2011 - 03:40:27 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:13 UTC