Stack overrun and double fault in UFS (with backtrace)

I updated one of the SMP package machines to 6.0, and it blew up overnight with this:

Fatal double fault:
eip = 0xc05709c7
esp = 0xe7ec7f94
ebp = 0xe7ec8168
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
KDB: enter: panic

jhb [=BigKnife] suggested the following:

<BigKnife> try this for "fun"
<BigKnife> well, first do a 'show pcpu' to get the value of curthread
<BigKnife> then do
<BigKnife> call db_backtrace(0xc25be960, 0, 0xe7ec8168, 0xc05709c7)
<BigKnife> (that's db_backtrace(curthread, NULL, ebp, eip)
<BigKnife> and see if it can give you a backtrace..
<BigKnife> ugh
<BigKnife> , woops
<BigKnife> add another arg of '-1'
<BigKnife> forgot to specify count of frames to do :-P

db> show pcpu
cpuid        = 0
curthread    = 0xc25be960: pid 51128 "rm"
curpcb       = 0xe7ec9da0
fpcurthread  = none
idlethread   = 0xc224e960: pid 14 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x30
spin locks held:
db> call db_backtrace(0xc25be960, 0, 0xe7ec8168, 0xc05709c7, -1)
buf_splay(170c00,0,0,d6f16740,0) at buf_splay+0x2b
gbincore(c25fee70,170c00,0,c25fee70,0) at gbincore+0x8d
getblk(c25fee70,170c00,0,4000,0) at getblk+0xa0
breadn(c25fee70,170c00,0,4000,0) at breadn+0x31
bread(c25fee70,170c00,0,4000,0) at bread+0x20
ffs_update(ca712630,1,c0519a72,c0716910,0) at ffs_update+0x224
ffs_truncate(ca712630,0,0,c00,0) at ffs_truncate+0xbaf
ufs_inactive(e7ec8520,e7ec8538,c05722e6,e7ec8520,c070c460) at ufs_inactive+0xbc
ufs_vnoperate(e7ec8520,c070c460,ca712630,c25be960,e7ec855c) at ufs_vnoperate+0x13
vput(ca712630,c06fc7d8,ce9df080,ca712630,cd819760) at vput+0x10e
handle_workitem_remove(cd819760,ca712630) at handle_workitem_remove+0x101
process_worklist_item(0,10,0,10,0) at process_worklist_item+0x161
request_cleanup(1,1) at request_cleanup+0x66
inodedep_lookup(c25ad000,19550,1,e7ec85dc,c06fc7d8) at inodedep_lookup+0xb2
softdep_change_linkcnt(c2b05690) at softdep_change_linkcnt+0x25
ufs_inactive(e7ec862c,e7ec8644,c05722e6,e7ec862c,c070c460) at ufs_inactive+0x133
ufs_vnoperate(e7ec862c,c070c460,c4cb8528,c25be960,e7ec8668) at ufs_vnoperate+0x13
vput(c4cb8528,c06fc7d8,cef69100,c4cb8528,c5fcf500) at vput+0x10e
handle_workitem_remove(c5fcf500,c4cb8528) at handle_workitem_remove+0x101
process_worklist_item(0,10,0,1954f,e7ec86c4) at process_worklist_item+0x161
request_cleanup(1,1) at request_cleanup+0x5d
inodedep_lookup(c25ad000,1954f,1,e7ec86e0,c06fc7d8) at inodedep_lookup+0xb2
softdep_change_linkcnt(cacb4ec4) at softdep_change_linkcnt+0x25
ufs_inactive(e7ec8730,e7ec8748,c05722e6,e7ec8730,c070c460) at ufs_inactive+0x133
ufs_vnoperate(e7ec8730,c070c460,ca8aae70,c25be960,e7ec876c) at ufs_vnoperate+0x13
vput(ca8aae70,c06fc7d8,cfbf9e80,ca8aae70,d181d940) at vput+0x10e
handle_workitem_remove(d181d940,ca8aae70) at handle_workitem_remove+0x101
process_worklist_item(0,10,0,10,0) at process_worklist_item+0x161
request_cleanup(1,1) at request_cleanup+0x66
inodedep_lookup(c25ad000,1954e,1,e7ec87ec,c06fc7d8) at inodedep_lookup+0xb2
softdep_change_linkcnt(c5a8c71c) at softdep_change_linkcnt+0x25
ufs_inactive(e7ec883c,e7ec8854,c05722e6,e7ec883c,c070c460) at ufs_inactive+0x133
ufs_vnoperate(e7ec883c,c070c460,cba16630,c25be960,e7ec8878) at ufs_vnoperate+0x13
vput(cba16630,c06fc7d8,cdf36a80,cba16630,cfac2060) at vput+0x10e
handle_workitem_remove(cfac2060,cba16630) at handle_workitem_remove+0x101
process_worklist_item(0,10,0,10,0) at process_worklist_item+0x161

[...about another dozen iterations of this...]

request_cleanup(1,1) at request_cleanup+0x66
inodedep_lookup(c25ad000,19534,1,e7ec9ab4,c06fc7d8) at inodedep_lookup+0xb2
softdep_change_linkcnt(c4424c94) at softdep_change_linkcnt+0x25
ufs_inactive(e7ec9b04,e7ec9b1c,c05722e6,e7ec9b04,c070c460) at ufs_inactive+0x133
ufs_vnoperate(e7ec9b04,c070c460,c49df840,c25be960,e7ec9b40) at ufs_vnoperate+0x13
vput(c49df840,c06fc7d8,cd388c80,c49df840,cfe438c0) at vput+0x10e
handle_workitem_remove(cfe438c0,c49df840) at handle_workitem_remove+0x101
process_worklist_item(0,10,0,c646f604,e7ec9ba0) at process_worklist_item+0x161
request_cleanup(2,0) at request_cleanup+0x5d
newdirrem(d6e61488,c646f604,c5c58dac,0,e7ec9bbc) at newdirrem+0x3b
softdep_setup_remove(d6e61488,c646f604,c5c58dac,0,c5c58dac) at softdep_setup_remove+0x1c
ufs_dirremove(c4d78d68,c5c58dac,800c,0,e7ec9c34) at ufs_dirremove+0x135
ufs_remove(e7ec9c38,e7ec9cc4,c0576b30,e7ec9c38,c2558c00) at ufs_remove+0x4b
ufs_vnoperate(e7ec9c38) at ufs_vnoperate+0x13
kern_unlink(c25be960,8056aa8,0,c0716540,0) at kern_unlink+0x144
unlink(c25be960,e7ec9d14,1,7d,292) at unlink+0x29
syscall(805002f,804002f,bfbf002f,1,804c000) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (10, FreeBSD ELF32, unlink), eip = 0x280c258f, esp = 0xbfbfe82c, ebp = 0xbfbfe858 ---

<BigKnife> looks like infinite recursion until it blew teh stack

Kris

--
In God we Trust -- all others must submit an X.509 certificate.
    -- Charles Forsythe <forsythe_at_alum.mit.edu>