Re: Nasty non-recursive lockmgr panic on softdep only enabled UFS partition when filesystem full

On 4 May 2011 10:42, Garrett Cooper <yanegomi_at_gmail.com> wrote:
> On Tue, May 3, 2011 at 10:59 PM, Kirk McKusick <mckusick_at_mckusick.com> wrote:
>>> Date: Tue, 3 May 2011 22:40:26 -0700
>>> Subject: Nasty non-recursive lockmgr panic on softdep only enabled UFS
>>>  partition when filesystem full
>>> From: Garrett Cooper <yanegomi_at_gmail.com>
>>> To: Jeff Roberson <jeff_at_freebsd.org>,
>>>         Marshall Kirk McKusick <mckusick_at_mckusick.com>
>>> Cc: FreeBSD Current <freebsd-current_at_freebsd.org>
>>>
>>> Hi Jeff and Dr. McKusick,
>>>     Ran into this panic when /usr ran out of space doing a make
>>> universe on amd64/r221219 (it took ~15 minutes for the panic to occur
>>> after the filesystem ran out of space -- wasn't quite sure what it was
>>> doing at the time):
>>>
>>> ...
>>>
>>>     Let me know what other commands you would like for me to run in kgdb.
>>> Thanks,
>>> -Garrett
>>
>> You did not indicate whether you are running an 8.X system or a 9-current
>> system. It would be helpful to know that.
>
> I've actually been running CURRENT for a few years now, but you're right --
> I didn't mention that part.
>
>> Jeff thinks that there may be a potential race in the locking code for
>> softdep_request_cleanup. If so, this patch for 9-current should fix it:
>>
>> Index: ffs_softdep.c
>> ===================================================================
>> --- ffs_softdep.c       (revision 221385)
>> +++ ffs_softdep.c       (working copy)
>> _at__at_ -11380,7 +11380,8 _at__at_
>>                                continue;
>>                        }
>>                        MNT_IUNLOCK(mp);
>> -                       if (vget(lvp, LK_EXCLUSIVE | LK_INTERLOCK, curthread)) {
>> +                       if (vget(lvp, LK_EXCLUSIVE | LK_NOWAIT | LK_INTERLOCK,
>> +                           curthread)) {
>>                                MNT_ILOCK(mp);
>>                                continue;
>>                        }
>>

FYI,
I was playing with head (w/o the above patch) to reproduce the panic and got
this LOR when filesystem was eventually filled.
I'm not sure the patch would fix the panic but I think it should at
least fix the LOR.

kernel: pid 66153 (dd), uid 0 inumber 4 on /mnt: filesystem full
lock order reversal:
 1st 0xfffffe001d7d3310 ufs (ufs) _at_ /usr/src/sys/kern/vfs_vnops.c:614
 2nd 0xffffff807ba8a800 bufwait (bufwait) _at_ /usr/src/sys/kern/vfs_bio.c:2658
 3rd 0xfffffe001ade7588 ufs (ufs) _at_ /usr/src/sys/kern/vfs_subr.c:2126
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff802d9eba = db_trace_self_wrapper+0x2a
kdb_backtrace() at 0xffffffff80475d17 = kdb_backtrace+0x37
_witness_debugger() at 0xffffffff8048b4fe = _witness_debugger+0x2e
witness_checkorder() at 0xffffffff8048c7a7 = witness_checkorder+0x807
__lockmgr_args() at 0xffffffff80427553 = __lockmgr_args+0xd63
ffs_lock() at 0xffffffff806578fc = ffs_lock+0x9c
VOP_LOCK1_APV() at 0xffffffff806f285f = VOP_LOCK1_APV+0xbf
_vn_lock() at 0xffffffff804e87c7 = _vn_lock+0x57
vget() at 0xffffffff804dbb5b = vget+0x7b
softdep_request_cleanup() at 0xffffffff80649f31 = softdep_request_cleanup+0x311
ffs_alloc() at 0xffffffff80630b64 = ffs_alloc+0x134
ffs_balloc_ufs2() at 0xffffffff8063426c = ffs_balloc_ufs2+0x11ac
ffs_write() at 0xffffffff8065889f = ffs_write+0x22f
VOP_WRITE_APV() at 0xffffffff806f33dd = VOP_WRITE_APV+0x14d
vn_write() at 0xffffffff804e9a42 = vn_write+0x2a2
dofilewrite() at 0xffffffff8048df25 = dofilewrite+0x85
kern_writev() at 0xffffffff8048f740 = kern_writev+0x60
write() at 0xffffffff8048f845 = write+0x55
syscallenter() at 0xffffffff80483cbb = syscallenter+0x1cb
syscall() at 0xffffffff806abaf0 = syscall+0x60
Xfast_syscall() at 0xffffffff8069670d = Xfast_syscall+0xdd
--- syscall (4, FreeBSD ELF64, write), rip = 0x8009438fc, rsp =
0x7fffffffda68, rbp = 0xa00000 ---

-- 
wbr,
pluknet