Re: panic: lockmgr still held [tmpfs] [vm_map_remove()->vdropl()] (r262186: Thu Feb 20)

On Monday, March 03, 2014 9:49:39 pm Bryan Drewery wrote:
> On 3/3/2014 12:06 PM, John Baldwin wrote:
> > On Sunday, March 02, 2014 10:58:45 am Bryan Drewery wrote:
> >> On 2/28/2014 3:18 PM, John Baldwin wrote:
> >>> On Friday, February 28, 2014 9:18:51 am Bryan Drewery wrote:
> >>>> While using poudriere:
> >>>>
> >>>>> Unread portion of the kernel message buffer:
> >>>>> panic: lockmgr still held
> >>>>> cpuid = 12
> >>>>> KDB: stack backtrace:
> >>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
0xfffffe124804f7a0
> >>>>> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe124804f850
> >>>>> vpanic() at vpanic+0x126/frame 0xfffffe124804f890
> >>>>> kassert_panic() at kassert_panic+0x139/frame 0xfffffe124804f900
> >>>>> lockdestroy() at lockdestroy+0x3b/frame 0xfffffe124804f920
> >>>>> vdropl() at vdropl+0x1c8/frame 0xfffffe124804f960
> >>>>> vm_object_deallocate() at vm_object_deallocate+0x10b/frame 
0xfffffe124804f9c0
> >>>>> vm_map_process_deferred() at vm_map_process_deferred+0x89/frame 
0xfffffe124804f9f0
> >>>>> vm_map_remove() at vm_map_remove+0xc8/frame 0xfffffe124804fa20
> >>>>> vmspace_exit() at vmspace_exit+0xc9/frame 0xfffffe124804fa60
> >>>>> exit1() at exit1+0x541/frame 0xfffffe124804fad0
> >>>>> sys_sys_exit() at sys_sys_exit+0xe/frame 0xfffffe124804fae0
> >>>>> ia32_syscall() at ia32_syscall+0x270/frame 0xfffffe124804fbf0
> >>>>> Xint0x80_syscall() at Xint0x80_syscall+0x95/frame 0xfffffe124804fbf0
> >>>>> --- syscall (1, FreeBSD ELF32, sys_sys_exit), rip = 0x281014df, rsp = 
0xffffc45c, rbp = 0xffffc468 ---
> >>>>
> >>>>> #4  0xffffffff808c00db in lockdestroy (lk=0xfffff80a88a285f0) at 
/usr/src/sys/kern/kern_lock.c:440
> >>>>> 440             KASSERT(lk->lk_lock == LK_UNLOCKED, ("lockmgr still 
held"));
> >>>>> (kgdb) print *lk
> >>>>> $1 = {lock_object = {lo_name = 0xffffffff8201a1bd "tmpfs", lo_flags = 
116588552, lo_data = 0, lo_witness = 0xfffffe00006fec00}, lk_lock = 
> >>> 18446735288132049184, lk_exslpfail = 0,
> >>>>>   lk_timo = 51, lk_pri = 96}
> >>>
> >>> Can you please grab people.freebsd.org/~jhb/gdb/*
> >>>
> >>> and then do 'cd /path/to/files', 'source gdb6', 'frame 4', 
'lockmgr_owner lk'?
> >>>
> >>
> >> (kgdb) lockmgr_owner lk
> >> td: 0xfffff80272c61920
> >> pid: 55040, p_comm: testprog
> > 
> > Can you get a stack trace of that program?  ('proc 55040', 'bt')
> > 
> 
> It's the same as original:

So I'm not sure how to fix this.  The crash is in this code in 
vm_object_deallocate():

			if (object->type == OBJT_SWAP &&
			    (object->flags & OBJ_TMPFS) != 0) {
				vp = object->un_pager.swp.swp_tmpfs;
				vhold(vp);
				VM_OBJECT_WUNLOCK(object);
				vn_lock(vp, LK_EXCLUSIVE | LK_RETRY);
				vdrop(vp);
				VM_OBJECT_WLOCK(object);
				if (object->type == OBJT_DEAD ||
				    object->ref_count != 1) {
					VM_OBJECT_WUNLOCK(object);
					VOP_UNLOCK(vp, 0);
					return;
				}
				if ((object->flags & OBJ_TMPFS) != 0)
					VOP_UNSET_TEXT(vp);
				VOP_UNLOCK(vp, 0);
			}

The vdrop() is dropping the count to zero and trying to free the vnode.  The 
real problem I think is that swp_tmpfs doesn't have an implicit vhold() on the 
vnode, so in this case, the code is doing a vhold/vn_lock/vdrop of an already-
free vnode.  For OBJT_VNODE objects, the reference from the object back to the 
vnode holds a vref() that gets released by a vput() in 
vm_object_vndeallocate().

One fix might be to chagne smp_tmpfs to hold a vhold reference.  This is 
untested but might work (but I'm also not sure that this is the right thing in 
that I don't know what other effects it might have).  Can you reproduce this 
easily?

Index: fs/tmpfs/tmpfs_subr.c
===================================================================
--- fs/tmpfs/tmpfs_subr.c	(revision 262711)
+++ fs/tmpfs/tmpfs_subr.c	(working copy)
_at__at_ -440,10 +440,11 _at__at_
 
 	VM_OBJECT_WLOCK(obj);
 	VI_LOCK(vp);
+	KASSERT(obj->un_pager.swp.swp_tmpfs == vp, "vp mismatch");
 	vm_object_clear_flag(obj, OBJ_TMPFS);
 	obj->un_pager.swp.swp_tmpfs = NULL;
-	VI_UNLOCK(vp);
 	VM_OBJECT_WUNLOCK(obj);
+	vdropl(vp);
 }
 
 /*
_at__at_ -578,6 +579,7 _at__at_
 		VI_LOCK(vp);
 		KASSERT(vp->v_object == NULL, ("Not NULL v_object in tmpfs"));
 		vp->v_object = object;
+		vholdl(vp);
 		object->un_pager.swp.swp_tmpfs = vp;
 		vm_object_set_flag(object, OBJ_TMPFS);
 		VI_UNLOCK(vp);
Index: vm/vm_object.c
===================================================================
--- vm/vm_object.c	(revision 262711)
+++ vm/vm_object.c	(working copy)
_at__at_ -547,6 +547,7 _at__at_
 				if ((object->flags & OBJ_TMPFS) != 0)
 					VOP_UNSET_TEXT(vp);
 				VOP_UNLOCK(vp, 0);
+				vdrop(vp);
 			}
 			if (object->shadow_count == 0 &&
 			    object->handle == NULL &&

-- 
John Baldwin