Re: LOR in vnode interlock and system map

On Mon, May 29, 2006 at 12:09:24AM +0200, V??clav Haisman wrote:
> See attached file. The kernel is todays CVS 6.1.
> 
> --
> Vaclav Haisman
> 
> 

> May 28 19:16:55 logout kernel: lock order reversal:
> May 28 19:16:55 logout kernel: 1st 0xc3d4bb1c vnode interlock (vnode interlock) _at_ /usr/src/sys/kern/vfs_subr.c:2218
> May 28 19:16:55 logout kernel: 2nd 0xc1043144 system map (system map) _at_ /usr/src/sys/vm/vm_kern.c:295
> May 28 19:16:55 logout kernel: KDB: stack backtrace:
> May 28 19:16:55 logout kernel: kdb_backtrace(c081a0c7,c1043144,c0819bfa,c0819bfa,c082f5af) at kdb_backtrace+0x2e
> May 28 19:16:55 logout kernel: witness_checkorder(c1043144,9,c082f5af,127,c08bdbc0) at witness_checkorder+0x5ef
> May 28 19:16:55 logout kernel: _mtx_lock_flags(c1043144,0,c082f5af,127,c104e460) at _mtx_lock_flags+0x32
> May 28 19:16:55 logout kernel: _vm_map_lock(c10430c0,c082f5af,127,c3375900,c3375900) at _vm_map_lock+0x37
> May 28 19:16:55 logout kernel: kmem_malloc(c10430c0,1000,101,d5688afc,c0766493) at kmem_malloc+0x3a
> May 28 19:16:55 logout kernel: page_alloc(c104d300,1000,d5688aef,101,c05e0ed6) at page_alloc+0x27
> May 28 19:16:55 logout kernel: slab_zalloc(c104d300,101,c082ed67,8a2,80246) at slab_zalloc+0xd3
> May 28 19:16:55 logout kernel: uma_zone_slab(c104d300,1,c082ed67,8a2,c082ed67) at uma_zone_slab+0x112
> May 28 19:16:55 logout kernel: uma_zalloc_internal(c104d300,0,1,0,d5688b94) at uma_zalloc_internal+0x41
> May 28 19:16:55 logout kernel: bucket_alloc(80,1,c082ed67,95e,105) at bucket_alloc+0x3b
> May 28 19:16:55 logout kernel: uma_zfree_arg(c104dc00,c389712c,0,d5688bbc,c072e964) at uma_zfree_arg+0x253
> May 28 19:16:55 logout kernel: mac_labelzone_free(c389712c) at mac_labelzone_free+0x22
> May 28 19:16:55 logout kernel: mac_vnode_label_free(c389712c,c3d4baa0,d5688be8,c061aa00,c3d4baa0) at mac_vnode_label_free+0x94
> May 28 19:16:55 logout kernel: mac_destroy_vnode(c3d4baa0,0,c081e172,30a,c3d4baa0) at mac_destroy_vnode+0x18
> May 28 19:16:55 logout kernel: vdestroy(c3d4baa0,d5688c1c,0,c3d4e9cc,c3d4e9cc) at vdestroy+0x60
> May 28 19:16:55 logout kernel: vdropl(c3d4baa0,d5688c1c,c081e172,835,c08b1440) at vdropl+0x50
> May 28 19:16:55 logout kernel: vput(c3d4baa0,0,c082c8a2,df7,c09117a0) at vput+0x1ae
> May 28 19:16:55 logout kernel: handle_workitem_remove(c36de9e0,0,2,3b3,1) at handle_workitem_remove+0x15a
> May 28 19:16:55 logout kernel: process_worklist_item(c35a0400,0,c082c8a2,348,4479db07) at process_worklist_item+0x1e1
> May 28 19:16:55 logout kernel: softdep_process_worklist(c35a0400,0,c082c8a2,2f2,3e8) at softdep_process_worklist+0x73
> May 28 19:16:55 logout kernel: softdep_flush(0,d5688d38,c08150d4,31d,0) at softdep_flush+0x193
> May 28 19:16:55 logout kernel: fork_exit(c0744170,0,d5688d38) at fork_exit+0x78
> May 28 19:16:55 logout kernel: fork_trampoline() at fork_trampoline+0x8
> May 28 19:16:55 logout kernel: --- trap 0x1, eip = 0, esp = 0xd5688d6c, ebp = 0 ---


I already reported similar LOR to Jeff Roberson, when latest big pile of
VFS fixes where to be MFCed. Try the following patch (note, to trigger the
situation you need really high load, and, desirably, low amount
of memory installed).

My backtrace was almost identical, the difference is that your trace
coming from softdepflush daemon, and mine comes from vnlru.

lock order reversal:
 1st 0xc1a018f0 vnode interlock (vnode interlock) _at_ /usr/home/kostik/work/b=
sd/sys/kern/vfs_subr.c:2449
 2nd 0xc0c43144 system map (system map) _at_ /usr/home/kostik/work/bsd/sys/vm/=
vm_kern.c:295
KDB: stack backtrace:
kdb_backtrace(0,ffffffff,c06676b0,c0667700,c0636024) at 0xc049d3c9 =3D kdb_=
backtrace+0x29
witness_checkorder(c0c43144,9,c061fe28,127) at 0xc04a80c2 =3D witness_check=
order+0x582
_mtx_lock_flags(c0c43144,0,c061fe28,127) at 0xc047b998 =3D _mtx_lock_flags+=
0x58
_vm_map_lock(c0c430c0,c061fe28,127) at 0xc059eb46 =3D _vm_map_lock+0x26
kmem_malloc(c0c430c0,1000,101,c819fbe0,c059679f) at 0xc059e0d2 =3D kmem_mal=
loc+0x32
page_alloc(c0c4d300,1000,c819fbd3,101,c06a3bf8) at 0xc0596bda =3D page_allo=
c+0x1a
slab_zalloc(c0c4d300,101,c0c4d300,c0647a64,c0c4e460) at 0xc059679f =3D slab=
_zalloc+0x9f
uma_zone_slab(c0c4d300,1,c0c4e468,0,c061f05a,8a2) at 0xc0597dec =3D uma_zon=
e_slab+0xec
uma_zalloc_internal(c0c4d300,0,1,0,c0c4dc48) at 0xc0598129 =3D uma_zalloc_i=
nternal+0x29
bucket_alloc(80,1,c0c380a0,0,c19ab6a4) at 0xc0595eac =3D bucket_alloc+0x2c
uma_zfree_arg(c0c4dc00,c19ab6a4,0) at 0xc0598483 =3D uma_zfree_arg+0x283
mac_labelzone_free(c19ab6a4,c1a01828,e8,c819fc9c,c0565ad2) at 0xc055dab3 =
=3D mac_labelzone_free+0x13
mac_vnode_label_free(c19ab6a4,c1a01828,c819fcac,c04d8766,c1a01828) at 0xc05=
65aaa =3D mac_vnode_label_free+0x6a
mac_destroy_vnode(c1a01828) at 0xc0565ad2 =3D mac_destroy_vnode+0x12
vdestroy(c1a01828,c1a01828,c819fcec,c04d8142,c1a01828) at 0xc04d8766 =3D vd=
estroy+0x1c6
vdropl(c1a01828,7,a8,c0653ee0,c1a01828) at 0xc04dad1e =3D vdropl+0x3e
vlrureclaim(c15e8000,c1529000,c156f000,c04d8360,c156f000) at 0xc04d8142 =3D=
 vlrureclaim+0x282
vnlru_proc(0,c819fd38,0,c04d8360,0) at 0xc04d84e3 =3D vnlru_proc+0x183
fork_exit(c04d8360,0,c819fd38) at 0xc046de7d =3D fork_exit+0x9d
fork_trampoline() at 0xc05d33bc =3D fork_trampoline+0x8
--- trap 0x1, eip =3D 0, esp =3D 0xc819fd6c, ebp =3D 0 ---

A patch to fix the LOR (seems to be relevant for CURRENT too, added -current
to CC:):

--- sys/kern/vfs_subr.c.orig	Sat Mar  4 10:44:47 2006
+++ sys/kern/vfs_subr.c	Sat Mar  4 10:45:21 2006
_at__at_ -787,9 +787,6 _at__at_
 	VNASSERT(bo->bo_dirty.bv_root =3D=3D NULL, vp, ("dirtyblkroot not NULL"));
 	VNASSERT(TAILQ_EMPTY(&vp->v_cache_dst), vp, ("vp has namecache dst"));
 	VNASSERT(LIST_EMPTY(&vp->v_cache_src), vp, ("vp has namecache src"));
-#ifdef MAC
-	mac_destroy_vnode(vp);
-#endif
 	if (vp->v_pollinfo !=3D NULL) {
 		knlist_destroy(&vp->v_pollinfo->vpi_selinfo.si_note);
 		mtx_destroy(&vp->v_pollinfo->vpi_lock);
_at__at_ -801,6 +798,9 _at__at_
 #endif
 	lockdestroy(vp->v_vnlock);
 	mtx_destroy(&vp->v_interlock);
+#ifdef MAC
+	mac_destroy_vnode(vp);
+#endif
 	uma_zfree(vnode_zone, vp);
 }

Patch assumes that no mac_destroy_vnode does not mess with vnode internals,
as it is the case now.