Re: lock order reversal etc.

On Wednesday 09 September 2009 12:15:54 pm Norbert Koch wrote:
> Hello.
> 
> Is this the right mailing list to post
> kernel stack traces from Freebsd-8-BETA4?
> If yes, see below, otherwise just ignore this message.
> 
> uma_zalloc_arg: zone "64" with the following non-sleepable locks held:
> exclusive sleep mutex jail mutex (jail mutex) r = 0 (0xc0b91328) locked
> _at_
> /usr/src/sys/modules/syscons/daemon/../../../dev/syscons/daemon/daemon_saver.c:355
> KDB: stack backtrace:
> db_trace_self_wrapper(c0abf50f,e69b391c,c07ab3b5,c4a0580a,163,...) at
> db_trace_self_wrapper+0x26
> kdb_backtrace(c4a0580a,163,ffffffff,c0d261c4,e69b3954,...) at
> kdb_backtrace+0x29
> _witness_debugger(c0ac198e,e69b3968,4,1,0,...) at _witness_debugger+0x25
> witness_warn(5,0,c0ae4b92,c0add27b,e69b3984,...) at witness_warn+0x1fd
> uma_zalloc_arg(c148b000,0,2,2,14,...) at uma_zalloc_arg+0x34
> malloc(29,c0b92dd0,2,163,c46562e4,...) at malloc+0xe4
> daemon_init(c0d61940,c0aba83e,e69b3a28,c46f5600,c4a06cc0,...) at
> daemon_init+0x7b
> splash_test(7b,c46f5600,c4a06cc0,c46f5600,e69b3a50,...) at splash_test+0x91
> splash_register(c4a06c40,0,0,76,0,...) at splash_register+0x48

Try this patch for this one:

Index: daemon_saver.c
===================================================================
--- daemon_saver.c	(revision 196974)
+++ daemon_saver.c	(working copy)
_at__at_ -351,11 +351,23 _at__at_
 static int
 daemon_init(video_adapter_t *adp)
 {
+	size_t hostlen;
 
 	mtx_lock(&prison0.pr_mtx);
-	messagelen = strlen(prison0.pr_hostname) + 3 + strlen(ostype) + 1 + 
-	    strlen(osrelease);
-	message = malloc(messagelen + 1, M_DEVBUF, M_WAITOK);
+	for (;;) {
+		hostlen = strlen(prison0.pr_hostname);
+		mtx_unlock(&prison0.pr_mtx);
+	
+		messagelen = hostlen + 3 + strlen(ostype) + 1 +
+		    strlen(osrelease);
+		message = malloc(messagelen + 1, M_DEVBUF, M_WAITOK);
+		mtx_lock(&prison0.pr_mtx);
+		if (hostlen < strlen(prison0.pr_hostname)) {
+			free(message, M_DEVBUF);
+			continue;
+		}
+		break;
+	}
 	sprintf(message, "%s - %s %s", prison0.pr_hostname, ostype, osrelease);
 	mtx_unlock(&prison0.pr_mtx);
 	blanked = 0;

> lock order reversal:
>  1st 0xc4afca2c filedesc structure (filedesc structure) _at_
> /usr/src/sys/kern/kern_descrip.c:1088
>  2nd 0xc4be3488 ufs (ufs) _at_ /usr/src/sys/kern/vfs_subr.c:4091
> KDB: stack backtrace:

This is the right order.  I think several of these have been fixed, but there 
still might be some place that uses UFS -> filedesc.  You can try hardcoding 
that order into witness to find it.

> lock order reversal:
>  1st 0xd82d03f0 bufwait (bufwait) _at_ /usr/src/sys/kern/vfs_bio.c:2559
>  2nd 0xc4f64800 dirhash (dirhash) _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:285

This is known to be harmless.

> lock order reversal:
>  1st 0xc464337c ufs (ufs) _at_ /usr/src/sys/kern/vfs_subr.c:2083
>  2nd 0xd81d14b0 bufwait (bufwait) _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:6177
>  3rd 0xc673a7ac ufs (ufs) _at_ /usr/src/sys/kern/vfs_subr.c:2083
> KDB: stack backtrace:
> db_trace_self_wrapper(c0abf50f,e6b103cc,c07ab3b5,c079c13b,c0ac23fb,...)
> at db_trace_self_wrapper+0x26
> kdb_backtrace(c079c13b,c0ac23fb,c4122e90,c4126290,e6b10428,...) at
> kdb_backtrace+0x29
> _witness_debugger(c0ac23fb,c673a7ac,c0ab4f66,c4126290,c0ac96be,...) at
> _witness_debugger+0x25
> witness_checkorder(c673a7ac,9,c0ac96be,823,0,...) at
> witness_checkorder+0x839
> __lockmgr_args(c673a7ac,80100,c673a7c8,0,0,...) at __lockmgr_args+0x7a7
> ffs_lock(e6b10538,c07ab15b,c0ac8bb1,80100,c673a754,...) at ffs_lock+0x8a
> VOP_LOCK1_APV(c0bae360,e6b10538,c4c08524,c0bc5660,c673a754,...) at
> VOP_LOCK1_APV+0xb5
> _vn_lock(c673a754,80100,c0ac96be,823,4,...) at _vn_lock+0x5e
> vget(c673a754,80100,c4c08480,50,0,...) at vget+0xb9
> vfs_hash_get(c46e1508,18ef074,80000,c4c08480,e6b10694,...) at
> vfs_hash_get+0xe6
> ffs_vgetf(c46e1508,18ef074,80000,e6b10694,1,...) at ffs_vgetf+0x49
> softdep_sync_metadata(c4643324,0,c0ae2efa,146,0,...) at
> softdep_sync_metadata+0x5ba
> ffs_syncvnode(c4643324,1,c0abaa2c,c0ab45da,3,...) at ffs_syncvnode+0x3e2
> ffs_truncate(c4643324,600,0,880,c465d280,...) at ffs_truncate+0x66a
> ufs_direnter(c4643324,c673a754,e6b10a1c,e6b10c00,d82ec1d0,...) at
> ufs_direnter+0x8f6
> ufs_mkdir(e6b10c28,e6b10c3c,0,0,e6b10b6c,...) at ufs_mkdir+0x8a7
> VOP_MKDIR_APV(c0bae360,e6b10c28,e6b10c00,e6b10b6c,0,...) at
> VOP_MKDIR_APV+0xa5
> kern_mkdirat(c4c08480,ffffff9c,804c2e8,0,41fd,...) at kern_mkdirat+0x22b
> kern_mkdir(c4c08480,804c2e8,0,41fd,e6b10d2c,...) at kern_mkdir+0x2e
> mkdir(c4c08480,e6b10cf8,8,c0ac2cbd,c0b8d920,...) at mkdir+0x29
> syscall(e6b10d38) at syscall+0x2a3
> Xint0x80_syscall() at Xint0x80_syscall+0x20

This is probably harmless.

-- 
John Baldwin