r3077XX: LOR in vfs

Since r307157 I'm bugged with sporadic reboots/crashes of all boxes running CURRENT. That
is different CPU types (XEON C2D, XEON Haswell, XEON IvyBridge, mobile Haswell and
desktop IvyBridge).

Below, you'll finde some messages I gathered from the console. I hope this is of any use,
if not: I'm very sorry.

I'm also sorry being incapable of providing you with more informations of the crashes. I
do not have any clue how the crashes are related - I try to configure some boxes I have
direct access to with debugging options, but unfortunately they all have nvidia GPUs and
configured vt() for the UEFI and non UEFI systems - that means, the console is wreckage
since there is an unresolved buf as long as the nvidia-mode.ko is loaded. So far, the
crashing boxes didn't capture a core.

From the reports I got of the other crashing servers running recent CURRENT, last process
in braces is in most cases "(swapper)" or "(pager)" - I got this report via phone, sorry.

Anothe observation is that boxes with larger amounts of RAM last longer - a workstation
with 32 GB and r3076XX lasted for ~ 32 hours befor it crashed, a server with 16 GB
takes several hours, sometimes only minutes and another machine with 8 GB goes down mor
frequently.

Anyway, try to give some mor infos in case I'm able to capture those. I'm really
surprised that it seems that I'm the only one facing those problems at the moment.  

[...]
Oct 21 05:09:59 <0.2> thor kernel: lock order reversal:
Oct 21 05:09:59 <0.2> thor kernel: 1st 0xfffffe01e98449f8 bufwait (bufwait)
_at_ /usr/src/sys/kern/vfs_bio.c:3505 Oct 21 05:09:59 <0.2> thor kernel: 2nd
0xfffff800098c2800 dirhash (dirhash) _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:281 Oct 21
05:09:59 <0.2> thor kernel: stack backtrace: Oct 21 05:09:59 <0.2> thor kernel: #0
0xffffffff80818410 at witness_debugger+0x70 Oct 21 05:09:59 <0.2> thor kernel: #1
0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 05:09:59 <0.2> thor kernel: #2
0xffffffff807bbdc2 at _sx_xlock+0x72 Oct 21 05:09:59 <0.2> thor kernel: #3
0xffffffff80a8b507 at ufsdirhash_remove+0x37 Oct 21 05:09:59 <0.2> thor kernel: #4
0xffffffff80a8e760 at ufs_dirremove+0x120 Oct 21 05:09:59 <0.2> thor kernel: #5
0xffffffff80a961a8 at ufs_rename+0x14a8 Oct 21 05:09:59 <0.2> thor kernel: #6
0xffffffff80b6ce88 at VOP_RENAME_APV+0xe8 Oct 21 05:09:59 <0.2> thor kernel: #7
0xffffffff80891fa8 at kern_renameat+0x4a8 Oct 21 05:09:59 <0.2> thor kernel: #8
0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 05:09:59 <0.2> thor kernel: #9
0xffffffff80ad6f1b at Xfast_syscall+0xfb Oct 21 05:10:00 <3.3> thor nslcd[543]: [d0722e]
<group/member="root"> no available LDAP server found: Server is unavailable: Resource
temporarily unavailable Oct 21 05:10:00 <3.3> thor nslcd[543]: [d0722e]
<group/member="root"> no available LDAP server found: Server is unavailable: Resource
temporarily unavailable Oct 21 05:10:00 <0.2> thor kernel: acquiring duplicate lock of
same type: "os.lock_sx" Oct 21 05:10:00 <0.2> thor kernel: 1st os.lock_sx _at_
nvidia_os.c:599 Oct 21 05:10:00 <0.2> thor kernel: 2nd os.lock_sx _at_ nvidia_os.c:599 Oct
21 05:10:00 <0.2> thor kernel: stack backtrace: Oct 21 05:10:00 <0.2> thor kernel: #0
0xffffffff80818410 at witness_debugger+0x70 Oct 21 05:10:00 <0.2> thor kernel: #1
0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 05:10:00 <0.2> thor kernel: #2
0xffffffff807bbdc2 at _sx_xlock+0x72 Oct 21 05:10:00 <0.2> thor kernel: #3
0xffffffff81e792b2 at os_acquire_mutex+0x32 Oct 21 05:10:00 <0.2> thor kernel: #4
0xffffffff81e48bc8 at _nv017102rm+0x18 Oct 21 05:10:00 <0.2> thor kernel: acquiring
duplicate lock of same type: "os.lock_mtx" Oct 21 05:10:00 <0.2> thor kernel: 1st
os.lock_mtx _at_ nvidia_os.c:778 Oct 21 05:10:00 <0.2> thor kernel: 2nd os.lock_mtx _at_
nvidia_os.c:778 Oct 21 05:10:00 <0.2> thor kernel: stack backtrace: Oct 21 05:10:00 <0.2>
thor kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21 05:10:00 <0.2> thor
kernel: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 05:10:00 <0.2> thor
kernel: #2 0xffffffff80792604 at __mtx_lock_flags+0xa4 Oct 21 05:10:00 <0.2> thor kernel:
#3 0xffffffff81e7962b at os_acquire_spinlock+0x1b Oct 21 05:10:00 <0.2> thor kernel: #4
0xffffffff81b97705 at _nv011755rm+0x185 Oct 21 05:10:00 <0.6> thor kernel:
nvidia-modeset: Allocated GPU:0 (GPU-85fde95a-7974-9962-f1a4-d7c164413929) _at_
PCI:0000:01:00.0

[...]

Oct 21 05:23:19 <0.2> thor kernel: lock order reversal:
Oct 21 05:23:19 <0.2> thor kernel: 1st 0xfffff8017ee32d50 ufs (ufs)
_at_ /usr/src/sys/kern/vfs_lookup.c:522 Oct 21 05:23:19 <0.2> thor kernel: 2nd
0xfffffe01ea131fa8 bufwait (bufwait) _at_ /usr/src/sys/ufs/ffs/ffs_vnops.c:278 Oct 21
05:23:19 <0.2> thor kernel: 3rd 0xfffff8017eebe5f0 ufs (ufs)
_at_ /usr/src/sys/kern/vfs_subr.c:2592 Oct 21 05:23:19 <0.2> thor kernel: stack backtrace:
Oct 21 05:23:19 <0.2> thor kernel: #0 0xffffffff80818410 at witness_debugger+0x70 Oct 21
05:23:19 <0.2> thor kernel: #1 0xffffffff8081830d at witness_checkorder+0xf0d Oct 21
05:23:19 <0.2> thor kernel: #2 0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 05:23:19
<0.2> thor kernel: #3 0xffffffff80a855f6 at ffs_lock+0xa6 Oct 21 05:23:19 <0.2> thor
kernel: #4 0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 05:23:19 <0.2> thor kernel: #5
0xffffffff808955ca at _vn_lock+0x9a Oct 21 05:23:19 <0.2> thor kernel: #6
0xffffffff80884eb4 at vget+0x64 Oct 21 05:23:19 <0.2> thor kernel: #7 0xffffffff80876d7e
at vfs_hash_get+0xfe Oct 21 05:23:19 <0.2> thor kernel: #8 0xffffffff80a81220 at
ffs_vgetf+0x40 Oct 21 05:23:19 <0.2> thor kernel: #9 0xffffffff80a76dee at
softdep_sync_buf+0xd4e Oct 21 05:23:19 <0.2> thor kernel: #10 0xffffffff80a86973 at
ffs_syncvnode+0x2a3 Oct 21 05:23:19 <0.2> thor kernel: #11 0xffffffff80a59fd2 at
ffs_truncate+0x7d2 Oct 21 05:23:19 <0.2> thor kernel: #12 0xffffffff80a8e24b at
ufs_direnter+0x68b Oct 21 05:23:19 <0.2> thor kernel: #13 0xffffffff80a978bd at
ufs_makeinode+0x59d Oct 21 05:23:19 <0.2> thor kernel: #14 0xffffffff80a935c3 at
ufs_create+0x33 Oct 21 05:23:19 <0.2> thor kernel: #15 0xffffffff80b6b2da at
VOP_CREATE_APV+0xda Oct 21 05:23:19 <0.2> thor kernel: #16 0xffffffff80894de3 at
vn_open_cred+0x573 Oct 21 05:23:19 <0.2> thor kernel: #17 0xffffffff8088d52c at
kern_openat+0x25c

[...]

Oct 21 20:16:57 <0.2> thor kernel: All buffers synced.
Oct 21 20:16:57 <0.2> thor kernel: lock order reversal:
Oct 21 20:16:57 <0.2> thor kernel: 1st 0xfffff800080419a0 ufs (ufs)
_at_ /usr/src/sys/kern/vfs_mount.c:1253 Oct 21 20:16:57 <0.2> thor kernel: 2nd
0xfffff80004f2a240 syncer (syncer) _at_ /usr/src/sys/kern/vfs_subr.c:2732 Oct 21 20:16:57
<0.2> thor kernel: stack backtrace: Oct 21 20:16:57 <0.2> thor kernel: #0
0xffffffff80818410 at witness_debugger+0x70 Oct 21 20:16:57 <0.2> thor kernel: #1
0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 20:16:57 <0.2> thor kernel: #2
0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 20:16:57 <0.2> thor kernel: #3
0xffffffff8087318c at vop_stdlock+0x3c Oct 21 20:16:57 <0.2> thor kernel: #4
0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 20:16:57 <0.2> thor kernel: #5
0xffffffff808955ca at _vn_lock+0x9a Oct 21 20:16:57 <0.2> thor kernel: #6
0xffffffff80885539 at vputx+0x169 Oct 21 20:16:57 <0.2> thor kernel: #7
0xffffffff8087cfeb at dounmount+0x5eb Oct 21 20:16:57 <0.2> thor kernel: #8
0xffffffff80886ccb at vfs_unmountall+0x6b Oct 21 20:16:57 <0.2> thor kernel: #9
0xffffffff80863441 at bufshutdown+0x421 Oct 21 20:16:57 <0.2> thor kernel: #10
0xffffffff807b2fc9 at kern_reboot+0x189 Oct 21 20:16:57 <0.2> thor kernel: #11
0xffffffff807b2de3 at sys_reboot+0x3e3 Oct 21 20:16:57 <0.2> thor kernel: #12
0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 20:16:57 <0.2> thor kernel: #13
0xffffffff80ad6f1b at Xfast_syscall+0xfb Oct 21 20:16:57 <0.2> thor kernel: lock order
reversal: Oct 21 20:16:57 <0.2> thor kernel: 1st 0xfffff80008166240 devfs (devfs)
_at_ /usr/src/sys/kern/vfs_mount.c:1253 Oct 21 20:16:57 <0.2> thor kernel: 2nd
0xfffff80008165d50 syncer (syncer) _at_ /usr/src/sys/kern/vfs_subr.c:2732 Oct 21 20:16:57
<0.2> thor kernel: stack backtrace: Oct 21 20:16:57 <0.2> thor kernel: #0
0xffffffff80818410 at witness_debugger+0x70 Oct 21 20:16:57 <0.2> thor kernel: #1
0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 20:16:57 <0.2> thor kernel: #2
0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 20:16:57 <0.2> thor kernel: #3
0xffffffff8087318c at vop_stdlock+0x3c Oct 21 20:16:57 <0.2> thor kernel: #4
0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 20:16:57 <0.2> thor kernel: #5
0xffffffff808955ca at _vn_lock+0x9a Oct 21 20:16:57 <0.2> thor kernel: #6
0xffffffff80885539 at vputx+0x169 Oct 21 20:16:57 <0.2> thor kernel: #7
0xffffffff8087cfeb at dounmount+0x5eb Oct 21 20:16:57 <0.2> thor kernel: #8
0xffffffff80886ccb at vfs_unmountall+0x6b Oct 21 20:16:57 <0.2> thor kernel: #9
0xffffffff80863441 at bufshutdown+0x421 Oct 21 20:16:57 <0.2> thor kernel: #10
0xffffffff807b2fc9 at kern_reboot+0x189 Oct 21 20:16:57 <0.2> thor kernel: #11
0xffffffff807b2de3 at sys_reboot+0x3e3 Oct 21 20:16:57 <0.2> thor kernel: #12
0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 20:16:57 <0.2> thor kernel: #13
0xffffffff80ad6f1b at Xfast_syscall+0xfb Oct 21 20:16:57 <0.2> thor kernel: lock order
reversal: Oct 21 20:16:57 <0.2> thor kernel: 1st 0xfffff800081669a0 ufs (ufs)
_at_ /usr/src/sys/kern/vfs_mount.c:1253 Oct 21 20:16:57 <0.2> thor kernel: 2nd
0xfffff80004f2b068 devfs (devfs) _at_ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1386 Oct 21 20:16:57
<0.2> thor kernel: stack backtrace: Oct 21 20:16:57 <0.2> thor kernel: #0
0xffffffff80818410 at witness_debugger+0x70 Oct 21 20:16:57 <0.2> thor kernel: #1
0xffffffff8081830d at witness_checkorder+0xf0d Oct 21 20:16:57 <0.2> thor kernel: #2
0xffffffff80788e5f at __lockmgr_args+0x4df Oct 21 20:16:57 <0.2> thor kernel: #3
0xffffffff8087318c at vop_stdlock+0x3c Oct 21 20:16:57 <0.2> thor kernel: #4
0xffffffff80b6d9f0 at VOP_LOCK1_APV+0xe0 Oct 21 20:16:57 <0.2> thor kernel: #5
0xffffffff808955ca at _vn_lock+0x9a Oct 21 20:16:57 <0.2> thor kernel: #6
0xffffffff80a81168 at ffs_flushfiles+0x118 Oct 21 20:16:57 <0.2> thor kernel: #7
0xffffffff80a62aa9 at softdep_flushfiles+0x569 Oct 21 20:16:57 <0.2> thor kernel: #8
0xffffffff80a837a2 at ffs_unmount+0x82 Oct 21 20:16:57 <0.2> thor kernel: #9
0xffffffff8087d10d at dounmount+0x70d Oct 21 20:16:57 <0.2> thor kernel: #10
0xffffffff80886ccb at vfs_unmountall+0x6b Oct 21 20:16:57 <0.2> thor kernel: #11
0xffffffff80863441 at bufshutdown+0x421 Oct 21 20:16:57 <0.2> thor kernel: #12
0xffffffff807b2fc9 at kern_reboot+0x189 Oct 21 20:16:57 <0.2> thor kernel: #13
0xffffffff807b2de3 at sys_reboot+0x3e3 Oct 21 20:16:57 <0.2> thor kernel: #14
0xffffffff80af6934 at amd64_syscall+0x314 Oct 21 20:16:57 <0.2> thor kernel: #15
0xffffffff80ad6f1b at Xfast_syscall+0xfb