LOR/page fault panic vfs_mountroot

Current as of a few minutes ago.  LOR/panic.  Dual processor box.

kernel has vlan, ipfw, and dummynet enabled, but this doesn't
look like the problem.  

Curiously, booting single user and mounting root there doesn't 
panic, but it does panic if you try to 'exit' to multiuser.

[...]
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled,
default to accept, logging disabled
ad0: 19092MB <WDC WD200EB-32CSF0 04.01B04> at ata0-master UDMA33
acd0: CDROM <TOSHIBA CD-ROM XM-6702B/1007> at ata1-master UDMA33
ATA PseudoRAID loaded
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad0s1a
lock order reversal
 1st 0xc0a2d740 vm page queue mutex (vm page queue mutex)
_at_ /usr/src/sys/kern/vfs_bio.c:1485
 2nd 0xc25e4d6c vnode interlock (vnode interlock)
_at_ /usr/src/sys/kern/vfs_subr.c:1992
KDB: stack backtrace:
kdb_backtrace(c090b56b,c25e4d6c,c09107f7,c09107f7,c09106c6) at
kdb_backtrace+0x2e
witness_checkorder(c25e4d6c,9,c09106c6,7c8,c229f480) at
witness_checkorder+0x6aa
_mtx_lock_flags(c25e4d6c,0,c09106c6,7c8,c105c294) at _mtx_lock_flags+0x8a
vdrop(c25e4cf0,1,c09242b5,265,c10a74d8) at vdrop+0x32
vm_page_remove(c10a74d8,1,c09242b5,3f1,125) at vm_page_remove+0x11f
vm_page_free_toq(c10a74d8,40,c10a74d8,e35dd870,c06e2978) at
vm_page_free_toq+0xb0
vm_page_free(c10a74d8,0,c090edfb,5cd,d633a050) at vm_page_free+0x22
vfs_vmio_release(d633a050,0,c090edfb,511,0) at vfs_vmio_release+0xc8
brelse(d633a050,c25fb000,800,0,c229d180) at brelse+0x56d
ffs_mountfs(c25e4cf0,c25af400,c229f480,0,0) at ffs_mountfs+0x668
ffs_mount(c25af400,c229f480,c246f690,c229f480,e35ddaa0) at ffs_mount+0xbfa
vfs_domount(c229f480,c246f6c0,c246f690,4001,c246f7e0) at vfs_domount+0x667
vfs_donmount(c229f480,4001,e35ddbec,c259c600,6) at vfs_donmount+0x107
kernel_mount(c246faf0,4001,c25ed800,ffffffff) at kernel_mount+0x7e
kernel_vmount(4001,c091012d,c25a8040,c0910134,c090b380) at
kernel_vmount+0x4d
vfs_mountroot_try(c246f910,c229ede4,c065a410,0,e35ddd00) at
vfs_mountroot_try+0x13c
vfs_mountroot(c09d1620,1,c0903906,206,0) at vfs_mountroot+0xd4
start_init(0,e35ddd38,c0904fe9,30d,0) at start_init+0x64
fork_exit(c065a410,0,e35ddd38) at fork_exit+0xc1
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe35ddd6c, ebp = 0 ---
Pre-seeding PRNG:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 01
fault virtual address   = 0x4ac0c092
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc0703f88
stack pointer           = 0x28:0xe5092b78
frame pointer           = 0x28:0xe5092b78
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 73 (sysctl)
[thread pid 73 tid 100060 ]
Stopped at      strlen+0x8:     cmpb    $0,0(%edx)
db> show alllocks
Process 73 (sysctl) thread 0xc23b2600 (100060)
exclusive sx sysctl lock r = 0 (0xc09d1c60) locked
_at_ /usr/src/sys/kern/kern_sysctl.c:1335
exclusive sleep mutex Giant r = 0 (0xc09d1620) locked
_at_ /usr/src/sys/kern/kern_sysctl.c:1273

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);