geom / lock page fault crashes

I'm seeing crashes every 15 mins - 2 hours with a -current kernel.

Its always a page fault in the witness check code, from GEOM (process is
always g_up), which I don't use at all (that I know of).  Its on a 
uni-processor system, and SMP is not compiled into the kernel.  I'm 
attempting to recompile a kernel without WITNESS enabled but I'm having a 
hard time getting the system to live long enough to even compile a kernel.
But its probably from some change in the last couple of weeks, since a kernel 
from before then was stable.

kgdb output is attached. Dump & kernel are available on freefall in
~mpp/crash, but I can provide more information if needed.

-Mike

Script started on Wed Aug  1 21:55:26 2007
mail 1#	kgdb kernel.53 vmcore.53
kgdb: kvm_nlist(_stopped_cpus): 
kgdb: kvm_nlist(_stoppcbs): 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0x83004006
fault code		= supervisor read, page not present
instruction pointer	= 0x20:0xc0547a55
stack pointer	        = 0x28:0xcbb3cbb8
frame pointer	        = 0x28:0xcbb3cbf8
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 3 (g_up)
trap number		= 12
panic: page fault
Uptime: 8m42s
Physical memory: 247 MB
Dumping 60 MB: 45 29 13

#0  doadump () at pcpu.h:195
195	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc05111be in boot (howto=260) at ../../../kern/kern_shutdown.c:409
#2  0xc051141c in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:563
#3  0xc06a0a63 in trap_fatal (frame=0xcbb3cb78, eva=2197831686)
    at ../../../i386/i386/trap.c:872
#4  0xc06a0c53 in trap_pfault (frame=0xcbb3cb78, usermode=0, eva=2197831686)
    at ../../../i386/i386/trap.c:785
#5  0xc06a15e5 in trap (frame=0xcbb3cb78) at ../../../i386/i386/trap.c:463
#6  0xc068f70b in calltrap () at ../../../i386/i386/exception.s:139
#7  0xc0547a55 in witness_checkorder (lock=0xc073ebe8, flags=9, 
    file=0xc06ca5ae "../../../geom/geom_io.c", line=68)
    at ../../../kern/subr_witness.c:849
#8  0xc050607c in _mtx_lock_flags (m=0xc073ebe8, opts=0, 
    file=0xc06ca5ae "../../../geom/geom_io.c", line=68)
    at ../../../kern/kern_mutex.c:183
#9  0xc04c7959 in g_bioq_lock (bq=Variable "bq" is not available.
) at ../../../geom/geom_io.c:68
#10 0xc04c7ed6 in g_io_deliver (bp=0xc1eed480, error=0)
    at ../../../geom/geom_io.c:447
#11 0xc04ccaba in g_std_done (bp=0xc1e121b0) at ../../../geom/geom_subr.c:849
#12 0xc05732b9 in biodone (bp=0xc1e121b0) at ../../../kern/vfs_bio.c:3015
#13 0xc04c8657 in g_io_schedule_up (tp=0xc1cd9e00)
    at ../../../geom/geom_io.c:587
#14 0xc04c8e2d in g_up_procbody () at ../../../geom/geom_kern.c:95
#15 0xc04f3758 in fork_exit (callout=0xc04c8da0 <g_up_procbody>, arg=0x0, 
---Type <return> to continue, or q <return> to quit---
    frame=0xcbb3cd38) at ../../../kern/kern_fork.c:797
#16 0xc068f780 in fork_trampoline () at ../../../i386/i386/exception.s:205
(kgdb) up
#1  0xc05111be in boot (howto=260) at ../../../kern/kern_shutdown.c:409
409			doadump();
(kgdb) up
#2  0xc051141c in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:563
563		boot(bootopt);
(kgdb) up
#3  0xc06a0a63 in trap_fatal (frame=0xcbb3cb78, eva=2197831686)
    at ../../../i386/i386/trap.c:872
872			panic("%s", trap_msg[type]);
(kgdb) up
#4  0xc06a0c53 in trap_pfault (frame=0xcbb3cb78, usermode=0, eva=2197831686)
    at ../../../i386/i386/trap.c:785
785			trap_fatal(frame, eva);
(kgdb) up
#5  0xc06a15e5 in trap (frame=0xcbb3cb78) at ../../../i386/i386/trap.c:463
463				(void) trap_pfault(frame, FALSE, eva);
(kgdb) up
#6  0xc068f70b in calltrap () at ../../../i386/i386/exception.s:139
139		call	trap
Current language:  auto; currently asm
(kgdb) up
#7  0xc0547a55 in witness_checkorder (lock=0xc073ebe8, flags=9, 
    file=0xc06ca5ae "../../../geom/geom_io.c", line=68)
    at ../../../kern/subr_witness.c:849
849		if (witness_cold || witness_watch == 0 || lock->lo_witness == NULL ||
Current language:  auto; currently c
(kgdb) list
844		struct lock_class *class;
845		struct witness *w, *w1;
846		struct thread *td;
847		int i, j;
848	
849		if (witness_cold || witness_watch == 0 || lock->lo_witness == NULL ||
850		    panicstr != NULL)
851			return;
852	
853		/*
(kgdb) print witness_cold
$1 = 0
(kgdb) print witness_watch
$2 = 1
(kgdb) print *lock
$3 = {lo_name = 0xc06c2381 "bio queue", lo_type = 0xc06c2381 "bio queue", 
  lo_flags = 16973824, lo_witness_data = {lod_list = {
      stqe_next = 0xc074d828}, lod_witness = 0xc074d828}}
(kgdb) print *lock->lo_witness_data->lod_witness
$4 = {w_name = 0xc06c2381 "bio queue", w_class = 0xc070c064, w_list = {
    stqe_next = 0xc074d850}, w_typelist = {stqe_next = 0xc074d850}, 
  w_children = 0x0, w_file = 0xc06ca5b7 "geom/geom_io.c", w_line = 68, 
  w_level = 0, w_refcount = 3, w_Giant_squawked = 0 '\0', 
  w_other_squawked = 0 '\0', w_same_squawked = 0 '\0', w_displayed = 0 '\0'}
(kgdb) quit