"panic: malloc(M_WAITOK) in interrupt context" while in DDB

Got the following rather drawn out stack trace while working with DDB on a
panic earlier this afternoon.  I got a good three panics/traps:

(1) Breaking to DDB using the serial console to evaluate a hang. (fine)

(2) Calling panic from DDB to get a dump, which generated a trap because
    it tried to sync().  (Remind me: why do we do that?)

(3) Getting a panic because I called panic from that session of DDB
    because KDB tried to perform an orderly shutdown, resulting in GEOM
    panicking on trying to generate safe shutdown events by mallocing.

It sounds like we should really turn off trying to sync() in panic().  It
sounds like we also should also have a better way of having module
shutdown take place from the debugger.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research

boot() called on cpu#0
Uptime: 3m55s
Dumping 1023 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320
336 3
52 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640
656 6
72 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 960
976 9
92 1008
Dump complete
panic: malloc(M_WAITOK) in interrupt context
cpuid = 0;
KDB: enter: panic
[thread 100101]
Stopped at      kdb_enter+0x2b: nop
db> trace
kdb_enter(c07d2c35) at kdb_enter+0x2b
panic(c07d12d2,c0805a88,c224f000,c224b000,2) at panic+0x131
malloc(8,c0828920,102) at malloc+0x167
g_modevent(c224f000,2,c08069e0) at g_modevent+0x35
module_shutdown(0,104,c224b00c,0,c07d2c59) at module_shutdown+0x2d
boot(104,104,c275c2c0,0,c07f6ef4) at boot+0x5b7
panic(c079f803,e6e90664,c045ea34,c06067a7,0) at panic+0x17f
db_panic(c06067a7,0,ffffffff,e6e905d8,0) at db_panic+0xd
db_command(c0875a30,c07ffe00,c07f6ef4,c07f6f0c,c079f811) at
db_command+0x21c
db_command_loop(e6e90a6c,0,e6e90690,e6e9067c,e6e906cc) at
db_command_loop+0x5c
db_trap(3,0,3,e6e90714,c076b674) at db_trap+0xcd
kdb_trap(3,0,e6e9071c) at kdb_trap+0x75
trap(e6e90018,c0600010,c07d0010,c07f12a6,0) at trap+0x444
calltrap() at calltrap+0x5
--- trap 0x3, eip = 0xc06067a7, esp = 0xe6e9075c, ebp = 0xe6e9075c ---
kdb_enter(c07d2c35) at kdb_enter+0x2b
panic(c07f12a6,c0795a30,cf2b5000,e6e907d0,100) at panic+0x131
pmap_invalidate_range(c09ad2e0,cf2b4000,cf2b5000,cf2b4000,0) at
pmap_invalidate_
range+0x2e
pmap_qenter(cf2b4000,e6e907cc,1,c1df69a0,0) at pmap_qenter+0x50
vm_hold_load_pages(ceb2ed98,cf2b4000,cf2b7000) at vm_hold_load_pages+0xdb
allocbuf(ceb2ed98,3000,200000a4,ceb2c0e8,ceb0fb28) at allocbuf+0x2b8
geteblk(3000,c2715d68,ceb0fb28,e6e90874,c062db3f) at geteblk+0x30
ibwrite(ceb2c0e8,e6e90874,c062ea0f,ceb2c0e8,e6e90894) at ibwrite+0x17d
bwrite(ceb2c0e8,e6e90894,c06365ad,ceb2c0e8,64) at bwrite+0x32
bawrite(ceb2c0e8,64,0,e6e908d4,0) at bawrite+0x13
vop_stdfsync(e6e908d4,c2715d68,e6e908b0,c05bdd47,e6e908d4) at
vop_stdfsync+0x11d
spec_fsync(e6e908d4,e6e90900,c0714620,e6e908d4,0) at spec_fsync+0x23
spec_vnoperate(e6e908d4) at spec_vnoperate+0x13
ffs_sync(c24da800,2,c225a200,c088ec20,c24da800) at ffs_sync+0x2a4
sync(c088ec20,0,c07d2cae,e6e9098c,e6e90980) at sync+0xaf
boot(100,100,c275c2c0,0,c07f6ef4) at boot+0x13f
panic(c079f803,e6e90a40,c045ea34,80001000,0) at panic+0x17f
db_panic(80001000,0,ffffffff,e6e909b4,0) at db_panic+0xd
db_command(c0875a30,c07ffe00,c07f6ef4,c07f6f0c,c079f811) at
db_command+0x21c
db_command_loop(0,0,e6e90a6c,e6e90a58,e6e90aa8) at db_command_loop+0x5c
db_trap(3,0,3,e6e90af0,c076b674) at db_trap+0xcd
kdb_trap(3,0,e6e90af8) at kdb_trap+0x75
trap(e6e90018,c0600010,c07d0010,c23aa000,f9) at trap+0x444
calltrap() at calltrap+0x5
--- trap 0x3, eip = 0xc06067a7, esp = 0xe6e90b38, ebp = 0xe6e90b38 ---
kdb_enter(c07edc36) at kdb_enter+0x2b
siointr1(c23aa000,c0996900,0,c07ed9fe,6ac) at siointr1+0xce
siointr(c23aa000) at siointr+0x21
intr_execute_handlers(c2257490,e6e90bac,4,e6e90bf8,c075a8c3) at
intr_execute_han
dlers+0xa5
lapic_handle_intr(34) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc0627093, esp = 0xe6e90bf0, ebp = 0xe6e90bf8 ---
sblastrecordchk(c27e72c8,c07d87bd,5b4,c26a4500,c27e72dc) at
sblastrecordchk+0x33
soreceive(c27e7278,e6e90c70,e6e90c7c,0,0) at soreceive+0xcce
recvit(c275c2c0,6,e6e90cc8,bfbfc77c) at recvit+0x176
recvfrom(c275c2c0,e6e90d14,6,1,292) at recvfrom+0x6c
syscall(2f,2f,2f,8059480,8059480) at syscall+0x217
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (29, FreeBSD ELF32, recvfrom), eip = 0x280d1fb7, esp =
0xbfbfc73c, e
bp = 0xbfbfc818 ---