new twist on IPI deadlock

Got this on my xeon today, with hyperthreading disabled, and dropping to
ddb with NMI after a hang.  Looks like smp_rendezvous_action() colliding
with smp_tlb_shootdown().

smp_rendezvous and smp_tlb_shootdown use different IPI vectors and
different locks, but I wonder if the operations aren't orthogonal, and
doing multiple IPIs at once can cause unexpected behavior.

kernel trap 19 with interrupts disabled
NMI ... going to debugger
[thread 100168]
Stopped at      smp_rendezvous_action+0x30:     cmpl    mp_ncpus,%eax
db>    tr
smp_rendezvous_action(fd) at smp_rendezvous_action+0x30
smp_rendezvous(0,c06a0724,0,c2a66420) at smp_rendezvous+0xd7
i386_ldt_grow(c2a66420,12,8,dfe2a000,c2a63f60) at i386_ldt_grow+0x1b1
i386_set_ldt(c2a66420,bfbfe968,c2a63de0,0,dfd61d40) at i386_set_ldt+0x2de
sysarch(c2a66420,dfd61d14,2,0,206) at sysarch+0x67
syscall(2f,2f,2f,2807f010,0) at syscall+0x287
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (165, FreeBSD ELF32, sysarch), eip = 0x2807196f, esp =
0xbfbfe954, ebp = 0x-
[...]
db> tr 28658
sched_switch(f6,dc511000,dc512000) at sched_switch+0x9b
smp_invlpg_range(dc511000,dc512000) at smp_invlpg_range+0x1c
pmap_invalidate_range(c0775e20,dc511000,dc512000,c2294780,c2277170) at
pmap_invalidate_5
pmap_qenter(dc511000,c2294790,1) at pmap_qenter+0x50
sf_buf_alloc(c1a329e0,0,0,0,0) at sf_buf_alloc+0x1a9
uiomove_fromphys(c28a9600,3000,8d8,dfed5c88,0) at uiomove_fromphys+0x92
pipe_read(c2a68bf4,dfed5c88,c2b7c400,0,c349a2c0) at pipe_read+0x238
dofileread(c349a2c0,c2a68bf4,0,812a000,4000) at dofileread+0x95
read(c349a2c0,dfed5d14,3,0,296) at read+0x3b
syscall(2f,2f,2f,80da500,80f7034) at syscall+0x287
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x80a5783, esp = 0xbfbfe69c,
ebp = 0xbfbfe6-
db> tr 28690
smp_rendezvous_action(fd) at smp_rendezvous_action+0x30
smp_rendezvous(0,c06a0724,0,c2a66420) at smp_rendezvous+0xd7
i386_ldt_grow(c2a66420,12,8,dfe2a000,c2a63f60) at i386_ldt_grow+0x1b1
i386_set_ldt(c2a66420,bfbfe968,c2a63de0,0,dfd61d40) at i386_set_ldt+0x2de
sysarch(c2a66420,dfd61d14,2,0,206) at sysarch+0x67
syscall(2f,2f,2f,2807f010,0) at syscall+0x287
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (165, FreeBSD ELF32, sysarch), eip = 0x2807196f, esp =
0xbfbfe954, ebp = 0x-


-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite_at_gumbysoft.com          |  www.FreeBSD.org