uma_zfree: Freeing to non free bucket index.

From: Robert Watson <rwatson_at_FreeBSD.org>
Date: Mon, 18 Oct 2004 18:13:00 -0400 (EDT)
I've not seen this UMA failure before -- saw it under a high web load on
an SMP Xeon here.  Some debugging details from DDB below.  I have a
workable core; a few kgdb output blips are below the DDB output. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research


Heavy web service load on hippy.rv.nailabs.com with GENERIC kernel and
accept lock patches. 

FreeBSD/i386 (hippy.rv.nailabs.com) (ttyd0)

login: panic: uma_zfree: Freeing to non free bucket index.
cpuid = 2
KDB: enter: panic
[thread 100014]
Stopped at      kdb_enter+0x2b: nop
db> trace
kdb_enter(c07fc72c) at kdb_enter+0x2b
panic(c0815e8e,1,2,c22583c0,c2821100) at panic+0x127
uma_zfree_arg(c101fc60,c2821100,0) at uma_zfree_arg+0xa5
mb_free_ext(c2821100) at mb_free_ext+0x39
m_freem(c2821100,0,0,1,1) at m_freem+0x21
tcp_input(c2821100,14,c2821100,0,0) at tcp_input+0x2d1c
ip_input(c2821100) at ip_input+0x50d
netisr_processqueue(c08eae58) at netisr_processqueue+0x6e
swi_net(0) at swi_net+0xbe
ithread_loop(c2260c00,e3384d48,c2260c00,c05f7d50,0) at ithread_loop+0x124
fork_exit(c05f7d50,c2260c00,e3384d48) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe3384d7c, ebp = 0 ---
db> show locks
exclusive sleep mutex UMA pcpu r = 0 (0xc08f8548) locked _at_ vm/uma_core.c:2215
exclusive sleep mutex inp (tcpinp) r = 0 (0xc2b4d2ac) locked _at_ netinet/tcp_input.c:743
exclusive sleep mutex tcp r = 0 (0xc08ec02c) locked _at_ netinet/tcp_input.c:617
db> show pcpu
cpuid        = 2
curthread    = 0xc2268600: pid 38 "swi1: net"
curpcb       = 0xe3384da0
fpcurthread  = none
idlethread   = 0xc2262780: pid 12 "idle: cpu2"
APIC ID      = 2
currentldt   = 0x28
spin locks held:
db> ps
  pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
  619 c2b1ce00 ef357000    0   507   507 0000100 [SLPQ kqread 0xc27fb300][SLP] httpd
  618 c2b1cc00 ef356000    0   507   507 0000100 [SLPQ kqread 0xc2aacd00][SLP] httpd
  617 c2931e00 ef240000    0   507   507 0000100 [SLPQ kqread 0xc2aad500][SLP] httpd
  616 c2b22600 ef35b000    0   507   507 0000100 [SLPQ kqread 0xc27fb600][SLP] httpd
  615 c2931800 ef23d000   80   507   507 0000100 [Can run] httpd
  614 c2931a00 ef23e000   80   507   507 0000100 [SLPQ accept 0xc2800916][SLP] httpd
  613 c2735000 ef16e000   80   507   507 0000100 [SLPQ sbwait 0xc2acac64][SLP] httpd
  589 c2afe200 ef301000   80   507   507 0000100 [SLPQ sbwait 0xc2b3cda8][SLP] httpd
  588 c2afe400 ef302000   80   507   507 0000100 [SLPQ sbwait 0xc2b3c388][SLP] httpd
  587 c2afe600 ef303000   80   507   507 0000100 [SLPQ sbwait 0xc2aca9dc][SLP] httpd
  586 c26eec00 ecf80000   80   507   507 0000100 [SLPQ sbwait 0xc2b44610][SLP] httpd
  585 c2735c00 ef194000   80   507   507 0000100 [SLPQ sbwait 0xc2b23b20][SLP] httpd
  584 c26eea00 ecf7f000   80   507   507 0000100 [SLPQ accept 0xc2800916][SLP] httpd
  583 c2795a00 ef1b5000   80   507   507 0000100 [SLPQ sbwait 0xc2aca754][SLP] httpd
  582 c2795400 ef1b2000   80   507   507 0000100 [SLPQ sbwait 0xc2b23da8][SLP] httpd
  581 c2797000 ef1b8000   80   507   507 0000100 [Can run] httpd
  580 c273a800 ef19a000   80   507   507 0000100 [Can run] httpd
  579 c2795000 ef1b0000   80   507   507 0000100 [SLPQ sbwait 0xc2b23100][SLP] httpd
  578 c273ae00 ef19d000   80   507   507 0000100 [Can run] httpd
  577 c2797400 ef1f9000   80   507   507 0000100 [SLPQ sbwait 0xc2b3cc64][SLP] httpd
  576 c273aa00 ef19b000   80   507   507 0000100 [SLPQ sbwait 0xc2acada8][SLP] httpd
  575 c2795e00 ef1b7000   80   507   507 0000100 [SLPQ sbwait 0xc2b234cc][SLP] httpd
  574 c26ed000 ecf36000   80   507   507 0000100 [Can run] httpd
  573 c2797200 ef1b9000   80   507   507 0000100 [SLPQ accept 0xc2800916][SLP] httpd
  572 c2795800 ef1b4000   80   507   507 0000100 [Can run] httpd
  571 c273ac00 ef19c000   80   507   507 0000100 [SLPQ sbwait 0xc2ac94cc][SLP] httpd
  570 c2930600 ef216000   80   507   507 0000100 [Can run] httpd
  569 c2930400 ef215000    0     1   569 0004002 [SLPQ ttyin 0xc24ab010][SLP] getty
  568 c2797c00 ef1fd000    0     1   568 0004002 [SLPQ ttyin 0xc24c9410][SLP] getty
  567 c2930e00 ef21a000    0     1   567 0004002 [SLPQ ttyin 0xc24ca410][SLP] getty
  566 c26ed600 ecf39000    0     1   566 0004002 [SLPQ ttyin 0xc24ca010][SLP] getty
  565 c2797a00 ef1fc000    0     1   565 0004002 [SLPQ ttyin 0xc24c8c10][SLP] getty
  564 c2931600 ef23c000    0     1   564 0004002 [SLPQ ttyin 0xc24c8810][SLP] getty
  563 c273a600 ef199000    0     1   563 0004002 [SLPQ ttyin 0xc24c0c10][SLP] getty
  562 c2797e00 ef1fe000    0     1   562 0004002 [SLPQ ttyin 0xc24c8010][SLP] getty
  561 c2797600 ef1fa000    0     1   561 0004002 [SLPQ ttyin 0xc24c8410][SLP] getty
  558 c2930c00 ef219000   88   511    65 000c182 (threaded)  mysqld
   thread 0xc279bc00 ksegrp 0xc2739a10 [SLPQ kserel 0xc2739a50][SLP]
   thread 0xc2932000 ksegrp 0xc27394d0 [SLPQ kserel 0xc2739510][SLP]
   thread 0xc279b600 ksegrp 0xc2739a10 [SLPQ kserel 0xc2739a50][SLP]
   thread 0xc279b900 ksegrp 0xc2739a10 [SLPQ select 0xc08e9ee4][SLP]
   thread 0xc2af8600 ksegrp 0xc2739a10 [SLPQ kserel 0xc2739a50][SLP]
   thread 0xc2af8300 ksegrp 0xc2739a10 [SLPQ kserel 0xc2739a50][SLP]
   thread 0xc2af8000 ksegrp 0xc2739540 [SLPQ sigwait 0xef2c0c2c][SLP]
   thread 0xc2932300 ksegrp 0xc27395b0 [SLPQ ksesigwait 0xc2930d3c][SLP]
  511 c26ed200 ecf37000    0     1    65 0004002 [SLPQ wait 0xc26ed200][SLP] sh
  507 c2735400 ef190000    0     1   507 0000000 [SLPQ select 0xc08e9ee4][SLP] httpd
  489 c2735800 ef192000    0     1   489 0000000 [SLPQ nanslp 0xc08bfccc][SLP] cron
  476 c2931400 ef23b000   25     1   476 0000100 [SLPQ pause 0xc2931438][SLP] sendmail
  472 c2735e00 ef195000    0     1   472 0000100 [SLPQ select 0xc08e9ee4][SLP] sendmail
  467 c273a200 ef197000    0     1   467 0000100 [SLPQ select 0xc08e9ee4][SLP] sshd
  441 c26ed400 ecf38000    0     1   441 0000000 [SLPQ select 0xc08e9ee4][SLP] lpd
  424 c2797800 ef1fb000    0     1   424 0000000 [SLPQ select 0xc08e9ee4][SLP] usbd
  400 c2931000 ef21b000    0     1   400 0000000 [SLPQ select 0xc08e9ee4][SLP] rpc.statd
  394 c2795c00 ef1b6000    0   390   390 0000000 [SLPQ - 0xc26c1a00][SLP] nfsd
  393 c2795200 ef1b1000    0   390   390 0000000 [SLPQ - 0xc26dbc00][SLP] nfsd
  392 c273a000 ef196000    0   390   390 0000000 [SLPQ - 0xc26dc400][SLP] nfsd
  391 c26eee00 ecf81000    0   390   390 0000000 [SLPQ - 0xc26cd200][SLP] nfsd
  390 c2930000 ef1ae000    0     1   390 0000000 [SLPQ select 0xc08e9ee4][SLP] nfsd
  388 c2735a00 ef193000    0     1   388 0000000 [SLPQ select 0xc08e9ee4][SLP] mountd
  322 c2930200 ef1af000    0     1   322 0000000 [SLPQ select 0xc08e9ee4][SLP] ypbind
  309 c2931200 ef21c000    0     1   309 0000000 [SLPQ select 0xc08e9ee4][SLP] rpcbind
  294 c2378e00 e4e81000    0     1   294 0000000 [SLPQ select 0xc08e9ee4][SLP] syslogd
  271 c273a400 ef198000    0     1   271 0000000 [SLPQ select 0xc08e9ee4][SLP] devd
  242 c2795600 ef1b3000    0     1   242 0000000 [SLPQ select 0xc08e9ee4][SLP] dhclient
   64 c26ed800 ecf3a000    0     0     0 0000204 [SLPQ - 0xe4e4fd14][SLP] schedcpu
   63 c26eda00 ecf3b000    0     0     0 0000204 [SLPQ - 0xc08f192c][SLP] nfsiod 3
   62 c26edc00 ecf3c000    0     0     0 0000204 [SLPQ - 0xc08f1928][SLP] nfsiod 2
   61 c26ede00 ecf3d000    0     0     0 0000204 [SLPQ - 0xc08f1924][SLP] nfsiod 1
   60 c26ee000 ecf3e000    0     0     0 0000204 [SLPQ - 0xc08f1920][SLP] nfsiod 0
   59 c26ee200 ecf3f000    0     0     0 0000204 [SLPQ vlruwt 0xc26ee200][SLP] vnlru
   58 c26ee400 ecf7c000    0     0     0 0000204 [SLPQ syncer 0xc08bfa4c][SLP] syncer
   57 c26ee600 ecf7d000    0     0     0 0000204 [SLPQ psleep 0xc08ea4ac][SLP] bufdaemon
   56 c26ee800 ecf7e000    0     0     0 000020c [SLPQ pgzero 0xc08f8270][SLP] pagezero
   55 c22d0400 e4e38000    0     0     0 0000204 [SLPQ psleep 0xc08f82c4][SLP] vmdaemon
   54 c22d0600 e4e39000    0     0     0 0000204 [SLPQ psleep 0xc08f8280][SLP] pagedaemon
   53 c22d0800 e4e3a000    0     0     0 0000204 [RUNQ] swi0: sio
   52 c22d0a00 e4e3b000    0     0     0 0000204 [SLPQ - 0xc23ac83c][SLP] fdc0
   51 c22d0c00 e4e3c000    0     0     0 0000204 [SLPQ usbevt 0xc249e210][SLP] usb1
   50 c22d0e00 e4e3d000    0     0     0 0000204 [SLPQ usbtsk 0xc08b7bb8][SLP] usbtask
   49 c2378000 e4e3e000    0     0     0 0000204 [SLPQ usbevt 0xc249a210][SLP] usb0
   48 c2378200 e4e3f000    0     0     0 0000204 [SLPQ idle 0xc2376600][SLP] aic_recovery0
   47 c2378400 e4e40000    0     0     0 0000204 [SLPQ idle 0xc2376600][SLP] aic_recovery0
    9 c2378600 e4e7d000    0     0     0 0000204 [SLPQ actask 0xc0a23a2c][SLP] acpi_task2
    8 c2378800 e4e7e000    0     0     0 0000204 [SLPQ actask 0xc0a23a2c][SLP] acpi_task1
    7 c2378a00 e4e7f000    0     0     0 0000204 [SLPQ actask 0xc0a23a2c][SLP] acpi_task0
   46 c2378c00 e4e80000    0     0     0 0000204 [IWAIT] swi6:+
   45 c22c3c00 e4e0e000    0     0     0 0000204 [IWAIT] swi6: task queue
   44 c22c3e00 e4e0f000    0     0     0 0000204 [IWAIT] swi6: acpitaskq
    6 c22cc000 e4e10000    0     0     0 0000204 [SLPQ - 0xc22f5640][SLP] kqueue taskq
   43 c22cc200 e4e11000    0     0     0 0000204 [IWAIT] swi2: cambio
   42 c22cc400 e4e12000    0     0     0 0000204 [IWAIT] swi5:+
    5 c22cc600 e4e13000    0     0     0 0000204 [SLPQ - 0xc22f5840][SLP] thread taskq
   41 c22cc800 e4e14000    0     0     0 0000204 [SLPQ - 0xc08b5900][SLP] yarrow
    4 c22cca00 e4e33000    0     0     0 0000204 [SLPQ - 0xc08ba568][SLP] g_down
    3 c22ccc00 e4e34000    0     0     0 0000204 [SLPQ - 0xc08ba564][SLP] g_up
    2 c22cce00 e4e35000    0     0     0 0000204 [SLPQ - 0xc08ba55c][SLP] g_event
   40 c22d0000 e4e36000    0     0     0 0000204 [IWAIT] swi3: vm
   39 c22d0200 e4e37000    0     0     0 000020c [RUNQ] swi4: clock sio
   38 c22b3600 e4de5000    0     0     0 0000204 [CPU 2] swi1: net
   37 c22b3800 e4de6000    0     0     0 0000204 [IWAIT] irq0: clk
   36 c22b3a00 e4de7000    0     0     0 0000204 [CPU 0] irq23: xl0 uhci1
   35 c22b3c00 e4de8000    0     0     0 0000204 [IWAIT] irq22: ahc0
   34 c22b3e00 e4de9000    0     0     0 0000204 [IWAIT] irq21:
   33 c22c3000 e4e08000    0     0     0 0000204 [IWAIT] irq20: em0
   32 c22c3200 e4e09000    0     0     0 0000204 [IWAIT] irq19: uhci0
   31 c22c3400 e4e0a000    0     0     0 0000204 [IWAIT] irq18:
   30 c22c3600 e4e0b000    0     0     0 0000204 [IWAIT] irq17:
   29 c22c3800 e4e0c000    0     0     0 0000204 [IWAIT] irq16: fwohci0
   28 c22c3a00 e4e0d000    0     0     0 0000204 [IWAIT] irq15: ata1
   27 c226b200 e339c000    0     0     0 0000204 [IWAIT] irq14: ata0
   26 c226b400 e339d000    0     0     0 0000204 [IWAIT] irq13:
   25 c226b600 e339e000    0     0     0 0000204 [IWAIT] irq12:
   24 c226b800 e33bd000    0     0     0 0000204 [IWAIT] irq11:
   23 c226ba00 e33be000    0     0     0 0000204 [IWAIT] irq10:
   22 c226bc00 e33bf000    0     0     0 0000204 [IWAIT] irq9: acpi0
   21 c226be00 e33c0000    0     0     0 0000204 [IWAIT] irq8: rtc
   20 c22b3000 e4de2000    0     0     0 0000204 [IWAIT] irq7: ppc0
   19 c22b3200 e4de3000    0     0     0 0000204 [IWAIT] irq6: fdc0
   18 c22b3400 e4de4000    0     0     0 0000204 [IWAIT] irq5:
   17 c2261000 e3357000    0     0     0 0000204 [IWAIT] irq4: sio0
   16 c2261200 e3394000    0     0     0 0000204 [IWAIT] irq3: sio1
   15 c2261400 e3395000    0     0     0 0000204 [IWAIT] irq1: atkbd0
   14 c2261600 e3396000    0     0     0 000020c [Can run] idle: cpu0
   13 c2261800 e3397000    0     0     0 000020c [CPU 1] idle: cpu1
   12 c2261a00 e3398000    0     0     0 000020c [Can run] idle: cpu2
   11 c2261c00 e3399000    0     0     0 000020c [CPU 3] idle: cpu3
    1 c2261e00 e339a000    0     0     1 0004200 [SLPQ wait 0xc2261e00][SLP] init
   10 c226b000 e339b000    0     0     0 0000204 [SLPQ ktrace 0xc08bdc58][SLP] ktrace
    0 c08ba6c0 c0c1f000    0     0     0 0000200 [SLPQ sched 0xc08ba6c0][SLP] swapper
db> trace 615
sched_switch(c2932900,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c2b3c9dc,0,ef231bac,c060f686,c2b3c9dc) at sleepq_switch+0xe0
sleepq_wait_sig(c2b3c9dc,0,100,c0802936,34a) at sleepq_wait_sig+0xc
msleep(c2b3c9dc,c2b3c9ac,158,c0802bbc,0) at msleep+0x2da
sbwait(c2b3c994,c2b3c944,c2b3c944,c2b3c9ac,0) at sbwait+0x4e
sosend(c2b3c8dc,0,ef231c88,0,0) at sosend+0x33c
soo_write(c271a550,ef231c88,c2adf800,0,c2932900) at soo_write+0x46
dofilewrite(c2932900,c271a550,3,bfbfcb50,2000) at dofilewrite+0xa8
write(c2932900,ef231d14,3,5,296) at write+0x39
syscall(2f,2f,2f,2000,809a044) at syscall+0x227
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x2812558b, esp = 0xbfbfca4c, ebp = 0xbfbfca68 ---
db> show locks 615
db> trace 581
sched_switch(c237a780,c2268300,6) at sched_switch+0x16f
mi_switch(6,c2268300,c2268450,c2268300,e4e70cc8) at mi_switch+0x264
maybe_preempt(c2268300) at maybe_preempt+0x156
sched_add(c2268300,4,c2260d00,c2268300,c22b3a00) at sched_add+0x153
setrunqueue(c2268300,4) at setrunqueue+0xab
ithread_schedule(c2260d00,17,c237a780,2819c5ec,80e2300) at ithread_schedule+0xb3
intr_execute_handlers(c225a658,e4e70d44,17,bfbfcba8,c0780c83) at intr_execute_handlers+0xf5
lapic_handle_intr(47) at lapic_handle_intr+0x2e
Xapic_isr2() at Xapic_isr2+0x33
--- interrupt, eip = 0x2818ead2, esp = 0xbfbfcb74, ebp = 0xbfbfcba8 ---
db> show locks 581
db> trace 580
sched_switch(c26f0780,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
turnstile_wait(c08ec02c,c26ef780,c08ec02c,2,c07fbabd,21e) at turnstile_wait+0x2f8
_mtx_lock_sleep(c08ec02c,c26f0780,0,c08091ed,26f) at _mtx_lock_sleep+0x142
_mtx_lock_flags(c08ec02c,0,c08091ed,26f,bfbfcbd0) at _mtx_lock_flags+0x85
tcp_usr_send(c2b44ca8,4,c2c3bc00,0,0) at tcp_usr_send+0x2c
sosend(c2b44ca8,0,ecf6fc88,c2c3bc00,0) at sosend+0x5e7
soo_write(c2719110,ecf6fc88,c2adf880,0,c26f0780) at soo_write+0x46
dofilewrite(c26f0780,c2719110,3,bfbfcbd0,2000) at dofilewrite+0xa8
write(c26f0780,ecf6fd14,3,a,292) at write+0x39
syscall(2f,2f,2f,2000,809a044) at syscall+0x227
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x2812558b, esp = 0xbfbfcacc, ebp = 0xbfbfcae8 ---
db> show locks 580
db> trace 578
sched_switch(c26f0300,c2268300,6) at sched_switch+0x16f
mi_switch(6,c2268300,c2268450,c2268300,ecf66cc8) at mi_switch+0x264
maybe_preempt(c2268300) at maybe_preempt+0x156
sched_add(c2268300,4,c2260d00,c2268300,c22b3a00) at sched_add+0x153
setrunqueue(c2268300,4) at setrunqueue+0xab
ithread_schedule(c2260d00,17,c26f0300,282085bc,80bf034) at ithread_schedule+0xb3
intr_execute_handlers(c225a658,ecf66d44,17,bfbfec98,c0780c83) at intr_execute_handlers+0xf5
lapic_handle_intr(47) at lapic_handle_intr+0x2e
Xapic_isr2() at Xapic_isr2+0x33
--- interrupt, eip = 0x28200047, esp = 0xbfbfe870, ebp = 0xbfbfec98 ---
db> show locks 578
db> trace 574
sched_switch(c2379c00,c2268300,6) at sched_switch+0x16f
mi_switch(6,c2268300,c2268450,c2268300,e4e5baa4) at mi_switch+0x264
maybe_preempt(c2268300) at maybe_preempt+0x156
sched_add(c2268300,4,c2260d00,c2268300,c22b3a00) at sched_add+0x153
setrunqueue(c2268300,4) at setrunqueue+0xab
ithread_schedule(c2260d00,17,c2379c00,c2268600,c08ec02c) at ithread_schedule+0xb3
intr_execute_handlers(c225a658,e4e5bb20,17,e4e5bb70,c0780c83) at intr_execute_handlers+0xf5
lapic_handle_intr(47) at lapic_handle_intr+0x2e
Xapic_isr2() at Xapic_isr2+0x33
--- interrupt, eip = 0xc06022d8, esp = 0xe4e5bb64, ebp = 0xe4e5bb70 ---
_mtx_lock_sleep(c08ec02c,c2379c00,0,c08091ed,26f) at _mtx_lock_sleep+0xf4
_mtx_lock_flags(c08ec02c,0,c08091ed,26f,bfbfd3d0) at _mtx_lock_flags+0x85
tcp_usr_send(c2acaa20,4,c2c20b00,0,0) at tcp_usr_send+0x2c
sosend(c2acaa20,0,e4e5bc88,c2c20b00,0) at sosend+0x5e7
soo_write(c271a50c,e4e5bc88,c2ac6d80,0,c2379c00) at soo_write+0x46
dofilewrite(c2379c00,c271a50c,3,bfbfcbd0,2000) at dofilewrite+0xa8
write(c2379c00,e4e5bd14,3,13,292) at write+0x39
syscall(2f,809002f,bfbf002f,2000,809a044) at syscall+0x227
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x2812558b, esp = 0xbfbfcacc, ebp = 0xbfbfcae8 ---
db> show locks 574
db> trace 572
sched_switch(c237ad80,0,2) at sched_switch+0x16f
mi_switch(2,0,c237ad80,b4,c08be1e0,0,c07ff747,f4) at mi_switch+0x264
ast(e4e7cd48) at ast+0x2d9
doreti_ast() at doreti_ast+0x17
db> trace 570
sched_switch(c2798480,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
turnstile_wait(c08ec02c,c26ef780,c08ec02c,2,c07fbabd,21e) at turnstile_wait+0x2f8
_mtx_lock_sleep(c08ec02c,c2798480,0,c08091ed,26f) at _mtx_lock_sleep+0x142
_mtx_lock_flags(c08ec02c,0,c08091ed,26f,bfbfd3d0) at _mtx_lock_flags+0x85
tcp_usr_send(c2b44144,0,c2c20600,0,0) at tcp_usr_send+0x2c
sosend(c2b44144,0,ef1c5c88,c2c20600,0) at sosend+0x5e7
soo_write(c2b07110,ef1c5c88,c2ac6c80,0,c2798480) at soo_write+0x46
dofilewrite(c2798480,c2b07110,3,bfbfcbd0,2000) at dofilewrite+0xa8
write(c2798480,ef1c5d14,3,15,292) at write+0x39
syscall(2f,2819002f,bfbf002f,2000,809a044) at syscall+0x227
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x2812558b, esp = 0xbfbfcacc, ebp = 0xbfbfcae8 ---
db> show locks 572
db> trace 53
sched_switch(c22cd180,0,1) at sched_switch+0x16f
mi_switch(1,0) at mi_switch+0x264
ithread_loop(c24a1e80,e4e1ad48,c24a1e80,c05f7d50,0) at ithread_loop+0x22d
fork_exit(c05f7d50,c24a1e80,e4e1ad48) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe4e1ad7c, ebp = 0 ---
db> show locks 53
db> trace 38
kdb_enter(c07fc72c) at kdb_enter+0x2b
panic(c0815e8e,1,2,c22583c0,c2821100) at panic+0x127
uma_zfree_arg(c101fc60,c2821100,0) at uma_zfree_arg+0xa5
mb_free_ext(c2821100) at mb_free_ext+0x39
m_freem(c2821100,0,0,1,1) at m_freem+0x21
tcp_input(c2821100,14,c2821100,0,0) at tcp_input+0x2d1c
ip_input(c2821100) at ip_input+0x50d
netisr_processqueue(c08eae58) at netisr_processqueue+0x6e
swi_net(0) at swi_net+0xbe
ithread_loop(c2260c00,e3384d48,c2260c00,c05f7d50,0) at ithread_loop+0x124
fork_exit(c05f7d50,c2260c00,e3384d48) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe3384d7c, ebp = 0 ---
db> show locks 38
exclusive sleep mutex UMA pcpu r = 0 (0xc08f8548) locked _at_ vm/uma_core.c:2215
exclusive sleep mutex inp (tcpinp) r = 0 (0xc2b4d2ac) locked _at_ netinet/tcp_input.c:743
exclusive sleep mutex tcp r = 0 (0xc08ec02c) locked _at_ netinet/tcp_input.c:617
db> trace 36
sched_switch(c0780fc1,c090e5a0,e3370018,c2260010,10) at sched_switch+0x16f
*** error reading from address e3370014 ***


(kgdb) bt
#0  doadump () at pcpu.h:159
#1  0xc04601ba in db_fncall (dummy1=0, dummy2=0, dummy3=-1064327584, 
    dummy4=0xe33849d0 "ìI8ã$!`À`¦\217À`¦\217ÀìI8ãø\003")
    at ../../../ddb/db_command.c:531
#2  0xc045ffc8 in db_command (last_cmdp=0xc08a1744, cmd_table=0x0, 
    aux_cmd_tablep=0xc082161c, aux_cmd_tablep_end=0xc0821638)
    at ../../../ddb/db_command.c:349
#3  0xc0460090 in db_command_loop () at ../../../ddb/db_command.c:455
#4  0xc0461bf5 in db_trap (type=3, code=0) at ../../../ddb/db_main.c:221
#5  0xc0620368 in kdb_trap (type=3, code=0, tf=0xe3384b14)
    at ../../../kern/subr_kdb.c:419
#6  0xc0792120 in trap (frame=
      {tf_fs = -482869224, tf_es = -1067319280, tf_ds = -1065418736,
tf_edi = -1065263474, tf_esi = 1, tf_ebp = -482849964, tf_isp =
-482849984, tf_ebx = -482849920, tf_edx = 0, tf_ecx = -1056882688, tf_eax
= 18, tf_trapno = 3, tf_err = 0, tf_eip = -1067319089, tf_cs = 8,
tf_eflags = 658, tf_esp = -482849932, tf_ss = -1067409941}) at
../../../i386/i386/trap.c:576
#7  0xc078087a in calltrap () at ../../../i386/i386/exception.s:140
#8  0xe3380018 in ?? ()
#9  0xc0620010 in kdb_alt_break (key=0, state=0x0)
    at ../../../kern/subr_kdb.c:179
#10 0xc0609deb in panic (
    fmt=0xc0815e8e "uma_zfree: Freeing to non free bucket index.")

---Type <return> to continue, or q <return> to quit---
    at ../../../kern/kern_shutdown.c:525
#11 0xc075b841 in uma_zfree_arg (zone=0xc101fc60, item=0xc2821100,
udata=0x0)
    at ../../../vm/uma_core.c:2228
#12 0xc063d50d in mb_free_ext (m=0xc2821100) at uma.h:302
#13 0xc063d425 in m_freem (mb=0x0) at mbuf.h:397
#14 0xc0693fa8 in tcp_input (m=0xc2821100, off0=686)
    at ../../../netinet/tcp_input.c:2435
#15 0xc068bb29 in ip_input (m=0xc2821100) at
../../../netinet/ip_input.c:739
#16 0xc067457a in netisr_processqueue (ni=0xc08eae58)
    at ../../../net/netisr.c:235
#17 0xc0674922 in swi_net (dummy=0x0) at ../../../net/netisr.c:348
#18 0xc05f7e74 in ithread_loop (arg=0xc2260c00)
    at ../../../kern/kern_intr.c:547
#19 0xc05f7284 in fork_exit (callout=0xc05f7d50 <ithread_loop>, 
    arg=0xc2260c00, frame=0xe3384d48) at ../../../kern/kern_fork.c:807
#20 0xc07808dc in fork_trampoline () at ../../../i386/i386/exception.s:209
(kgdb) frame 11
#11 0xc075b841 in uma_zfree_arg (zone=0xc101fc60, item=0xc2821100,
udata=0x0)
    at ../../../vm/uma_core.c:2228
2228                            KASSERT(bucket->ub_bucket[bucket->ub_cnt]
== NULL,
(kgdb) print bucket
$2 = 0xc2b38624
(kgdb) print *bucket
$3 = {ub_link = {le_next = 0x0, le_prev = 0xc101fc78}, ub_cnt = 78, 
  ub_entries = 128, ub_bucket = 0xc2b38630}
(kgdb) print bucket->ub_bucket[bucket->ub_cnt]
$4 = (void *) 0xc2ca5900
(kgdb) inspect *zone
$5 = {uz_name = 0xc07e455f "Packet", uz_lock = 0xc22583c8, 
  uz_keg = 0xc22583c0, uz_link = {le_next = 0x0, le_prev = 0xc101f9ac}, 
  uz_full_bucket = {lh_first = 0xc280ca3c}, uz_free_bucket = {
    lh_first = 0x0}, uz_ctor = 0xc0601310 <mb_ctor_pack>, 
  uz_dtor = 0xc060121c <mb_dtor_pack>, uz_init = 0xc06012a8
<mb_init_pack>, 
  uz_fini = 0xc06012e4 <mb_fini_pack>, uz_allocs = 16842, uz_fills = 0, 
  uz_count = 128, uz_cpu = {{uc_freebucket = 0xc2988418, 
      uc_allocbucket = 0xc286ba3c, uc_allocs = 133}}}
Received on Mon Oct 18 2004 - 20:13:09 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:18 UTC