Re: r194546 amd64: kernel panic in tcp_sack.c

Lawrence Stewart wrote:
> Ok. I'm working on a patch to address a different TCP/SACK issue, but 
> it may in fact be partially relevant to the cause of your panic... 
> can't promise when I'll be able to take a close look at this but I'm 
> aware of it now so that's a start. If you run into it again or find 
> the trigger for the panic, please let me know.
Reproduced. I don't know what was the trigger this time. The system runs 
lighttpd, fastcgi python and php, mysql and postgresql.
Also, in this core somehow everything looks quite similar (looks like 
another page fault but with panic() called prior to getting into fatal 
trap 12) to my two earlier dumps:
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008777.html
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008781.html
so maybe it's not really a problem with tcp_sack.c or netisr.c.

Should I attach the core.txt as well?


Unread portion of the kernel message buffer:
panic: tcp_sack_globalholes >= 0
cpuid = 0
KDB: enter: panic
Physical memory: 2014 MB
Dumping 1622 MB: 1607 1591 1575 1559 1543 1527 1511 1495 1479 1463 1447 
1431 1415 1399 1383 1367 1351 1335 1319 1303 1287 1271 1255 1239 1223 
1207 1191 1175 1159 1143 1127 1111 1095 1079 1063 1047 1031 1015 999 983 
967 951 935 919 903 887 871 855 839 823 807 791 775 759 743 727 711 695 
679 663 647 631 615 599 583 567 551 535 519 503 487 471 455 439 423 407 
391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 
103 87 71 55 39 23 7

Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from 
/boot/kernel/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_mirror.ko
Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from 
/boot/kernel/coretemp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/coretemp.ko
Reading symbols from /boot/kernel/ahci.ko...Reading symbols from 
/boot/kernel/ahci.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ahci.ko
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from 
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from 
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from 
/boot/kernel/nullfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/nullfs.ko
Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from 
/boot/kernel/fdescfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
Reading symbols from /boot/kernel/blank_saver.ko...Reading symbols from 
/boot/kernel/blank_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/blank_saver.ko
#0  doadump () at pcpu.h:223
223             __asm __volatile("movq %%gs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:223
#1  0xffffffff801f909c in db_fncall (dummy1=Variable "dummy1" is not 
available.
) at /usr/src/sys/ddb/db_command.c:548
#2  0xffffffff801f93d1 in db_command (last_cmdp=0xffffffff80bec5e0, 
cmd_table=Variable "cmd_table" is not available.
) at /usr/src/sys/ddb/db_command.c:445
#3  0xffffffff801f9620 in db_command_loop () at 
/usr/src/sys/ddb/db_command.c:498
#4  0xffffffff801fb5c9 in db_trap (type=Variable "type" is not available.
) at /usr/src/sys/ddb/db_main.c:229
#5  0xffffffff805c92e5 in kdb_trap (type=3, code=0, 
tf=0xffffff8000031900) at /usr/src/sys/kern/subr_kdb.c:534
#6  0xffffffff80862ad1 in trap (frame=0xffffff8000031900) at 
/usr/src/sys/amd64/amd64/trap.c:613
#7  0xffffffff80848f03 in calltrap () at 
/usr/src/sys/amd64/amd64/exception.S:223
#8  0xffffffff805c94bd in kdb_enter (why=0xffffffff8095b989 "panic", 
msg=0xa <Address 0xa out of bounds>) at cpufunc.h:63
#9  0xffffffff80599ecb in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:558
#10 0xffffffff80709537 in tcp_sackhole_remove (tp=0xffffffff81020fe0, 
hole=0xa) at /usr/src/sys/netinet/tcp_sack.c:295
#11 0xffffffff80709598 in tcp_free_sackholes (tp=0xffffff0030168b60) at 
/usr/src/sys/netinet/tcp_sack.c:554
#12 0xffffffff807100f3 in tcp_timer_rexmt (xtp=Variable "xtp" is not 
available.
) at /usr/src/sys/netinet/tcp_timer.c:482
#13 0xffffffff805ac041 in softclock (arg=Variable "arg" is not available.
) at /usr/src/sys/kern/kern_timeout.c:411
#14 0xffffffff805735d8 in intr_event_execute_handlers (p=Variable "p" is 
not available.
) at /usr/src/sys/kern/kern_intr.c:1145
#15 0xffffffff80574232 in ithread_loop (arg=0xffffff000131d640) at 
/usr/src/sys/kern/kern_intr.c:1158
#16 0xffffffff8057153a in fork_exit (callout=0xffffffff80574180 
<ithread_loop>, arg=0xffffff000131d640, frame=0xffffff8000031c90)
    at /usr/src/sys/kern/kern_fork.c:842
#17 0xffffffff8084938e in fork_trampoline () at 
/usr/src/sys/amd64/amd64/exception.S:552
#18 0x0000000000000000 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000001 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000eb9000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0xffffffff80c28ec0 in affinity ()
#45 0xffffffff80c28ec0 in affinity ()
#46 0xffffff0001321390 in ?? ()
#47 0xffffff8000031b90 in ?? ()
#48 0xffffff8000031b48 in ?? ()
#49 0xffffff0001332390 in ?? ()
#50 0xffffffff805bccf0 in sched_switch (td=0xffffff000131d640, 
newtd=0xffffffff80574180, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1858
Previous frame inner to this frame (corrupt stack?)

--
Kamigishi Rei
KREI-RIPE