Re: another trap 12 while in kernel mode (now with trace)

From: Bjoern A. Zeeb <bzeeb-lists_at_lists.zabbadoz.net>
Date: Wed, 19 Nov 2003 22:30:44 +0000 (UTC)
On Wed, 19 Nov 2003, Robert Watson wrote:

> You should still be able to turn the bad instruction pointer into at least
> a function name using nm on your compiled kernel.  Run nm on the kernel
> binary, and search through it until you find the symbols just before
> (lower) and just after (higher) than the value listed for the instruction
> pointer below.  That would at least tell us what function the fault is
> occuring in.  I suspect to get anywhere useful, we'll need the full
> debugging results, but it's probably a useful start, and it will allow us
> to be sure the panics you might get after you have debugging support
> compiled in are the same failure mode.


Ok, I could trigger it or s.th. very related by doing this:

crisco# ps ax | grep rac | grep -v grep
  348  ??  Is     0:04.50 /usr/local/sbin/racoon -f /usr/local/etc/racoon/racoon.conf
crisco# kill -9 348


Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0xdeadc0df
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc055088c
stack pointer           = 0x10:0xc9b5c8dc
frame pointer           = 0x10:0xc9b5c8e4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 485 (sshd)
kernel: type 12 trap, code=0
Stopped at      key_cmpspidx_withmask+0x2c:     movb    0x1(%ebx),%al
db> trace
key_cmpspidx_withmask(deadc0de,c9b5c914) at key_cmpspidx_withmask+0x2c
key_allocsp(0,c9b5c914,2,16000210,c1426f0a) at key_allocsp+0x8b
ipsec4_getpolicybyaddr(c2fa7c00,2,0,c9b5ca88,0) at
ipsec4_getpolicybyaddr+0x69
ip_output(c2fa7c00,0,c3a36b00,0,0) at ip_output+0x770
tcp_output(c3a386f4,0,c3a36ac4,0,30) at tcp_output+0xbf0
tcp_usr_send(c3a091e0,0,c2fa7700,0,0) at tcp_usr_send+0x18b
sosend(c3a091e0,0,c9b5cc88,c2fa7700,0) at sosend+0x405
soo_write(c3a9b7b4,c9b5cc88,c3a58c80,0,c2f83a00) at soo_write+0x37
dofilewrite(c2f83a00,c3a9b7b4,5,807a000,30) at dofilewrite+0x7b
write(c2f83a00,c9b5cd14,3,ab,286) at write+0x3e
syscall(2f,2f,2f,806cc08,8076860) at syscall+0x202
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (4, FreeBSD ELF32, write), eip = 0x28258b6f, esp = 0xbfbfe4cc, ebp = 0xbfbfe4f8 ---
db> where
key_cmpspidx_withmask(deadc0de,c9b5c914) at key_cmpspidx_withmask+0x2c
key_allocsp(0,c9b5c914,2,16000210,c1426f0a) at key_allocsp+0x8b
ipsec4_getpolicybyaddr(c2fa7c00,2,0,c9b5ca88,0) at ipsec4_getpolicybyaddr+0x69
ip_output(c2fa7c00,0,c3a36b00,0,0) at ip_output+0x770
tcp_output(c3a386f4,0,c3a36ac4,0,30) at tcp_output+0xbf0
tcp_usr_send(c3a091e0,0,c2fa7700,0,0) at tcp_usr_send+0x18b
sosend(c3a091e0,0,c9b5cc88,c2fa7700,0) at sosend+0x405
soo_write(c3a9b7b4,c9b5cc88,c3a58c80,0,c2f83a00) at soo_write+0x37
dofilewrite(c2f83a00,c3a9b7b4,5,807a000,30) at dofilewrite+0x7b
write(c2f83a00,c9b5cd14,3,ab,286) at write+0x3e
syscall(2f,2f,2f,806cc08,8076860) at syscall+0x202
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (4, FreeBSD ELF32, write), eip = 0x28258b6f, esp = 0xbfbfe4cc, ebp = 0xbfbfe4f8 ---
db> show locks
exclusive sleep mutex inp r = 0 (0xc3a36b74) locked _at_ HEAD/compile-20031119-1814/sys/netinet/tcp_usrreq.c:670
exclusive sleep mutex tcp r = 0 (0xc063bc6c) locked _at_ HEAD/compile-20031119-1814/sys/netinet/tcp_usrreq.c:653
exclusive sleep mutex Giant r = 0 (0xc0610680) locked _at_ HEAD/compile-20031119-1814/sys/vm/vm_fault.c:223
db> show reg
cs                 0x8
ds                0x10
es                0x10
fs                0x18
ss                0x10
eax                  0
ecx                  0
edx                  0
ebx         0xdeadc0de
esp         0xc9b5c8dc
ebp         0xc9b5c8e4
esi         0xc9b5c914
edi                0x2
eip         0xc055088c  key_cmpspidx_withmask+0x2c
efl            0x10286
dr0                  0
dr1                  0
dr2                  0
dr3                  0
dr4         0xffff0ff0
dr5              0x400
dr6         0xffff0ff0
dr7              0x400
key_cmpspidx_withmask+0x2c:     movb    0x1(%ebx),%al
db> ps
  pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
  804 c3adcc5c ca470000    0   455   804 0004002 [SLP]ttyin 0xc3976a10] csh
  550 c3ae0388 ca474000    0   549   550 0004002 [RUNQ] csh
  549 c3adc710 ca46d000 1060   486   549 0004102 [SLP]wait 0xc3adc710] su
  486 c3936c5c ca06f000 1060   485   486 0004002 [SLP]pause 0xc39e5000] csh
  485 c39af000 ca08f000 1060   482   482 0000100 [CPU 0] sshd
  482 c3a3f388 ca407000    0   284   482 0000100 [SLP]sbwait 0xc3ac442c] sshd
  460 c39b1e20 ca0c4000   25   458   458 0004100 [CV]select 0xc0638a04] qmgr
  459 c39b18d4 ca0c1000   25   458   458 0004100 [CV]select 0xc0638a04] pickup
  458 c39b1710 ca0c0000    0     1   458 0004100 [CV]select 0xc0638a04] master
  455 c39afa98 ca095000    0     1   455 0004102 [SLP]wait 0xc39afa98] login
  384 c39b1388 ca0be000    0     1   384 0000000 [CV]select 0xc0638a04] inetd
  342 c39b154c ca0bf000    0     1   342 0000000 [CV]select 0xc0638a04] ppp
  339 c39af388 ca091000    0     1   339 0000000 [CV]select 0xc0638a04] ppp
  329 c3a3f710 ca409000    0     1   329 0000000 [CV]select 0xc0638a04] dhcrelay
  305 c39b1c5c ca0c3000    0     1   305 0000000 [SLP]nanslp 0xc061251c] cron
  284 c3936a98 ca06e000    0     1   284 0000100 [CV]select 0xc0638a04] sshd
  273 c3936e20 ca070000    0     1   273 0000000 [CV]select 0xc0638a04] ntpd
  231 c39af54c ca092000    0     1   231 0000000 [CV]select 0xc0638a04] syslogd
  158 c39af710 ca093000    0     1   158 0000000 [CV]select 0xc0638a04] natd
   36 c39afc5c ca0ba000    0     0     0 0000204 [SLP]mdwait 0xc3986c00] md0
   35 c39afe20 ca0bb000    0     0     0 0000204 [SLP]vlruwt 0xc39afe20] vnlru
   34 c39b1000 ca0bc000    0     0     0 0000204 [SLP]syncer 0xc0611ee0] syncer
    9 c392754c ca03e000    0     0     0 0000204 [SLP]psleep 0xc0638e6c] bufdaemon
    8 c3927710 ca03f000    0     0     0 000020c [SLP]pgzero 0xc0649268] pagezero
    7 c39278d4 ca040000    0     0     0 0000204 [SLP]psleep 0xc06492c0] vmdaemon
    6 c3927a98 ca041000    0     0     0 0000204 [SLP]psleep 0xc06492ac] pagedaemon
   33 c3927c5c ca042000    0     0     0 0000204 [IWAIT] swi0: tty:sio
    5 c3927e20 ca043000    0     0     0 0000204 [SLP]tqthr 0xc0613108] taskqueue
   32 c3936000 ca044000    0     0     0 0000204 new [IWAIT] swi6:+
   31 c39361c4 ca069000    0     0     0 0000204 new [IWAIT] swi7: task queue
   30 c3936388 ca06a000    0     0     0 0000204 new [IWAIT] swi5:+
   29 c393654c ca06b000    0     0     0 0000204 [SLP]- 0xc06bd500] random
    4 c3936710 ca06c000    0     0     0 0000204 [SLP]- 0xc060cc60] g_down
    3 c39368d4 ca06d000    0     0     0 0000204 [SLP]- 0xc060cc5c] g_up
    2 c2f821c4 c9b3e000    0     0     0 0000204 [SLP]- 0xc060cc54] g_event
   28 c2f82388 c9b3f000    0     0     0 0000204 [IWAIT] swi1: net
   27 c2f8254c c9b40000    0     0     0 0000204 new [IWAIT] swi4: vm
   26 c2f82710 c9b41000    0     0     0 000020c [RUNQ] swi8: tty:sio clock
   25 c2f828d4 c9b66000    0     0     0 0000204 [IWAIT] irq15: fxp2 ata1
   24 c2f82a98 c9b67000    0     0     0 0000204 new [IWAIT] irq14: ata0
   23 c2f82c5c c9b68000    0     0     0 0000204 new [IWAIT] irq13:
   22 c2f82e20 c9b69000    0     0     0 0000204 [IWAIT] irq12: fxp1
   21 c3927000 ca03b000    0     0     0 0000204 [IWAIT] irq11: ed0
   20 c39271c4 ca03c000    0     0     0 0000204 new [IWAIT] irq10:
   19 c3927388 ca03d000    0     0     0 0000204 new [IWAIT] irq9:
   18 c2f7a000 c9aec000    0     0     0 0000204 new [IWAIT] irq8: rtc
   17 c2f7a1c4 c9b35000    0     0     0 0000204 [IWAIT] irq7: fxp0
   16 c2f7a388 c9b36000    0     0     0 0000204 new [IWAIT] irq6:
   15 c2f7a54c c9b37000    0     0     0 0000204 new [IWAIT] irq5:
   14 c2f7a710 c9b38000    0     0     0 0000204 new [IWAIT] irq4: sio0
   13 c2f7a8d4 c9b39000    0     0     0 0000204 new [IWAIT] irq3: sio1
   12 c2f7aa98 c9b3a000    0     0     0 0000204 new [IWAIT] irq1:
   11 c2f7ac5c c9b3b000    0     0     0 0000204 new [IWAIT] irq0: clk
   10 c2f7ae20 c9b3c000    0     0     0 000020c [Can run] idle
    1 c2f82000 c9b3d000    0     0     1 0004200 [SLP]wait 0xc2f82000] init
    0 c060cd00 c281f000    0     0     0 0000200 [SLP]sched 0xc060cd00] swapper



bz_at_noc:...obj.../HEAD/compile-20031119-1814/sys/ZAB1-2003111901-CF-DEBUG> gdb -k kernel.debug
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
(kgdb) l *0xc055088c
0xc055088c is in key_cmpspidx_withmask (HEAD/compile-20031119-1814/sys/netkey/key.c:4105).
4100                    return 1;
4101
4102            if (spidx0 == NULL || spidx1 == NULL)
4103                    return 0;
4104
4105            if (spidx0->src.ss_family != spidx1->src.ss_family ||
4106                spidx0->dst.ss_family != spidx1->dst.ss_family ||
4107                spidx0->src.ss_len != spidx1->src.ss_len ||
4108                spidx0->dst.ss_len != spidx1->dst.ss_len)
4109                    return 0;



-- 
Bjoern A. Zeeb				bzeeb at Zabbadoz dot NeT
56 69 73 69 74				http://www.zabbadoz.net/
Received on Wed Nov 19 2003 - 13:31:14 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:30 UTC