Re: Fatal Trap 12: Page fault while in kernel mode (racoon/amd64/5.3-RELEASE-p4)

From: Matthew Sullivan <matthew_at_uq.edu.au>
Date: Fri, 14 Jan 2005 11:47:26 +1000
Phil Schulz wrote:

> [lost the original mail...]
>
> On 01/13/05 18:41, Doug White wrote:
>
>> On Thu, 13 Jan 2005, Matthew Sullivan wrote:
>>
>>> I'm going to have to put this machine into production within the next 7
>>> days so any help would be really great, also any extra info anyone
>>> requires is available.  As I said in my last this is 100% reproducable.
>>> Dumps are not available - calling panic will lock the system solid.
>>> Calling boot(0) seems to work fine though...
>>
>
> When you are in the debugger, can you type "call doadump"?

Yes thanks, between you mailing and my last post I spotted this call in 
someone elses woes an took immediate advantage ;-)

Results are at: http://www.au.sorbs.net/~matthew/freebsd/ along with the 
core, the kernel, the symbols etc etc etc.. ;-)

> Last time I tried to debug a panic, I've had problems with panics not 
> generating a core dump as well. Calling the doadump() function 
> manually worked, though.

Gets me here:

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x39
fault code              = supervisor write, page not present
instruction pointer     = 0x8:0xffffffff80307a70
stack pointer           = 0x10:0xffffffff93cc0860
frame pointer           = 0x10:0xffffffff93cc0960
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 480 (racoon)
[thread 100068]
Stopped at      keydb_newsecasvar+0x100:        decl    %ecx
db> w
Nothing written.
db> where
keydb_newsecasvar() at keydb_newsecasvar+0x100
raw_usend() at raw_usend+0x60
key_send() at key_send+0xa
sosend() at sosend+0x626
kern_sendit() at kern_sendit+0x113
sendit() at sendit+0x5f
sendto() at sendto+0x4d
syscall() at syscall+0x50c
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x800a63da8, rsp = 0x7fffffffec38, rbp = 0x2 ---
db> call doadump
Dumping 479 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464
Dump complete
0xf


Then we can proceed ;-)

(kgdb) file /usr/obj/usr/src/sys/DESPERADO/kernel.debug
Reading symbols from /usr/obj/usr/src/sys/DESPERADO/kernel.debug...done.
(kgdb) where
#0  doadump () at pcpu.h:167
#1  0xffffffff80172736 in db_fncall (dummy1=0, dummy2=0, dummy3=0, 
dummy4=0x0) at /usr/src/sys/ddb/db_command.c:531
#2  0xffffffff80172bc5 in db_command_loop () at 
/usr/src/sys/ddb/db_command.c:349
#3  0xffffffff80174a53 in db_trap (type=-1815345680, code=0) at 
/usr/src/sys/ddb/db_main.c:221
#4  0xffffffff8023070b in kdb_trap (type=12, code=0, 
tf=0xffffffff93cc07b0) at /usr/src/sys/kern/subr_kdb.c:418
#5  0xffffffff80371dae in trap_fatal (frame=0xffffffff93cc07b0, 
eva=18446742974681318688)
    at /usr/src/sys/amd64/amd64/trap.c:626
#6  0xffffffff80372143 in trap_pfault (frame=0xffffffff93cc07b0, 
usermode=0) at /usr/src/sys/amd64/amd64/trap.c:554
#7  0xffffffff803723a4 in trap (frame=
      {tf_rdi = -1099028463104, tf_rsi = 640, tf_rdx = -1815344784, 
tf_rcx = -1815344833, tf_r8 = 160, tf_r9 = -1099028232928
, tf_rax = -1815345008, tf_rbx = -2144306579, tf_rbp = -1815344800, 
tf_r10 = -2142160512, tf_r11 = -1815344624, tf_r12 = 57,
tf_r13 = 0, tf_r14 = 0, tf_r15 = -1099140303240, tf_trapno = 12, tf_addr 
= 57, tf_flags = -1099132378656, tf_err = 2, tf_rip
= -2144306576, tf_cs = 8, tf_rflags = 66054, tf_rsp = -1815345040, tf_ss 
= 16}) at /usr/src/sys/amd64/amd64/trap.c:333
#8  0xffffffff80361bab in calltrap () at 
/usr/src/sys/amd64/amd64/exception.S:171
#9  0xffffff001ccc8200 in ?? ()
#10 0x0000000000000280 in ?? ()
#11 0xffffffff93cc0970 in ?? ()
#12 0xffffffff93cc093f in ?? ()
#13 0x00000000000000a0 in ?? ()
#14 0xffffff001cd00520 in ?? ()
#15 0xffffffff93cc0890 in ?? ()
#16 0xffffffff80307a6d in keydb_newsecasvar () at 
/usr/src/sys/netkey/keydb.c:187
#17 0xffffffff8029cfc0 in raw_usend (so=0x0, flags=0, m=0x0, nam=0x0, 
control=0x0, td=0x0)
    at /usr/src/sys/net/raw_usrreq.c:263
#18 0xffffffff8030845a in key_send (so=0x0, flags=0, m=0x0, nam=0x0, 
control=0x0, td=0x0)
    at /usr/src/sys/netkey/keysock.c:442
#19 0xffffffff80253bc6 in sosend (so=0xffffff001621f678, addr=0x0, 
uio=0xffffffff93cc0a80, top=0xffffff001ccc8200,
    control=0x0, flags=0, td=0xffffff001cd00520) at 
/usr/src/sys/kern/uipc_socket.c:815
#20 0xffffffff8025ba73 in kern_sendit (td=0xffffff001cd00520, s=4, 
mp=0xffffffff93cc0b50, flags=0, control=0x0)
    at /usr/src/sys/kern/uipc_syscalls.c:738
#21 0xffffffff8025ca5f in sendit (td=0xffffff001cd00520, s=4, 
mp=0xffffffff93cc0b50, flags=0)
    at /usr/src/sys/kern/uipc_syscalls.c:682
#22 0xffffffff8025cbed in sendto (td=0x0, uap=0x0) at 
/usr/src/sys/kern/uipc_syscalls.c:795
#23 0xffffffff80372b6c in syscall (frame=
      {tf_rdi = 4, tf_rsi = 5660864, tf_rdx = 16, tf_rcx = 0, tf_r8 = 0, 
tf_r9 = 0, tf_rax = 133, tf_rbx = 5660880, tf_rbp =
2, tf_r10 = -2141993928, tf_r11 = 514, tf_r12 = 5660864, tf_r13 = 16, 
tf_r14 = 7, tf_r15 = 4, tf_trapno = 12, tf_addr = 42840
00, tf_flags = 0, tf_err = 2, tf_rip = 34370633128, tf_cs = 43, 
tf_rflags = 514, tf_rsp = 140737488350264, tf_ss = 35})
    at /usr/src/sys/amd64/amd64/trap.c:763
#24 0xffffffff80361ce8 in Xfast_syscall () at 
/usr/src/sys/amd64/amd64/exception.S:248
.
.
.
(kgdb) frame 16
#16 0xffffffff80307a6d in keydb_newsecasvar () at 
/usr/src/sys/netkey/keydb.c:187
187             p->id = said;
(kgdb) info loc
p = (struct secasvar *) 0x39
q = (struct secasvar *) 0xffffffff80307a6d
said = 0
(kgdb) l *0xffffffff80307a70
0xffffffff80307a70 is in keydb_newsecasvar 
(/usr/src/sys/netkey/keydb.c:191).
186             bzero(p, sizeof(*p));
187             p->id = said;
188             if (q)
189                     TAILQ_INSERT_AFTER(&satailq, q, p, tailq);
190             else
191                     TAILQ_INSERT_TAIL(&satailq, p, tailq);
192             return p;
193     }
194
195     void
(kgdb) l 156,193
156     /*
157      * secasvar management (reference counted)
158      */
159     struct secasvar *
160     keydb_newsecasvar()
161     {
162             struct secasvar *p, *q;
163             static u_int32_t said = 0;
164
165             p = (struct secasvar *)malloc(sizeof(*p), M_SECA, M_NOWAIT);
166             if (!p)
167                     return p;
168
169     again:
170             said++;
171             if (said == 0)
172                     said++;
173             TAILQ_FOREACH(q, &satailq, tailq) {
174                     if (q->id == said)
175                             goto again;
176                     if (TAILQ_NEXT(q, tailq)) {
177                             if (q->id < said && said < TAILQ_NEXT(q, 
tailq)->id)
178                                     break;
179                             if (q->id + 1 < TAILQ_NEXT(q, tailq)->id) {
180                                     said = q->id + 1;
181                                     break;
182                             }
183                     }
184             }
185
186             bzero(p, sizeof(*p));
187             p->id = said;
188             if (q)
189                     TAILQ_INSERT_AFTER(&satailq, q, p, tailq);
190             else
191                     TAILQ_INSERT_TAIL(&satailq, p, tailq);
192             return p;
193     }

Without a clue on the FreeBSD kernel and not enough time to get stuck in 
(atm) I'm not geing to be able to get further without help.

Regards,

-- 
Matthew Sullivan
Specialist Systems Programmer
Information Technology Services
The University of Queensland


Received on Fri Jan 14 2005 - 00:47:54 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:26 UTC