Re: KGDB stack traces in the kernel.

From: Julian Elischer <julian_at_freebsd.org>
Date: Wed, 06 Apr 2011 13:31:51 -0700
On 4/6/11 10:17 AM, Julian Elischer wrote:
> On 4/6/11 5:45 AM, John Baldwin wrote:
>> On Tuesday, April 05, 2011 4:35:44 pm Navdeep Parhar wrote:
>>> On Tue, Apr 5, 2011 at 1:33 PM, Julian 
>>> Elischer<julian_at_freebsd.org>  wrote:
>>>> On 4/4/11 6:04 PM, Justin Hibbits wrote:
>>>>
>>> What does ddb do?  It always seems to get this stuff correct.
>> ddb knows to stop when it gets to a non-kernel address, and it uses 
>> string
>> compares on function names to identify trap frames.  For example in
>> sys/amd64/amd64/db_trace.c:
>>
>>
>>                  if (strcmp(name, "calltrap") == 0 ||
>>                      strcmp(name, "fork_trampoline") == 0 ||
>>                      strcmp(name, "nmi_calltrap") == 0 ||
>>                      strcmp(name, "Xdblfault") == 0)
>>                          frame_type = TRAP;
>>
>> Hah, kgdb just needs to be updated (this is from trgt_amd64.c):
>>
>> const struct frame_unwind *
>> kgdb_trgt_trapframe_sniffer(struct frame_info *next_frame)
>> {
>>          char *pname;
>>          CORE_ADDR pc;
>>
>>          pc = frame_pc_unwind(next_frame);
>>          pname = NULL;
>>          find_pc_partial_function(pc,&pname, NULL, NULL);
>>          if (pname == NULL)
>>                  return (NULL);
>>          if (strcmp(pname, "calltrap") == 0 ||
>>              strcmp(pname, "nmi_calltrap") == 0 ||
>>              (pname[0] == 'X'&&  pname[1] != '_'))
>>                  return (&kgdb_trgt_trapframe_unwind);
>>          /* printf("%s: %lx =%s\n", __func__, pc, pname); */
>>          return (NULL);
>> }
>>
>
> I'll give that a try
>
>> Can probably just add 'fork_trampoline' to that conditional.  I 
>> think i386
>> needs a similar fix in kgdb.  Not sure about other architectures:
>>
>> Index: trgt_amd64.c
>> ===================================================================
>> --- trgt_amd64.c    (revision 220190)
>> +++ trgt_amd64.c    (working copy)
>> _at__at_ -184,6 +184,7 _at__at_
>>       if (pname == NULL)
>>           return (NULL);
>>       if (strcmp(pname, "calltrap") == 0 ||
>> +        strcmp(pname, "fork_trampoline") == 0 ||
BTW this is in 8.x

I don't have 9 on  a machine I can test this with.

While ddb finds fork_trampoline, (k)gdb does not.
so I had to add fork_exit a well, but then it gets an error as well.

firstly without fork_exit()
#0  0xffffffff8144c529 in XXXXXXXX () from blah.ko
#1  0xffffffff814dfea5 in XXXXXXXX () from blah.ko
#2  0xffffff0063a1f600 in ?? ()
#3  0xffffffff814dfe30 in XXXXXXXX () from blah.ko
#4  0xffffff0063a265f0 in ?? ()
#5  0xffffff8244290c30 in ?? ()
#6  0xffffffff805c4b8e in fork_exit (callout=0xffffffff814dfe30 
<XXXXXXX>, arg=0x89e475f700000000, frame=0xbad089e445890000) at 
../../../kern/kern_fork.c:859
#7  0xf445c7a07d894868 in ?? ()
#8  0x095d77e800000000 in ?? ()
#9  0x00095c40e8c78900 in ?? ()
#10 0x000001bfd8458948 in ?? ()
[... etc forever]

if I add for_exit, it does work but:

#6  0xffffff82441c8c30 in ?? ()
#7  0xffffffff805c4b8e in fork_exit (callout=0xffffffff814dfe30 
<XXXXXXX>, arg=0x89e475f700000000, frame=0xbad089e445890000) at 
../../../kern/kern_fork.c:859
Previous frame identical to this frame (corrupt stack?)

that last error message makes ddd throw up its hands and get confused.
it seems to occur all the time however. for example when you use 
sysctl debug.kdb.enter to enter the debugger,
the same thing occurs:  this easy to see.

#0  kdb_enter (why=0xffffffff80a3494c "sysctl", msg=0xa <Address 0xa 
out of bounds>) at ../../../kern/subr_kdb.c:352
#1  0xffffffff80626bc9 in kdb_sysctl_enter (oidp=0xffffffff80c6c360, 
arg1=Variable "arg1" is not available.
) at ../../../kern/subr_kdb.c:174
#2  0xffffffff805fddc3 in sysctl_root (oidp=Variable "oidp" is not 
available.
) at ../../../kern/kern_sysctl.c:1420
#3  0xffffffff805fe00e in userland_sysctl (td=0xffffff0009a7c000, 
name=0xffffff824433fa20, namelen=3, old=0x0, oldlenp=Variable 
"oldlenp" is not available.
) at ../../../kern/kern_sysctl.c:1524
#4  0xffffffff805fe4ba in __sysctl (td=0xffffff0009a7c000, 
uap=0xffffff824433fbb0) at ../../../kern/kern_sysctl.c:1450
#5  0xffffffff8063429f in syscallenter (td=0xffffff0009a7c000, 
sa=0xffffff824433fba0) at ../../../kern/subr_trap.c:315
#6  0xffffffff808feafc in syscall (frame=0xffffff824433fc40) at 
../../../amd64/amd64/trap.c:888
#7  0xffffffff808e6092 in Xfast_syscall () at 
../../../amd64/amd64/exception.S:377
#8  0x000000080073db0c in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

do you have any idea what that is all about?

looking at frames 7,8,6  I see:
Previous frame inner to this frame (corrupt stack?)
(kgdb) up 7
#7  0xffffffff808e6092 in Xfast_syscall () at 
../../../amd64/amd64/exception.S:377
377             call    syscall
Current language:  auto; currently asm
(kgdb) info frame
Stack level 7, frame at 0xffffff824433fc40:
  rip = 0xffffffff808e6092 in Xfast_syscall 
(../../../amd64/amd64/exception.S:377); saved rip 0x80073db0c
  called by frame at 0x7fffffffe190, caller of frame at 0xffffff824433fc40
  source language asm.
  Arglist at 0xffffff824433fc38, args:
  Locals at 0xffffff824433fc38, Previous frame's sp at 0xffffff824433fcf0
  Saved registers:
   rax at 0xffffff824433fc70, rbx at 0xffffff824433fc78, rcx at 
0xffffff824433fc58, rdx at 0xffffff824433fc50, rsi at 
0xffffff824433fc48, rdi at 0xffffff824433fc40, rbp at 0xffffff824433fc80,
   r8 at 0xffffff824433fc60, r9 at 0xffffff824433fc68, r10 at 
0xffffff824433fc88, r11 at 0xffffff824433fc90, r12 at 
0xffffff824433fc98, r13 at 0xffffff824433fca0, r14 at 0xffffff824433fca8,
   r15 at 0xffffff824433fcb0, rip at 0xffffff824433fcd8, eflags at 
0xffffff824433fce8, cs at 0xffffff824433fce0, ss at 0xffffff824433fcf8
(kgdb) up
#8  0x000000080073db0c in ?? ()
(kgdb) info frame
Stack level 8, frame at 0x7fffffffe190:
  rip = 0x80073db0c; saved rip 0x4023a5
  caller of frame at 0xffffff824433fc40
  Arglist at 0x7fffffffe180, args:
  Locals at 0x7fffffffe180, Previous frame's sp is 0x7fffffffe190
  Saved registers:
   rax at 0xffffff824433fc70, rbx at 0xffffff824433fc78, rcx at 
0xffffff824433fc58, rdx at 0xffffff824433fc50, rsi at 
0xffffff824433fc48, rdi at 0xffffff824433fc40, rbp at 0xffffff824433fc80,
   r8 at 0xffffff824433fc60, r9 at 0xffffff824433fc68, r10 at 
0xffffff824433fc88, r11 at 0xffffff824433fc90, r12 at 
0xffffff824433fc98, r13 at 0xffffff824433fca0, r14 at 0xffffff824433fca8,
   r15 at 0xffffff824433fcb0, rip at 0x7fffffffe188, eflags at 
0xffffff824433fce8, cs at 0xffffff824433fce0, ss at 0xffffff824433fcf8
(kgdb) down
#7  0xffffffff808e6092 in Xfast_syscall () at 
../../../amd64/amd64/exception.S:377
377             call    syscall
(kgdb) down
#6  0xffffffff808feafc in syscall (frame=0xffffff824433fc40) at 
../../../amd64/amd64/trap.c:888
888             error = syscallenter(td, &sa);
Current language:  auto; currently c
(kgdb) info frame
Stack level 6, frame at 0xffffff824433fc40:
  rip = 0xffffffff808feafc in syscall 
(../../../amd64/amd64/trap.c:888); saved rip 0xffffffff808e6092
  called by frame at 0xffffff824433fc40, caller of frame at 
0xffffff824433fb30
  source language c.
  Arglist at 0xffffff824433fc30, args: frame=0xffffff824433fc40
  Locals at 0xffffff824433fc30, Previous frame's sp is 0xffffff824433fc40
  Saved registers:
   rbx at 0xffffff824433fc08, rbp at 0xffffff824433fc30, r12 at 
0xffffff824433fc10, r13 at 0xffffff824433fc18, r14 at 
0xffffff824433fc20, r15 at 0xffffff824433fc28, rip at 0xffffff824433fc38
(kgdb)




>>           strcmp(pname, "nmi_calltrap") == 0 ||
>>           (pname[0] == 'X'&&  pname[1] != '_'))
>>           return (&kgdb_trgt_trapframe_unwind);
>> Index: trgt_i386.c
>> ===================================================================
>> --- trgt_i386.c    (revision 220190)
>> +++ trgt_i386.c    (working copy)
>> _at__at_ -374,6 +374,7 _at__at_
>>       if (strcmp(pname, "dblfault_handler") == 0)
>>           return (&kgdb_trgt_dblfault_unwind);
>>       if (strcmp(pname, "calltrap") == 0 ||
>> +        strcmp(pname, "fork_trampoline") == 0 ||
>>           (pname[0] == 'X'&&  pname[1] != '_'))
>>           return (&kgdb_trgt_trapframe_unwind);
>>       /* printf("%s: %llx =%s\n", __func__, pc, pname); */
>>
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to 
> "freebsd-current-unsubscribe_at_freebsd.org"
>
Received on Wed Apr 06 2011 - 18:31:24 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:13 UTC