Re: FreeBSD 5.3 crash (core with debug symbols available)

On Sat, 26 Feb 2005, Dariusz Kulinski wrote:

> I posted this on comp.unix.bsd.freebsd.misc newsgroup, but I got just
> one reply, which claims that core I got is not really useful. I just
> want to check here also, before I throw it out.
>
> BTW: The system is FreeBSD 5.3-RELEASE-p5.
>
> Here is my original post:
>
> I get 2 crashes on my FreeBSD 5.3, after the second one, I decided to
> compile it with debugging symbols.
>
> The crash happens rarely but it's usually while dumping the system do
> backup (I use snapshot since I'm backing up a live system).
>
> There is one bad thing though, while I compiled kernel debugger in, I
> forgot to set debug.debugger_on_panic=0. Because of that while it crashed,
> ddb was started. Since I'm not really experienced with ddb, I decided to
> write continue (it was 2 times).
>
> Do you think, that this dump will be useful?
> What should do to get some useful info for developers, it looks different
> than debugging userland application.

Could you post the tail end of 'print msgbuf' so we can see what the
faulting address was?

> | [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
> | GNU gdb 6.1.1 [FreeBSD]
> | Copyright 2004 Free Software Foundation, Inc.
> | GDB is free software, covered by the GNU General Public License, and you are
> | welcome to change it and/or distribute copies of it under certain conditions.
> | Type "show copying" to see the conditions.
> | There is absolutely no warranty for GDB.  Type "show warranty" for details.
> | This GDB was configured as "i386-marcel-freebsd".
> | doadump () at pcpu.h:159
> | (kgdb) bt
> | #0  doadump () at pcpu.h:159
> | #1  0xc04e6024 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:397
> | #2  0xc04e63d9 in panic (fmt=0xc06df826 "vm_fault: fault on nofault entry, addr: %lx")
> |     at /usr/src/sys/kern/kern_shutdown.c:553
> | #3  0xc0644249 in vm_fault (map=0xc103a000, vaddr=3509317632, fault_type=1 '\001', fault_flags=0)
> |     at /usr/src/sys/vm/vm_fault.c:278
> | #4  0xc069dca3 in trap_pfault (frame=0xcbd25be4, usermode=0, eva=3509320804) at /usr/src/sys/i386/i386/trap.c:716
> | #5  0xc069d860 in trap (frame=
> |       {tf_fs = -1040646120, tf_es = 16, tf_ds = -1043267568, tf_edi = 0, tf_esi = 0, tf_ebp = -875406268, tf_isp = -875406320, tf_ebx = -1048825456, tf_edx = 8192, tf_ecx = 13, tf_eax = -785646492, tf_trapno = 12, tf_err = 0, tf_eip = -1068589761, tf_cs = 8, tf_eflags = 65666, tf_esp = 0, tf_ss = -1051819632}) at /usr/src/sys/i386/i386/trap.c:417
> | #6  0xc068ad7a in calltrap () at /usr/src/sys/i386/i386/exception.s:140
> | #7  0xc1f90018 in ?? ()
> | #8  0x00000010 in ?? ()
> | #9  0xc1d10010 in ?? ()
> | #10 0x00000000 in ?? ()
> | #11 0x00000000 in ?? ()
> | #12 0xcbd25c44 in ?? ()
> | #13 0xcbd25c10 in ?? ()
> | #14 0xc17c3190 in ?? ()
> | #15 0x00002000 in ?? ()
> | #16 0x0000000d in ?? ()
> | #17 0xd12bfc64 in ?? ()
> | #18 0x0000000c in ?? ()
> | #19 0x00000000 in ?? ()
> | #20 0xc04e9d3f in sigtd (p=0xc16948d4, sig=14, prop=129) at /usr/src/sys/kern/kern_sig.c:1581

This is (in RELENG_5_3):

1580         FOREACH_THREAD_IN_PROC(p, td) {
1581                 if (td->td_waitset != NULL &&
1582                     SIGISMEMBER(*(td->td_waitset), sig)) {
1583                                 mtx_unlock_spin(&sched_lock);
1584                                 return (td);

Looks like it ran over a spammed thread, but I'll want to see the fault
address. Bets on whether its 0xdeadc0de+offset?

> | #21 0xc04e9e2b in psignal (p=0x0, sig=14) at
/usr/src/sys/kern/kern_sig.c:1634
> | #22 0xc04f5170 in realitexpire (arg=0xc16948d4) at /usr/src/sys/kern/kern_time.c:554
> | #23 0xc04f595e in softclock (dummy=0x0) at /usr/src/sys/kern/kern_timeout.c:259
> | #24 0xc04cbed8 in ithread_loop (arg=0xc14dd580) at /usr/src/sys/kern/kern_intr.c:547
> | #25 0xc04cac10 in fork_exit (callout=0xc04cbd20 <ithread_loop>, arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:811
> | #26 0xc068addc in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:209
> | (kgdb) up
> | No stack.
>
>
>

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite_at_gumbysoft.com          |  www.FreeBSD.org