Re: FreeBSD 5.3 crash (core with debug symbols available)

From: Dariusz Kulinski <d.kulinski_at_gmail.com>
Date: Tue, 1 Mar 2005 21:29:34 -0800
Hello Doug,

Tuesday, March 1, 2005, 6:59:42 PM, you wrote:

> Some more exploration is necessary. Can you make the crashdump and debug
> kernel available?  Also, what was running when this panic tripped? ("info
> threads" in kgdb may be useful.)

This crash occured while doing backup on a live system (snapshot was
enabled to get consistent backup).

But it might not be related to the snapshot, today (Tuesday) I noticed
on morning that system crashed again. I posted information about
that before (Message-ID: <596413574.20050301104045_at_takeda.tk>). This
crash also occured in same line in the code, but the difference is
that it happened when I wasn't around. So it's hard to me to tell if
the line is just coincidence, or the crashes are related.

From the logfile (of the second crash (not this one)):
[...]
Mar  1 07:18:32 chinatsu httpd: PHP Fatal error:  Allowed memory size
of 8388608 bytes exhausted (tried to allocate 240 bytes) in
/usr/local/www/phpwiki-1.3.10/lib/WikiUserNew.php on line 1012 
Mar  1 09:59:26 chinatsu syslogd: restart
Mar  1 09:59:26 chinatsu syslogd: kernel boot file is /boot/kernel/kernel
[...]

Looks like it happened somewhere after 7:18:32 (9:59:26 was when I
went to the console and asked to dump core file)

I showed the log file, because the message is strange. Maybe is related
to the crash. Please note that this is from the second crash that
occured today, not the one we're analyzing currently.

As for the list of threads from the first core dump, here it goes:
(kgdb) info threads
  149 TID 100050  sched_switch (td=0xc1695000, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  148 TID 100254  sched_switch (td=0xc2d9a640, newtd=0xc14e3e10, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  147 TID 100176  sched_switch (td=0xc1d0b960, newtd=0xc211e000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  146 TID 100219  sched_switch (td=0xc1f9ae10, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1791da0)
  145 TID 100262  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d136ada0)
  144 TID 100104  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1454da0)
  143 TID 100143  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14e8da0)
  142 TID 100200  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14b0da0)
  141 TID 100158  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1317da0)
  140 TID 100095  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d137cda0)
  139 TID 100110  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1794da0)
  138 TID 100263  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1842da0)
  137 TID 100271  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1851da0)
  136 TID 100276  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d168eda0)
  135 TID 100305  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d16a0da0)
  134 TID 100311  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1394da0)
  133 TID 100118  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d173bda0)
  132 TID 100333  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1314da0)
  131 TID 100094  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1869da0)
  130 TID 100284  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d142dda0)
  129 TID 100130  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14a1da0)
  128 TID 100153  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1445da0)
  127 TID 100138  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d14b3da0)
  126 TID 100159  0x00000000 in ?? ()
  125 TID 100172  sched_switch (td=0xc1d0b320, newtd=0xc1addaf0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1732da0)
  124 TID 100330  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d179dda0)
  123 TID 100266  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13ecda0)
  122 TID 100129  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1618da0)
  121 TID 100242  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d152fda0)
  120 TID 100204  0x00000000 in ?? ()
  119 TID 100220  sched_switch (td=0xc211e000, newtd=0xc2e59af0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d172cda0)
  118 TID 100328  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1592da0)
  117 TID 100192  0x00000000 in ?? ()
  116 TID 100277  sched_switch (td=0xc2e59af0, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d130bda0)
  115 TID 100091  0x00000000 in ?? ()
  114 TID 100214  sched_switch (td=0xc1f9a640, newtd=0xc18db640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  113 TID 100152  sched_switch (td=0xc1c54320, newtd=0xc2d9a640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  112 TID 100066  sched_switch (td=0xc1696960, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  111 TID 100150  sched_switch (td=0xc1c54000, newtd=0xc1696960, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  110 TID 100173  sched_switch (td=0xc1d0b4b0, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d130eda0)
  109 TID 100092  0x00000000 in ?? ()
  108 TID 100052  sched_switch (td=0xc1695320, newtd=0xc1f9a640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1685da0)
  107 TID 100302  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09a6da0)
  106 TID 100054  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d16a3da0)
  105 TID 100312  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d175ada0)
  104 TID 100339  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1694da0)
  103 TID 100307  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d165dda0)
  102 TID 100294  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d167cda0)
  101 TID 100299  0x00000000 in ?? ()
  100 TID 100210  sched_switch (td=0xc1f9a000, newtd=0xc1d0b000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  99 TID 100170  sched_switch (td=0xc1d0b000, newtd=0xc1cd6000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  98 TID 100132  sched_switch (td=0xc1adc320, newtd=0xc19dbc80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  97 TID 100327  sched_switch (td=0xc189daf0, newtd=0xc1adc320, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  96 TID 100338  sched_switch (td=0xc19dbc80, newtd=0xc1c54000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  95 TID 100285  sched_switch (td=0xc2e5a7d0, newtd=0xc1695960, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  94 TID 100310  sched_switch (td=0xc1cd6000, newtd=0xc2e5a7d0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d15cfda0)
  93 TID 100240  0x00000000 in ?? ()
  92 TID 100195  sched_switch (td=0xc2a4a7d0, newtd=0xc2d9a640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d09afda0)
  91 TID 100057  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1708da0)
  90 TID 100322  0x00000000 in ?? ()
  89 TID 100320  sched_switch (td=0xc189d000, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d1574da0)
  88 TID 100182  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d16acda0)
  87 TID 100315  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1595da0)
  86 TID 100193  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09cdda0)
  85 TID 100067  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d15a7da0)
  84 TID 100199  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d158cda0)
  83 TID 100190  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12c8da0)
  82 TID 100084  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1589da0)
  81 TID 100189  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12a1da0)
  80 TID 100071  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d168bda0)
  79 TID 100304  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d183fda0)
  78 TID 100270  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1571da0)
  77 TID 100181  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1682da0)
  76 TID 100301  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12bfda0)
  75 TID 100081  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1654da0)
  74 TID 100291  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d15e5da0)
  73 TID 100217  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d144eda0)
  72 TID 100141  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13e0da0)
  71 TID 100125  0x00000000 in ?? ()
  70 TID 100156  sched_switch (td=0xc1c54960, newtd=0xc2d9a640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  69 TID 100114  sched_switch (td=0xc18db640, newtd=0xc17c2c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  68 TID 100115  sched_switch (td=0xc18db7d0, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d14bfda0)
  67 TID 100163  0x00000000 in ?? ()
  66 TID 100144  sched_switch (td=0xc1add640, newtd=0xc2d9a640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  65 TID 100147  sched_switch (td=0xc1addaf0, newtd=0xc18db640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
kgdb: kvm_read: invalid address (d143fda0)
  64 TID 100136  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1308da0)
  63 TID 100090  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09b2da0)
  62 TID 100058  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d131dda0)
  61 TID 100097  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1370da0)
  60 TID 100106  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13e6da0)
  59 TID 100127  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d12b3da0)
  58 TID 100077  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1379da0)
  57 TID 100109  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (cbd2eda0)
  56 TID 100024  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1364da0)
  55 TID 100102  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d136dda0)
  54 TID 100105  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d13e3da0)
  53 TID 100126  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1311da0)
  52 TID 100093  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d1376da0)
  51 TID 100108  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09b8da0)
  50 TID 100060  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09b5da0)
  49 TID 100059  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09c1da0)
  48 TID 100063  0x00000000 in ?? ()
kgdb: kvm_read: invalid address (d09beda0)
  47 TID 100062  0x00000000 in ?? ()
  46 TID 100056  sched_switch (td=0xc1695960, newtd=0xc14e8e10, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  45 TID 100023  sched_switch (td=0xc14e84b0, newtd=0xc14e8c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  44 TID 100049  sched_switch (td=0xc1692e10, newtd=0xc14e84b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  43 TID 100048  sched_switch (td=0xc1692c80, newtd=0xc1692e10, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  42 TID 100047  sched_switch (td=0xc1692af0, newtd=0xc1692c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  41 TID 100046  sched_switch (td=0xc1692960, newtd=0xc1692af0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  40 TID 100045  sched_switch (td=0xc16927d0, newtd=0xc18db7d0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  39 TID 100044  sched_switch (td=0xc1692640, newtd=0xc2d9a640, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  38 TID 100043  sched_switch (td=0xc16924b0, newtd=0xc14e84b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  37 TID 100042  sched_switch (td=0xc1692320, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  36 TID 100041  sched_switch (td=0xc1692190, newtd=0xc14e0c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  35 TID 100040  sched_switch (td=0xc1692000, newtd=0xc14e8c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  34 TID 100012  sched_switch (td=0xc14e3320, newtd=0xc14e8c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  33 TID 100011  sched_switch (td=0xc14e3190, newtd=0xc14e8960, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  32 TID 100035  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  31 TID 100034  sched_switch (td=0xc150b640, newtd=0xc14e3190, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  30 TID 100033  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  29 TID 100032  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  28 TID 100031  sched_switch (td=0xc150b190, newtd=0xc18d87d0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  27 TID 100030  sched_switch (td=0xc150b000, newtd=0xc1d0b320, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  26 TID 100029  sched_switch (td=0xc14e8e10, newtd=0xc1c54000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  25 TID 100028  sched_switch (td=0xc14e8c80, newtd=0xc14e04b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  24 TID 100027  sched_switch (td=0xc14e8af0, newtd=0xc1692000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  23 TID 100026  sched_switch (td=0xc14e8960, newtd=0xc14e8c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  22 TID 100000  sched_switch (td=0xc14e0000, newtd=0xc14e8190, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  21 TID 100022  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
* 20 TID 100021  doadump () at pcpu.h:159
  19 TID 100020  sched_switch (td=0xc14e8000, newtd=0xc150b190, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  18 TID 100019  sched_switch (td=0xc14e3e10, newtd=0xc14e8c80, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  17 TID 100018  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  16 TID 100017  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  15 TID 100016  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  14 TID 100015  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  13 TID 100014  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  12 TID 100013  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  11 TID 100010  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  10 TID 100009  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  9 TID 100008  sched_switch (td=0xc14e0c80, newtd=0xc1695320, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  8 TID 100007  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  7 TID 100006  sched_switch (td=0xc14e0960, newtd=0xc150b000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  6 TID 100005  sched_switch (td=0xc14e07d0, newtd=0xc14e8190, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  5 TID 100004  fork_trampoline () at /usr/src/sys/i386/i386/exception.s:206
  4 TID 100003  sched_switch (td=0xc14e04b0, newtd=0xc14e8190, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  3 TID 100002  sched_switch (td=0xc14e0320, newtd=0xc17c2000, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  2 TID 100001  sched_switch (td=0xc14e0190, newtd=0xc1b954b0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
  1 TID 0  sched_switch (td=0xc07203c0, newtd=0xc14e8af0, flags=1) at /usr/src/sys/kern/sched_4bsd.c:866
(kgdb)

I can provide kernel.debug, but as for publishing core dump... I
belive there is a lot of sensitive data (besides that is a core dump
of a kernel, the crash occured while doing a backup). The worst thing
is that it's not only my data but also data of my users.

Isn't there a better way?

-- 
Best regards,
 Dariusz                            mailto:d.kulinski_at_gmail.com
http://www.takeda.tk
Received on Wed Mar 02 2005 - 04:29:40 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:29 UTC