Experiencing hangs on SMP box with no console messages given for clues. Details inside.

From: Tillman Hodgson <tillman_at_seekingfire.com>
Date: Thu, 8 Mar 2007 06:59:27 -0600
Howdy folks,

These has been happening every few days for a few weeks now. When it
occurs, there's no messages logged to the console or to syslog -- it
just silently hangs. I added the break-to-debugger option so that I can
at least reboot it remotely via the serial console.

I've been following the -current kernel fairly closely in hopes that it
was just due to a transitory -current problem. I don't mind rebuilding a
kernel with special options if it's useful -- I'll be rebuilding this
morning with WITNESS and INVARIANTS for sure. I have the core saved,
though I'm inexperienced with gdb.

FreeBSD/i386 (athena.seekingfire.prv) (ttyd0)
login: 
telnet> send brk
KDB: enter: Line break on console
[thread pid 11 tid 100005 ]
Stopped at      kdb_enter+0x2c: leave
db> ?
Bad character
?
db> help
print       p           examine     x           search      set
write       w           delete      d           break       b
dwatch      watch       dhwatch     hwatch      step        s
continue    c           until       next        match       trace
t           alltrace    where       bt          call        show
ps          gdb         halt        reboot      reset       kill
watchdog    thread      panic       ahd_dump    ahd_out     ahd_in
ahd_unpause ahd_pause   ahd_sunit
db> bt
Tracing pid 11 tid 100005 td 0xc3afe6c0
kdb_enter(c0956f95,c0,c3afe6c0,c3af7cc8,c3afb880,...) at kdb_enter+0x2c
siointr1(c3cb7b80,e25f0c84,c08cd60f,c3cb4000,c3afe6c0,...) at siointr1+0x3be
siointr(c3cb4000,c3afe6c0,0,0,c3bfb400,...) at siointr+0x4c
intr_execute_handlers(c3af7cc8,e25f0c94) at intr_execute_handlers+0xf3
Xapic_isr1() at Xapic_isr1+0x34
--- interrupt, eip = 0xc0baf599, esp = 0xe25f0cd4, ebp = 0xe25f0cd4 ---
acpi_cpu_c1(e25f0cec,c06e382d,c0a5cb60,c3afe6c0,c06e3ccc,...) at acpi_cpu_c1+0x5
acpi_cpu_idle(0,e25f0d24,c06b5db1,0,e25f0d38,...) at acpi_cpu_idle+0x15a
sched_idletd(0,e25f0d38,0,c3afdb40,0,...) at sched_idletd+0x8a
fork_exit(c06e3ccc,0,e25f0d38) at fork_exit+0x61
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe25f0d70, ebp = 0 ---
db> show proc
Process 11 (idle: cpu0) at 0xc3afdb40:
 state: NORMAL
 uid: 0  gids: 0
 parent: pid 0 at 0xc0a58d80
 ABI: null
 threads: 1
100005                   Run     CPU 0               [idle: cpu0]
db> panic
panic: from debugger
cpuid = 0
Uptime: 2d22h24m3s
Physical memory: 1015 MB
Dumping 200 MB: 185 169 153 137 121 105 89 73 57 41 25 9
Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort

[root_at_athena ~]# uname -a
FreeBSD athena.seekingfire.prv 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Sun
Mar  4 21:08:19 CST 2007     toor_at_athena.seekingfire.prv

(/usr/src was synced the same day)

[root_at_athena /usr/src/sys/i386/conf]# diff ATHENA GENERIC
24c24
< ident         ATHENA
---
> ident         GENERIC
29c29
< ### makeoptions       DEBUG=-g                # Build kernel with gdb(1) debug symbols
---
> makeoptions   DEBUG=-g                # Build kernel with gdb(1) debug symbols
67,73c67,70
< ###options    INVARIANTS              # Enable calls of extra sanity checking
< ###options    INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
< ###options    WITNESS                 # Enable checks to detect deadlocks and cycles
< ###options    WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
< 
< ### Tillman added 26Feb07 as per http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/serialconsole-setup.html
< options               BREAK_TO_DEBUGGER
---
> options       INVARIANTS              # Enable calls of extra sanity checking
> options       INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
> options       WITNESS                 # Enable checks to detect deadlocks and cycles
> options       WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed

[root_at_athena ~]# dmesg

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #0: Sun Mar  4 21:08:19 CST 2007
    toor_at_athena.seekingfire.prv:/usr/obj/usr/src/sys/ATHENA
ACPI APIC Table: <VIA694 AWRDACPI>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel Pentium III (997.17-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x68a  Stepping = 10
  Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory  = 1073676288 (1023 MB)
avail memory = 1041326080 (993 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <VIA694 AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0x4000-0x407f,0x4080-0x40ff,0x5000-0x500f,0x6000-0x607f on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem 0xf4000000-0xf4ffffff,0xf6241000-0xf6241fff irq 19 at device 6.0 on pci0
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd400-0xd40f at device 7.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 12 at device 7.2 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 12 at device 7.3 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
pci0: <bridge> at device 7.4 (no driver attached)
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xe000-0xe03f mem 0xf6240000-0xf6240fff,0xf6000000-0xf60fffff irq 17 at device 13.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:21:ad:e0
fxp0: [ITHREAD]
fxp1: <Intel 82559 Pro/100 Ethernet> port 0xe400-0xe43f mem 0xf6242000-0xf6242fff,0xf6100000-0xf61fffff irq 18 at device 14.0 on pci0
miibus1: <MII bus> on fxp1
inphy1: <i82555 10/100 media interface> PHY 1 on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: Ethernet address: 00:e0:81:21:ad:e1
fxp1: [ITHREAD]
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xe800-0xe83f mem 0xf6200000-0xf621ffff,0xf6220000-0xf623ffff irq 18 at device 16.0 on pci0
em0: Ethernet address: 00:0e:0c:c2:ce:4f
em0: [FILTER]
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ad0: 38166MB <Seagate ST340016A 3.75> at ata0-master UDMA100
acd0: CDROM <CDU5211/YYS7> at ata1-master UDMA33
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
/var: mount pending error: blocks 56 files 3

USER      PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root       11 97.6  0.0     0     8  ??  RL    6:39AM  13:14.72 [idle: cpu0]


-T



-- 
"To be nobody but yourself in a world which is doing its best to make
 you everybody else, means to fight the hardest human battle ever and to
 never stop fighting."
    -- e.e. cummings
Received on Thu Mar 08 2007 - 12:30:55 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:06 UTC