Re: beta6/7 machine freeze

On Mon, 11 Oct 2004, Volker wrote:

> Since using that test setup the server machine didn't had a lockup. So
> my guess might be probably right to have a problem somewhere at i4b or
> user-ppp in releng_5. 
> 
> How does one debug a dead machine? Is there any way to get a backtrace
> or call the debugger while the machine has been frozen? 

There's a pretty useful chapter in the handbook on getting setup to debug. 
For debugging hangs, the first thing I'll generally do is get the box set
up with a serial console and see if I can get into the debugger by sending
a serial break (the break on the actualy console might also work, but is
sometimes less reliable when there's a hang).  You will want to compile at
least options KDB, DDB, and BREAK_TO_DEBUGGER into the kernel.  On the
real console, Ctrl-Alt-Escape is the break sequence; on the serial
console, it's up to your client: for tip and cu, it's ~#, I believe.

Assuming you can get into the debugger, the first thing you'll want to do
is use "show pcpu" and "trace" to tell you a little about the active
thread.  You might "continue", then drop in a gain a few seconds later and
see if the trace looks the same or similar.

If you can't get into the debugger using the typicaly responses are: you
really want to get into the debugger, so arrange an NMI or watchdog, or,
you give up on getting into the debugger and try various things to see if
removing one causes the hang to go away.  I have some test hardware with a
conveniently placed NMI button on the back -- you press the button and it
generates a non-maskable interrupt which almost always works.  On other
hardware, you can play nasty hardware tricks.  If you have SMP, you might
try the MP_WATCHDOG kernel option and sysctls (under-documented).

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research