Q&A on textdumps

Dear all,

I've received a few textdump-related questions that I thought I'd share my 
answers to.

(1) What information is in a textdump?

The textdump is stored as a tarfile with several subfiles in it:

config.txt - Kernel configuration, if compiled into kernel
ddb.txt - Captured DDB output, if present
msgbuf.txt - Kernel message buffer
panic.txt - Kernel panic message, if there was a panic
version.txt - Kernel version string

It is easy to add new files to textdumps, so if there's some easily 
extractable kernel state that you feel should go in there, drop me an e-mail 
and/or send a patch.

(2) Is there any information in a textdump that can't be acquired using kgdb 
and other available dump analysis tools?

In principle no, as normal dumps include all kernel memory, and textdumps 
operate by inspecting kernel memory using DDB, capturing only small but 
presumably relevant parts.  However, there are some important differences in 
approach that mean that textdumps can be used in ways that regular dumps can't 
easily be:

- DDB textdumps are very small. Including a full debugging session, kernel 
message buffer, and kernel configuration, my textdumps are frequently around 
100k uncompressed. This makes it possible to use them on very small machines, 
store them for an extended period, e-mail them around, etc, in a way that you 
can't currently do with kernel memory dumps. This improved usability will 
(hopefully) improve our bug and crash management.

- DDB is a specialized debugging tool with intimate knowledge of the kernel, 
and there are types of data trivially extracted with DDB that are awkward or 
quite difficult to extract using kgdb or other currently available dump 
analysis tools. Locking, waiting, and process information are examples of 
where automatic extraction is currently only possible with DDB, and one of the 
reasons many developers prefer to begin any diagnosis with an interactive DDB 
session.

- DDB textdumps can be used without the exact source tree, kernel 
configuration, built kernel, and debug symbols, as they interpret rather than 
save the pages of memory. They're even an architecture-independent file format 
so you don't need a cross-debugger. Having that additional context is useful 
(ability to map symbol+offset to line of code), but you can actually go a 
remarkable way without it, especially looking at the results in a PR 
potentially years later.

(3) What do I lose by using textdumps?

To be clear, there are also some important things that textdumps can't do -- 
principally, a textdump doesn't contain all kernel memory, so your textdump 
output is all you have. If you need to extract detailed structure information 
for something DDB doesn't understand, or that you don't think of in advance or 
during a DDB session, then there's nothing to fall back on except configuring 
a textdump or regular dump and waiting for the panic to happen again.

(4) When should I use textdumps?

Minidumps remain the default in 7.x and 8.x, and full dumps remain the default 
in 6.x and earlier. Textdumps must be specifically enabled by the 
administrator to be used.

DDB is an excellent live debugging tool whose use has been limited to 
situations where there is an accessible video console, or more ideally serial 
or firewire console to a second box, and generally requiring an experienced 
developer to be available to drive debugging. There are many problems that can 
be pretty much instantly understood with a couple of DDB commands, so these 
limitations impacted debugging effectiveness.

The goal of adding DDB capture output, scripting, and textdumps was to broaden 
the range of situations in which DDB could be used: now it is usable more 
easily for post-mortem analysis, no console or second machine is required, and 
a developer can install, or even e-mail, a script of DDB commands to run 
automatically. Developers can simply define a few scripts to handle various 
DDB cases, such as panic, and get a nice debugging bundle to look at later.

When I'm debugging network stack problems, I typically want a fairly small set 
of DDB commands to be run by the user, and the output sent back, and now it 
will go from "Read the chapter on kernel debugging, set up a serial console, 
run the following commands, copy and paste from your serial console -- oh, you 
don't have a serial console, perhaps hand-copy these fields or use a digital 
camera" to "run the following ddb(8) command and when the box reboots, send me 
the tarball in /var/crash".

I anticipate that textdumps will see use when developers are exchanging e-mail 
with users reporting problems and trying to gather concise summaries of 
information about a crash with minimum downtime and maximum portability, in 
embedded environments where dumping kernel memory to flash is tricky, or in 
order to save a transcript of an interactive DDB session when testing new 
features locally.

Another interesting advantage of textdumps is that it's easy to inspect them 
for confidential/identifying information and mask or purge it. When someone 
sends out a kernel memory dump, it potentially contains a lot of sensitive 
information, and most people (including me) would have difficulty making sure 
all sensitive information was purged safely.

(5) I want to collect DDB output, but still need memory dumps -- can I do 
both?

Yes and no.

Yes, you can use the DDB output capture buffer and scripting without using a 
textdump, as the capture buffer is stored in kernel memory. You can print it 
using kgdb, and we should probably add that capability to ddb(8) also. End 
your script with "call doadump; reset" but don't "textdump set". For example:

   ddb script kdb.enter.panic="capture on;show pcpu;trace;ps;show 
locks;alltrace;show alllocks;show lockedvnods;call doadump;reset"

No, because you must pick one of the three dump layouts (dump, minidump, 
textdump) to write to the swap partition -- you can't write out all three and 
then decide which to extract later. In principle this could be changed so that 
we actually write out a textdump section and a full/minidump, but that's not 
implemented.

(6) I have a serial console so don't need textudmps, can I still use DDB 
scripting to manage a crash?

Yes. You can set up scripts in exactly the same way as with textdumps, only 
omit the textdump bits and end with a "reset" to reboot the system when done. 
That way you can extract the results from the serial console log. I.e.,

   ddb script kdb.enter.panic="show pcpu;trace;show locks;ps;alltrace;show 
alllocks;show lockedvnods;reset"

(7) I'm in DDB and I suddenly realize I want to save the output, and I haven't 
configured textdumps. What do I do?

As with normal dumps, you must previously have configured support for a dump 
partition. These days, that is done automatically whenever you have swap 
configured on the box, so unless you're in single-user mode or don't have swap 
configured, you should be able to do the following:

Schedule a textdump using the "textdump set" command.

Turn on DDB output capture using "capture on", run your commands of interest, 
and turn it off using "capture off".

Type "call doadump" to dump memory, and "reset" to reboot.

(8) The buffer is small, can I pick and choose what DDB output is captured?

The capture buffer does have a size limit, so you might find you want to 
explore interactively at first to figure out what information to save. Then 
you can turn it on and off around output to capture with "capture on" and 
"capture off". Each time you turn capture back on, new output is appended 
after any existing output.

If you decide you want to clear the buffer, you can use "capture reset" to do 
that, and you can check the status of the buffer using "capture status".

You can also increase the buffer size by setting the debug.ddb.capture.bufsize 
sysctl to a larger size.  The sysctl will automatically round up to the next 
textdump blocksize.

(9) Can I continue the kernel after doing a textdump?

No. As with kernel memory dumps, textdumps invoke the storage controller 
dumper routine, which may hose up state in the device driver preventing its 
use after the dump is generated.

However, if you do plan to continue from DDB, just use DDB output capture 
without a textdump. You can then extract the contents of the DDB buffer using 
the debug.ddb.capture.data sysctl.