HEADS UP: DDB output capture, scripting, and textdumps imported

From: Robert Watson <rwatson_at_FreeBSD.org>
Date: Wed, 26 Dec 2007 11:54:17 +0000 (GMT)
Dear all:

Per the prior e-mail thread, I've added several new facilities to DDB(4):

- DDB output capture, in which the input and output to DDB can be captured
   selectively to a buffer for later inspection using a sysctl (after
   continue), or in a textdump or kernel dump.

- DDB scripting, which allows simple scripts to be defined and executed,
   either by hand, or as a result of a DDB event such as a panic, break key, or
   watchdog firing.

- Kernel textdumps, which can store DDB output, the kernel message buffer,
   kernel configuration, and other debugging information in a compact form in a
   dump partition, providing an alternative to full memory dumps where only DDB
   output is required.

I've included the key commit messages below, and have updated ddb(4) and 
textdump(4) to reflect these new facilities.

One idea that has been bounced around a bit is adding an automatic kernel dump 
reporting facility that could be enabled by a system administrator to submit 
textdumps to a specific e-mail address for inspection.  I find that most 
textdumps I generate, assuming I include the kernel message buffer and a 
reasonable set of information-gathering commands, are around 90k-100k, so 
quite manageable and portable compared to regular dumps, and not dependent on 
having an exactly synchronized set of kernel sources and debugging symbols. 
I've not implemented this, but it is (as they say) a small matter of shell 
scripting if someone is interested in giving this a spin.

Adding new information sources to textdumps is relatively easy (and can be 
made easier), should there be other information that people want to capture, 
such as $FreeBSD$ version tags, etc.

Many thanks to the various people who've given me feedback on this project, 
including Antoine Brodin, Peter Wemm, Wojciech Koszek, Nikolay Pavlov, and 
Michael Bushkov.

Robert N M Watson
Computer Laboratory
University of Cambridge

---------- Forwarded message ----------
Date: Tue, 25 Dec 2007 23:06:51 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sys/conf files src/sys/ddb db_capture.c
     db_command.c db_input.c db_main.c db_output.c ddb.h

rwatson     2007-12-25 23:06:51 UTC

   FreeBSD src repository

   Modified files:
     sys/conf             files
     sys/ddb              db_command.c db_input.c db_main.c
                          db_output.c ddb.h
   Added files:
     sys/ddb              db_capture.c
   Log:
   Add a new DDB(4) facility, output capture.  Input and output from DDB may be
   captured to a memory buffer for later inspection using sysctl(8), or in the
   future, to a textdump.

   A new DDB command, "capture", is added, which accepts arguments "on", "off",
   "reset", and "status".

   A new DDB sysctl tree, debug.ddb.capture, is added, which can be used to
   resize the capture buffer and extract buffer contents.

   MFC after:      3 months

   Revision  Changes    Path
   1.1258    +1 -0      src/sys/conf/files
   1.1       +301 -0    src/sys/ddb/db_capture.c (new)
   1.74      +1 -0      src/sys/ddb/db_command.c
   1.37      +1 -0      src/sys/ddb/db_input.c
   1.6       +3 -0      src/sys/ddb/db_main.c
   1.38      +8 -0      src/sys/ddb/db_output.c
   1.44      +15 -0     src/sys/ddb/ddb.h



---------- Forwarded message ----------
Date: Tue, 25 Dec 2007 23:25:04 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/share/man/man4 ddb.4

rwatson     2007-12-25 23:25:04 UTC

   FreeBSD src repository

   Modified files:
     share/man/man4       ddb.4
   Log:
   Document DDB capture facility.

   MFC after:      3 months

   Revision  Changes    Path
   1.43      +34 -1     src/share/man/man4/ddb.4



---------- Forwarded message ----------
Date: Wed, 26 Dec 2007 09:33:19 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sys/conf files src/sys/ddb db_command.c
     db_command.h db_lex.c db_lex.h db_main.c db_script.c ddb.h

rwatson     2007-12-26 09:33:19 UTC

   FreeBSD src repository

   Modified files:
     sys/conf             files
     sys/ddb              db_command.c db_command.h db_lex.c
                          db_lex.h db_main.c ddb.h
   Added files:
     sys/ddb              db_script.c
   Log:
   Add a simple scripting facility to DDB(4), allowing the user to
   define a set of named scripts.  Each script consists of a list of DDB
   commands separated by ";"s that will be executed verbatim.  No higher
   level language constructs, such as branching, are provided for:
   scripts are executed by sequentially injecting commands into the DDB
   input buffer.

   Four new commands are present in DDB: "run" to run a specific script,
   "script" to define or print a script, "scripts" to list currently
   defined scripts, and "unscript" to delete a script, modeled on shell
   alias commands.  Scripts may also be manipulated using sysctls in the
   debug.ddb.scripting MIB space, although users will prefer to use the
   soon-to-be-added ddb(8) tool for usability reasons.

   Scripts with certain names are automatically executed on various DDB
   events, such as entering the debugger via a panic, a witness error,
   watchdog, breakpoint, sysctl, serial break, etc, allowing customized
   handling.

   MFC after:      3 months

   Revision  Changes    Path
   1.1259    +1 -0      src/sys/conf/files
   1.75      +31 -5     src/sys/ddb/db_command.c
   1.14      +1 -0      src/sys/ddb/db_command.h
   1.23      +30 -1     src/sys/ddb/db_lex.c
   1.15      +7 -4      src/sys/ddb/db_lex.h
   1.7       +3 -0      src/sys/ddb/db_main.c
   1.1       +564 -0    src/sys/ddb/db_script.c (new)
   1.45      +29 -0     src/sys/ddb/ddb.h



---------- Forwarded message ----------
Date: Wed, 26 Dec 2007 09:38:22 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sbin Makefile src/sbin/ddb Makefile ddb.8 ddb.c
      ddb.h ddb_script.c

rwatson     2007-12-26 09:38:22 UTC

   FreeBSD src repository

   Modified files:
     sbin                 Makefile
   Added files:
     sbin/ddb             Makefile ddb.8 ddb.c ddb.h ddb_script.c
   Log:
   Add command-line tool ddb(8), which allows DDB(4) scripts to be
   managed from userspace.  It is largely a wrapper for sysctl()
   calls, but because the sysctls for adding and removing scripts
   are awkward to use directly, this provides an easier-to-use
   interface.

   MFC after:      3 months

   Revision  Changes    Path
   1.169     +1 -0      src/sbin/Makefile
   1.1       +8 -0      src/sbin/ddb/Makefile (new)
   1.1       +106 -0    src/sbin/ddb/ddb.8 (new)
   1.1       +67 -0     src/sbin/ddb/ddb.c (new)
   1.1       +37 -0     src/sbin/ddb/ddb.h (new)
   1.1       +160 -0    src/sbin/ddb/ddb_script.c (new)



---------- Forwarded message ----------
Date: Wed, 26 Dec 2007 09:51:38 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/share/man/man4 ddb.4

rwatson     2007-12-26 09:51:38 UTC

   FreeBSD src repository

   Modified files:
     share/man/man4       ddb.4
   Log:
   Add SCRIPTING section to describe new DDB scripting facilities.

   Update copyright.

   Revision  Changes    Path
   1.45      +147 -2    src/share/man/man4/ddb.4



---------- Forwarded message ----------
Date: Wed, 26 Dec 2007 11:32:33 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sys/conf files src/sys/ddb db_capture.c
     db_command.c db_textdump.c ddb.h src/sys/kern kern_shutdown.c

rwatson     2007-12-26 11:32:33 UTC

   FreeBSD src repository

   Modified files:
     sys/conf             files
     sys/ddb              db_capture.c db_command.c ddb.h
     sys/kern             kern_shutdown.c
   Added files:
     sys/ddb              db_textdump.c
   Log:
   Add textdump(4) facility, which provides an alternative form of kernel
   dump using mechanically generated/extracted debugging output rather than
   a simple memory dump.  Current sources of debugging output are:

   - DDB output capture buffer, if there is captured output to save
   - Kernel message buffer
   - Kernel configuration, if included in kernel
   - Kernel version string
   - Panic message

   Textdumps are stored in swap/dump partitions as with regular dumps, but
   are laid out as ustar files in order to allow multiple parts to be stored
   as a stream of sequentially written blocks.  Blocks are written out in
   reverse order, as the size of a textdump isn't known a priori.  As with
   regular dumps, they will be extracted using savecore(8).

   One new DDB(4) command is added, "textdump", which accepts "set",
   "unset", and "status" arguments.  By default, normal kernel dumps are
   generated unless "textdump set" is run in order to schedule a textdump.
   It can be canceled using "textdump unset" to restore generation of a
   normal kernel dump.

   Several sysctls exist to configure aspects of textdumps;
   debug.ddb.textdump.pending can be set to check whether a textdump is
   pending, or set/unset in order to control whether the next kernel dump
   will be a textdump from userspace.

   While textdumps don't have to be generated as a result of a DDB script
   run automatically as part of a kernel panic, this is a particular useful
   way to use them, as instead of generating a complete memory dump, a
   simple transcript of an automated DDB session can be captured using the
   DDB output capture and textdump facilities.  This can be used to
   generate quite brief kernel bug reports rich in debugging information
   but not dependent on kernel symbol tables or precisely synchronized
   source code.  Most textdumps I generate are less than 100k including
   the full message buffer.  Using textdumps with an interactive debugging
   session is also useful, with capture being enabled/disabled in order to
   record some but not all of the DDB session.

   MFC after:      3 months

   Revision  Changes    Path
   1.1260    +1 -0      src/sys/conf/files
   1.2       +55 -3     src/sys/ddb/db_capture.c
   1.76      +1 -0      src/sys/ddb/db_command.c
   1.1       +555 -0    src/sys/ddb/db_textdump.c (new)
   1.46      +23 -0     src/sys/ddb/ddb.h
   1.187     +9 -1      src/sys/kern/kern_shutdown.c



---------- Forwarded message ----------
Date: Wed, 26 Dec 2007 11:35:07 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/share/man/man4 Makefile ddb.4 textdump.4

rwatson     2007-12-26 11:35:07 UTC

   FreeBSD src repository

   Modified files:
     share/man/man4       Makefile ddb.4
   Added files:
     share/man/man4       textdump.4
   Log:
   Add textdump(4) man page to describe the textdump facility and provide
   some stock formulas for use.

   Update ddb(4) to reference the textdump(4) page, list the textdump
   commands, and suggest using them with scripts and output capture.
   Update HISTORY section.

   Hook up textdump(4) to build.

   MFC after:      3 months

   Revision  Changes    Path
   1.401     +1 -0      src/share/man/man4/Makefile
   1.46      +31 -0     src/share/man/man4/ddb.4
   1.1       +162 -0    src/share/man/man4/textdump.4 (new)



---------- Forwarded message ----------
Date: Wed, 26 Dec 2007 11:42:10 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sbin/savecore savecore.8 savecore.c

rwatson     2007-12-26 11:42:10 UTC

   FreeBSD src repository

   Modified files:
     sbin/savecore        savecore.8 savecore.c
   Log:
   Teach savecore(8) how to extract textdump(4) dumps.

   Update savecore(8) man page to reflect textdump additions.

   MFC after:      3 months

   Revision  Changes    Path
   1.26      +11 -0     src/sbin/savecore/savecore.8
   1.79      +160 -79   src/sbin/savecore/savecore.c
Received on Wed Dec 26 2007 - 10:54:18 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:24 UTC