On Tuesday 18 December 2007 14:10:46 Robert Watson wrote: > Dear all: > > I've been hacking on-and-off for a while on a side project to improve > our kernel debugging facilities. Primarily, my concern has been to > address three problems: > > - The complications of employing kernel core dumps for debugging, > including the large size of dumps making them unwieldy to distribute > or store for any extended period (even with minidumps), the requirement > to have relatively synchronized kernel source in order to use the dumps, > the need to have a kernel with debugging symbols, and the problems with > fsck causing sufficient swap use to invalidate dumps before they can be > extracted. > > - The decreasing likelihood that notebooks will ship with serial ports > that can be used for interactive debugging using DDB. Making > end-users type in stack traces is cruel, photos are a pain, and X11 > rules out both. > > - The fact that a great many problems are most easily diagnosed using > utility routines present in DDB, but not as easily using kgdb for > offline analysis. I find that for many bugs I analyze, simply > looking at the DDB output is sufficient to identify the source of the > problem. > > An idea I punted around a bit at BSDCan earlier this year (or perhaps it > was at EuroBSDCon the previous year) was an idea of a "textdump" -- that > is, a new type of kernel dump based on capturing automatically extracted > debugging information generated by DDB. The result would be an ASCII > text file that could be filed as a bug report, perhaps even > automatically. > > To this end, I have implemented three new facilities for use with DDB: > > (1) DDB output capture. The output of DDB is stored in a memory buffer, > and can be extracted using a sysctl or textdumps (see below). This > can be turned on and off, both for use manually ("I'll want this > later, but not that") and as part of scripts (see below). > > (2) DDB scripting. A limited number of named scripts can be defined to > run a series of DDB commands. No loops, etc, just simple command > lists. These can be caused to run automatically on entering DDB > for various scenarios, including WITNESS violations and kernel panics. > They can also be run by hand in order to save a bit of typing if you use > DDB in a repetitive way (as I do). > > (3) Textdumps. A new dump type that stores a series of data files > containing various pieces of information, including the DDB capture > buffer, kernel message buffer, kernel configuration (if compiled > into the kernel), panic message, and kernel version string. These are > stored in the ustar format inside the dump partition (aligned to the > end) so can be easily extended, and savecore(8) requires almost no new > logic to deal with them (it just drops numbered tar files in > /var/crash). This makes it straight forward to extend the textdump > format to include new types of information and avoids the issue of how > to safely simultaneously represent information in many different formats > in the same file. > > These are pretty flexible tools, and you can imagine doing the following > sorts of things: > > - Setting the kdb.enter.panic script to automatically turn on output > capture, do full backtraces of all threads, show open file > information, dump UMA stats, and save it all to a textdump and then > reboot. > > - Setting the kdb.enter.witness script to show lock information, > generate a coredump, and reboot. Or, just to automatically do "show > allocks" and drop to the DDB prompt. > > - Adding a flag to rc.conf to automatically submit textdumps via e-mail > to a specific address, perhaps including GNATS or an automated bug > system. These could be unpacked and automatically analyzed, and do to > the compact size, kept for long-term trend analysis or to identify when > a problem started occuring. > > I've produced an initial snapshot of the above, which can be found here: > > http://www.watson.org/~robert/freebsd/20071218-ddb.tgz > > This adds three files to DDB, patches quite a few kernel files (to pass > more information into KDB about why it's being entered, in order to > trigger the right script), enhancements to savecore(8) to know how to > extract textdumps, adds a ddb(8) command line tool so that userspace can > manage DDB scripts from outside the debugger, extensions to the ddb(4) > man page, and a new textdump(4) man page. > > There are a number of known limitations; I've tried to document them at > the top of the pertinent files where I am aware of them. I also regret > to say that to date I've been able to test only on i386, and not other > platforms. I'd welcome any feedback -- I'd like to get these changes > into CVS in the next week or two. It looks like some files is not included in the patch. I have this error: make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP="cc -E" CC="cc" xargs mkdep -a -f .newdep -O -pipe -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding cc: /usr/src/sys/ddb/db_capture.c: No such file or directory cc: /usr/src/sys/ddb/db_script.c: No such file or directory cc: /usr/src/sys/ddb/db_textdump.c: No such file or directory mkdep: compile failed *** Error code 1 Stop in /usr/obj/usr/src/sys/GENERIC. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. root_at_orion-vm:/usr/src# ls -la /usr/src/sys/ddb/ total 424 drwxr-xr-x 2 root wheel 1024 Dec 19 16:49 ./ drwxr-xr-x 53 root wheel 1024 Oct 29 08:24 ../ -rw-r--r-- 1 root wheel 2591 Dec 4 2005 db_access.c -rw-r--r-- 1 root wheel 1431 Jan 6 2005 db_access.h -rw-r--r-- 1 root wheel 7737 Jan 6 2005 db_break.c -rw-r--r-- 1 root wheel 2098 Jan 6 2005 db_break.h -rw-r--r-- 1 root wheel 16579 Dec 19 16:49 db_command.c -rw-r--r-- 1 root wheel 15705 Jan 17 2007 db_command.c.orig -rw-r--r-- 1 root wheel 1633 Dec 19 16:49 db_command.h -rw-r--r-- 1 root wheel 1588 Jan 6 2005 db_command.h.orig -rw-r--r-- 1 root wheel 7270 Oct 27 20:19 db_examine.c -rw-r--r-- 1 root wheel 4811 Jan 6 2005 db_expr.c -rw-r--r-- 1 root wheel 7981 Dec 19 16:49 db_input.c -rw-r--r-- 1 root wheel 7931 Jan 6 2005 db_input.c.orig -rw-r--r-- 1 root wheel 5978 Dec 19 16:49 db_lex.c -rw-r--r-- 1 root wheel 5304 Jan 6 2005 db_lex.c.orig -rw-r--r-- 1 root wheel 1951 Dec 19 16:49 db_lex.h -rw-r--r-- 1 root wheel 1861 Jan 6 2005 db_lex.h.orig -rw-r--r-- 1 root wheel 5976 Dec 19 16:49 db_main.c -rw-r--r-- 1 root wheel 5787 Nov 6 2006 db_main.c.orig -rw-r--r-- 1 root wheel 6889 Dec 19 16:49 db_output.c -rw-r--r-- 1 root wheel 6639 Oct 10 2006 db_output.c.orig -rw-r--r-- 1 root wheel 1444 Oct 8 2006 db_output.h -rw-r--r-- 1 root wheel 2013 Apr 14 2005 db_print.c -rw-r--r-- 1 root wheel 10956 Nov 13 13:43 db_ps.c -rw-r--r-- 1 root wheel 8810 Apr 14 2005 db_run.c -rw-r--r-- 1 root wheel 7815 Jun 16 2006 db_sym.c -rw-r--r-- 1 root wheel 3591 Jan 6 2005 db_sym.h -rw-r--r-- 1 root wheel 5143 Jan 17 2007 db_thread.c -rw-r--r-- 1 root wheel 3402 Jan 6 2005 db_variables.c -rw-r--r-- 1 root wheel 1854 Jan 6 2005 db_variables.h -rw-r--r-- 1 root wheel 7218 Nov 17 2006 db_watch.c -rw-r--r-- 1 root wheel 1509 Jan 6 2005 db_watch.h -rw-r--r-- 1 root wheel 2264 Jan 6 2005 db_write_cmd.c -rw-r--r-- 1 root wheel 7467 Dec 19 16:49 ddb.h -rw-r--r-- 1 root wheel 5776 Jul 12 2006 ddb.h.orig -- ====================================================================== - Best regards, Nikolay Pavlov. <<<----------------------------------- ======================================================================
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:24 UTC