Re: Automated submission of kernel panic reports: sysutils/panicmail

On 11/4/13, 2:41 AM, Colin Percival wrote:
> Hi all,
>
> After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have
> now added sysutils/panicmail to the FreeBSD ports tree.  If you install this
> and add
> 	panicmail_enable="YES"
> to your /etc/rc.conf, a panic report will be generated and sent to root_at_ for
> you to review and submit (via email).  You can skip the reviewing step and
> submit panics automatically by setting panicmail_autosubmit="YES".
>
> The panics submitted are encrypted to an RSA key which I hold in order to keep
> them secure in transit; and I intend to keep the raw panic reports confidential
> except to the minimum extent necessary for other developers to help me process
> the incoming reports.
>
> If I receive enough panic reports to be useful, I hope to provide developers
> with aggregate statistics.  This may include:
>
> * regular email reports listing the "top panics", to help guide developers
> towards the most fertile areas for stability improvements;
>
> * email to specific developers alerting them to recurring panics in code they
> maintain (especially if it becomes clear that the panic has been recently
> introduced); and
>
> * guidance to re_at_ and secteam_at_ about how often a particular panic occurs if
> an errata notice is being considered
>
> as well as other yet-to-be-imagined reports of a similarly aggregate and
> anonymized nature.
>
> So please install the sysutils/panicmail port and enable it in rc.conf!  This
> all depends on getting useful data, and I can't do that without your help.
>
Colin, have you had a few minutes to check out the crash reporting 
facilities in FreeNAS?

The reason I ask is that:

1) we would like to share code.
2) we have this running for a few months now and have a huge corpus of 
information.
3) we are building a nice UI (screenshots attached) over it, we have a 
couple of thousands of lines of code we can share for this.

Our scripts can be found here:

1) A startup script that sends us the crashes on system start:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/etc/rc.d/ix_textdump 

2) A script to submit data at boot OR from command line that sends more 
comprehensive system information "ixdiagnose":
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/ixdiagnose 

3) A very simple script to upload that report:
https://github.com/freenas/freenas/blob/master/nanobsd/Files/usr/local/bin/crashuploader 


We send a minimal set of information: kernel stack trace, ddb buffer and 
hardware.  Just enough to get some very, very handy stuff.

I can share with you offline the crash server code, it's django and 
relatively straight forward.

The screenshots can also be seen at:
http://people.freebsd.org/~alfred/crashreporter/

We could modify our framework for FreeBSD to do so by checking for a 
sentinel file depending on the host type and only auto-sending if we see 
that.


-Alfred