[CFT] Callchain capture in PmcTools

I'm pleased to offer a patch implementing callchain capture for
hwpmc(4) for review.

Test reports, comments etc. are welcome.

Regards,
Koshy

=============================================
>>>> Summary  <<<<

hwpmc(4):

* hwpmc(4) can now walk kernel and user stacks and capture caller
  information for subsequent analysis by pmcstat(8).
* An additional flag in the API `PMC_F_CALLCHAIN' controls whether
  a PMC will capture a callchain.
* Tunable kern.hwpmc.callchaindepth controls the maximum depth to
  which stacks are walked.  The default is `8'.

pmcstat(8):

* pmcstat(8) now defaults to allocating PMCs that capture callchains.
  Use the new `-N' option (a toggle) to turn this off.
* The '-g' option (gmon.out generation) now writes call arc data for
  subsequent analysis by gprof(1).
* The new '-G' option generates system-wide callchain summaries.
  The new '-z maxprintdepth' option restricts the depth of the
  the callchain summary.


>>>> Patch Information <<<<

1) Download the patch

   % fetch \
   http://people.freebsd.org/~jkoshy/download/pmctools-callgraph-patch.gz

   MD5 (pmctools-callgraph-patch.gz) = 632185755d1004e82c3d2bbc69827307

2) Apply using patch -p1 against a recent (Aug 3rd) -current:

   % cvs checkout -P src
   % cd src; gzip -dc pmctools-callgraph-patch.gz | patch -p1

3) Build and update the kernel+world.

   You will need to add 'options HWPMC_HOOKS' to your kernel config
   before you can use hwpmc(4).

   % (follow the usual procedure, see src/UPDATING)

>>>> Using the patch <<<<

1) Load hwpmc into the kernel

   % kldload hwpmc

2) Collect measurements

   % pmcstat -S instructions -O logfile etc.

3) Use option -g to generate gprof(1) style 'gmon.out' files.

   % pmcstat -R logfile -g
   % gprof /boot/kernel/kernel <canonical-pmc-name>/kernel.gmon

4) Use option -G to generate a callchain summary:

   % pmcstat -R logfile -G summaryfile
   % vi summaryfile

>>>> Known Bugs with this patch <<<<

1) P4 HTT lockup

Symptom:    Lockup under load of Pentium 4 machines with HTT enabled.

Workaround: Restrict sampling to one CPU using the '-c' option:

	    # pmcstat -c 0 -S instructions -O logfile

2) pmcstat(8) stuck in an unkillable state.

Symptom:    When interrupted, pmcstat(8) gets stuck sleeping on
	    wait channel "pmcctx".
Workaround: Use the '-n' option to reduce sampling frequency:

	    # pmcstat -n 1048576 -S instructions ...other options...	


Other (older) known bugs are listed at http://wiki.freebsd.org/PmcTools.

>>>> Other Notes <<<<

pmcstat(8) works best with unstripped executables (e.g. set "STRIP="
in /etc/make.conf).

On the amd64 the heuristic used to determine the frame pointer given a
sampled PC address is not very good and can at times result in the
next to topmost frame being missed in the sampled callchain.

>>>> Thanks <<<<

 - To the users of PmcTools in the FreeBSD community for their
   feedback, encouragement and support.

 - To the FreeBSD Foundation and Google Inc., for supporting this work.

=============================================