RFC: DTrace probes for debugging or testing in userland programs

Hello,


>  You can try to compile a new syslogd, run it, and then attach
>  dtrace(1) to the syslogd process by "dtrace -q -CI./
>  -s ./syslogd_trace.d -p `pgrep syslogd`" in the same directory.

one thing that comes to mind is the lack of a way to actually fire these
probes without running plenty of DTrace scripts. The SDT provider
dynamically links onto the binary using the linker set, so that probes
can be called. This is accomplished using the DTrace command that you've
issued, would we need to issue one for each daemon that operates in such
a way?

>  Questions from me are the following:
> 
>   1. Where should foo_probes.d and foo_trace.d be installed?  And if
>      they depend on foo.h, where should foo.h be?
> 
>   2. Is documenting probes into foo.8 reasonable?

This would again depend on the way that they're implemented. If USDTs
are used, documenting them might be beneficial, as the user might at one
point want to turn them off dynamically or write their own script.

>   foo_probes.d and foo_trace.d are added.  The former is the
>   definition of USDT probes, and the latter is a D script example to
>   reproduce the original debug log by dprintf() or something like
>   that.  A section to describe what probes are available is added into
>   foo.8.  One can trace the foo daemon using "dtrace -Cs foo_trace.d -p
>   `pgrep foo`" on runtime, and also can create own script.
> 
>   foo.h may be added because foo_probes.d and foo_trace.d often require
>   information of data structure used in foo.c.
> 
>  * Possible incompatible change
> 
>   A debug flag to activate additional logging is no longer necessary
>   after this rewrite, so we can remove it (-d flag in the case of
>   syslogd).  And dump of the internal state can be implemented as a
>   SIGINFO handler.  In the syslogd example, SIGINFO dumps syslogd
>   configuration and access control list.

One thing that could be done as well is instead of using the SDT
provider, a new provider could be written, which could in turn be
controlled by the additional flags, or pehaps even a sysctl integer that
would specify the level of logging that one would want. This provider
could also be entirely disabled, causing it to disable all the existing
probes and, similarly to SDT, use the linker set to patch a couple of
nops back in.

The user could also attach to this provider with their own scripts,
should they wish to perform some other form of monitoring as well.

This might be redundant with the SDT provider though, so perhaps a more
generic, backwards-compatible way can be thought of that would allow
this sort of behaviour?

-- 
Best regards,
Domagoj Stolfa.