RFC: DTrace probes for debugging or testing in userland programs

Hi,

 I am trying to rewrite userland programs (especially daemons) to
 support userland DTrace probes to make it possible to trace the
 behavior by using dtrace(1).  The purpose is to provide a consistent
 interface to enable/collect debug log and show internal states.  A
 lot of daemons define their own debug output function like dprintf()
 and there is no consistency of where to be recorded.

 I would like your comment about this change because I want to know
 whether this change (or direction) is acceptable or not.  I put an
 example by using syslogd at the following URL:

 (diff)
  https://people.freebsd.org/~hrs/syslogd_usdt.20161220-1.diff
 (tarball)
  https://people.freebsd.org/~hrs/syslogd_usdt.20161220-1.tar.gz

 You can try to compile a new syslogd, run it, and then attach
 dtrace(1) to the syslogd process by "dtrace -q -CI./
 -s ./syslogd_trace.d -p `pgrep syslogd`" in the same directory.

 Basically this change is invisible for normal users.  This includes
 some rough edges but I think it is sufficient to understand the
 concept.  I do not intend to commit this soon.

 Questions from me are the following:

  1. Where should foo_probes.d and foo_trace.d be installed?  And if
     they depend on foo.h, where should foo.h be?

  2. Is documenting probes into foo.8 reasonable?

 The details are as follows.

 * Rewrite example

  This rewrite is twofold.  First, simply replace a call of the
  logging function into DTrace USDT like this:

  (old) | dprintf("an error occurred, errno=%d\n", errno);

  (new) | FOO_LOG(LOG_ERR, "an error occurred, errno=%d\n", errno);

  And then if it involves a result of a specific operation, replace
  trace probes for the simple logging with dedicated ones:

  (old) | error = bind(s, res->ai_addr, res->ar_addrlen);
        | if (error) {
        |     dprintf("bind failed, errno=%d\n", errno);
        |     close(s);
        |     return (-1);
        | }

  (new)
        | error = bind(s, res->ai_addr, res->ar_addrlen);
        | if (error) {
        |     FOO_SOCK_BIND_FAILED(errno, s, res->ai_addr, res->ar_addrlen);
        |     close(s);
        |     return (-1);
        | }
        | FOO_SOCK_BIND_SUCCESS(s, res->ai_addr, res->ar_addrlen);

  and implement the functionality of the original log message in D
  script:

  | foo$target:::sock-bind-failed
  | {
  |     printf("bind failed, errno=%d", arg0);
  |     /* The other argN can be used to report in more detail. */
  | }

 * Additional files in the existing directory layout

  After this rewrite, the directory layout for a daemon "foo" will look
  like the following:

  | Makefile
  | foo.8
  | foo.c
  | foo.h
  | foo_probes.d
  | foo_trace.d

  foo_probes.d and foo_trace.d are added.  The former is the
  definition of USDT probes, and the latter is a D script example to
  reproduce the original debug log by dprintf() or something like
  that.  A section to describe what probes are available is added into
  foo.8.  One can trace the foo daemon using "dtrace -Cs foo_trace.d -p
  `pgrep foo`" on runtime, and also can create own script.

  foo.h may be added because foo_probes.d and foo_trace.d often require
  information of data structure used in foo.c.

 * Possible incompatible change

  A debug flag to activate additional logging is no longer necessary
  after this rewrite, so we can remove it (-d flag in the case of
  syslogd).  And dump of the internal state can be implemented as a
  SIGINFO handler.  In the syslogd example, SIGINFO dumps syslogd
  configuration and access control list.

-- Hiroki