Re: dtrace not working on bhyve VM without invariant_tsc

From: Mark Johnston <>
Date: Tue, 10 Dec 2019 11:29:36 -0500
On Mon, Dec 09, 2019 at 09:27:01PM -0500, Ryan Stone wrote:
> I have a bhyve VM guest on my laptop where dtrace just constantly
> aborts whenever I try to use it:
> [rstone_at_ebpf dtrace]sudo dtrace -s fdcopy.d
> Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file
> /usr/home/rstone/git/bsd-worktree/ebpf-import/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c,
> line 3026.
> Abort trap
> I believe that the problem is caused by dtrace unconditionally using
> rdtsc() to implement dtrace_gethrtime(), assuming that the values will
> be stable for a given CPU.  The VM's vcpus seem to be getting migrated
> frequently.
> Should dtrace instead be using the system timecounter?  That should
> stand a much better chance of being monotonically increasing.

One complication here is that the timecounter must be readable without
calling any symbols in the kernel, to avoid probe recursion.  That is,
dtrace_gethrtime() cannot unconditionally use the system timecounter.
It might be possible to use the timecounter specifically for
dtbd_timestamp, but I suspect that will just shift the problem
elsewhere: dtrace uses the TSC to provide a global order for individual

As Eric noted you can pin the vCPUs, but I'm surprised that laptop does
not have an invariant TSC.  DTrace used to attempt to synchronize TSCs,
but I disabled this for guests in r345359.
Received on Tue Dec 10 2019 - 15:29:44 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:22 UTC