Re: dtrace not working on bhyve VM without invariant_tsc

On Mon, Dec 09, 2019 at 09:27:01PM -0500, Ryan Stone wrote:
> I have a bhyve VM guest on my laptop where dtrace just constantly
> aborts whenever I try to use it:
> 
> [rstone_at_ebpf dtrace]sudo dtrace -s fdcopy.d
> Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file
> /usr/home/rstone/git/bsd-worktree/ebpf-import/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c,
> line 3026.
> Abort trap
> 
> I believe that the problem is caused by dtrace unconditionally using
> rdtsc() to implement dtrace_gethrtime(), assuming that the values will
> be stable for a given CPU.  The VM's vcpus seem to be getting migrated
> frequently.
> 
> Should dtrace instead be using the system timecounter?  That should
> stand a much better chance of being monotonically increasing.

One complication here is that the timecounter must be readable without
calling any symbols in the kernel, to avoid probe recursion.  That is,
dtrace_gethrtime() cannot unconditionally use the system timecounter.
It might be possible to use the timecounter specifically for
dtbd_timestamp, but I suspect that will just shift the problem
elsewhere: dtrace uses the TSC to provide a global order for individual
records.

As Eric noted you can pin the vCPUs, but I'm surprised that laptop does
not have an invariant TSC.  DTrace used to attempt to synchronize TSCs,
but I disabled this for guests in r345359.