Re: Improving the kernel/i386 timecounter performance (GSoC proposal)

Poul-Henning,
Thanks for the feedback!

>[...] these must provide a monotonic timescale when queried interleaved
> ? Be aware that the TSC may not be, and may not stay synchronized across
> multiple cores.


The TSC is documented to be monotonically increasing across all x86
processors that implement it (that I'm aware of). I know that the TSC may
not stay synchronized across multiple cores *in theory*. Practically, across
most processors the only real issue has been an offset in the tsc of cores
relative to each other (which can be measured and accounted for), or one
core losing some ticks wrt the other during specific sleep states (this can
be disabled and is recommended by AMD and Red Hat Linux).

>Further more, the TSC is not constant frequency and in particular not
> "known frequency" at all times.


The TSC is guaranteed to be constant frequency on relatively modern
processors from Intel and AMD — whether the processor we are running on
supports constant TSC rate can be queried via a CPUID instruction. The
frequency can be measured at boot time by using another timing source such
as the PIT, or read directly off the CPU for some models.

>There are a lot of nasty cases to check,


I have implemented many such 'nasty checks' over the past several months
during my work with the xnu kernel — I might have missed some, however. They
are all done once during system boot (and during resume from sleep on some
AMD dual cores). They're not very involved in my opinion.

>and a nasty interpolation required,


Could you please elaborate or hint me on some terms I can google about the
interpolations that are required? Are you referring to the interpolation
needed during measuring the tsc frequency to account for the (weird)
duration of PIT? This happens during bootup only.


>which, in my tests some years back, totally negated any speedup from using
> the TSC in the first place.


This could be an issue: I have not made extensive benchmarks. The benefit of
using TSC could still be: the availability of a higher resolution timer
which can be accessed from userspace.

>At the very minimum, you will have to add a quirk table where known good
> {CPU+MOBO+BIOS} combinations can be entered, as we find them.


Perhaps. Or alternatively, a quirk table for known *bad* combinations. In my
experience, most current x86 processors are OK (tested on Intel Pentium 4
and above, and AMD Athlons and above, with a variety of motherboard/BIOSes).

>Rubbish.  Timecounters are not even closely associated with the tick or
> ticklessness of the kernel.


My understanding could be flawed here, but the reasoning was: for a tickles
kernel, we need some sort of monotonically increasing, known-rate counter as
a replacement for periodic timer interrupts. Using the TSC (or HPET) would
allow us to do so. Unless the alternative is to read the RTC at each call of
gettimeofday() et al, which itself is not foolproof (eg. the user updates
the hardware clock on a running system). I'm not aware of other
high-resolution counters on the x86 platform which can serve this purpose.
The PIT could be read, but it has too little range (16 bits iirc?) to be
useful unless proper wraparound is done. The TSC is 64bits wide and
guaranteed not to wrap around for 10 years or more (cf. Intel manuals).

>the bios may autonomously change the cpu speed


True. This could be an issue. For XNU and the SpeedStep driver we made, we
combat this by disabling such BIOS-initiated frequency changes (refer:
VoodooPower www.superhai.com/darwin.html )

>not knowing exactly _when_ and _how_ the cpu clock changed, is a
> significant number of microseconds, plenty of time to make strange things
> happen.


Yes, for BIOS-initiated cpu frequency changes. For cpufreq driver-initiated
changes, as I mentioned, the kernel can be notified before and after each
change, the duration can be timed using an external timer and accounted for.

>You will want to study carefully Dave Mills work to tame the alpha chips
> wandering SAW clocks.


Will do. I just started reading your paper on timecounters in FreeBSD which
has been quite informative!

Best,
Prashant Vaibhav




On Fri, Mar 27, 2009 at 3:00 AM, Poul-Henning Kamp <phk_at_phk.freebsd.dk>wrote:

> In message <17560ccf0903260551v1f5cba9eu87727c0bae7baa3_at_mail.gmail.com>,
> Prasha
> nt Vaibhav writes:
>
> >The gettimeofday() function's implementation will then be
> >changed to read the timestamp counter (TSC) from the processor, and use
> the
> >reading in conjunction with the timing info exported by the kernel to
> >calculate and return the time info in proper format.
>
> I take it as read, that you know that there are other relvant
> functions than gettimeofday() and that these must provide a
> monotonic timescale when queried interleaved ?
>
> Be aware that the TSC may not be, and may not stay synchronized
> across multiple cores.
>
> Further more, the TSC is not constant frequency and in particular
> not "known frequency" at all times.
>
> There are a lot of nasty cases to check, and a nasty interpolation
> required, which, in my tests some years back, totally negated any
> speedup from using the TSC in the first place.
>
> At the very minimum, you will have to add a quirk table where
> known good {CPU+MOBO+BIOS} combinations can be entered, as we
> find them.
>
> >This will also pave way for optionally making the
> >FreeBSD kernel tickless,
>
> Rubbish.  Timecounters are not even closely associated with the
> tick or ticklessness of the kernel. [1]
>
> >   - The TSC frequency might change on certain processors with
> non-constant
> >   TSC rate (because of SpeedStep, dynamic freq scaling etc.). The only
> way to
> >   combat this is that the kernel be notified every time the processor
> >   frequency changes. Every cpu frequency driver will need to be updated
> to
> >   notify the kernel before and after a cpu freq change.
>
> That is not good enough, the bios may autonomously change the cpu speed
> and the skew from not knowing exactly _when_ and _how_ the cpu clock
> changed, is a significant number of microseconds, plenty of time
> to make strange things happen.
>
> You will want to study carefully Dave Mills work to tame the alpha
> chips wandering SAW clocks.
>
> Poul-Henning
>
> [1] In my mind, reworking the callout system in the kernel would
> be a much better more neded and much more worthwhile project.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk_at_FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>