Re: RFC: New event timers infrastructure

On Tue, Jun 8, 2010 at 12:24 AM, Tsuyoshi Ozawa <ozawa_at_t-oza.net> wrote:
> 2010/6/6, Alexander Motin <mav_at_freebsd.org>:
>> Hi.
>>
>> Most of x86 systems now has at least 4 types of event timers: i8254,
>> RTC, LAPIC and HPET. Respective code in kernel is very tangled, heavily
>> hardcoded and absolutely not scalable. I have reimplemented it, trying
>> to solve these issues.
>>
>> I did such things:
>>  - created unified timer driver's API (sys/timeet.h, kernel/kern_et.c).
>> It supports global and per-CPU timers, periodic and one-shot. Provides
>> driver and consumer interfaces for choosing timers and operating them;
>>  - cleaned existing x86 event timer driver's code and modified it for
>> new API (x86/isa/atrtc.c, x86/isa/clock.c, x86/x86/local_apic.c). LAPIC
>> timer is now per-CPU and supports both periodic and one-shot modes;
>>  - extended HPET driver to support it's event timers in periodic and
>> one-shot mode (dev/acpica/acpi_hpet.c). Support for per-CPU operation
>> and FSB interrupts planned for later;
>>  - written mostly machine-independent mid-layer for managing any present
>> timers to provide clocks needed for kernel (x86/x86/timeevents.c). It
>> supports both global and per-CPU timers. Now it supports only periodic
>> mode, but one-shot mode support planned for later.
>>
>> All this stuff deeply configurable via both loader tunables on boot and
>> sysctls in real time:
>>
>> %sysctl kern.eventtimer
>> kern.eventtimer.choice: LAPIC(500) HPET(400) HPET1(390) HPET2(390)
>> i8254(100) RTC(0)
>> kern.eventtimer.et.LAPIC.flags: 7
>> kern.eventtimer.et.LAPIC.frequency: 99752386
>> kern.eventtimer.et.LAPIC.quality: 500
>> kern.eventtimer.et.HPET.flags: 3
>> kern.eventtimer.et.HPET.frequency: 14318180
>> kern.eventtimer.et.HPET.quality: 400
>> kern.eventtimer.et.HPET1.flags: 3
>> kern.eventtimer.et.HPET1.frequency: 14318180
>> kern.eventtimer.et.HPET1.quality: 390
>> kern.eventtimer.et.HPET2.flags: 3
>> kern.eventtimer.et.HPET2.frequency: 14318180
>> kern.eventtimer.et.HPET2.quality: 390
>> kern.eventtimer.et.RTC.flags: 1
>> kern.eventtimer.et.RTC.frequency: 32768
>> kern.eventtimer.et.RTC.quality: 0
>> kern.eventtimer.et.i8254.flags: 1
>> kern.eventtimer.et.i8254.frequency: 1193182
>> kern.eventtimer.et.i8254.quality: 100
>> kern.eventtimer.timer2: NONE
>> kern.eventtimer.timer1: i8254
>> kern.eventtimer.singlemul: 2
>>
>> By default system chooses two timers with highest "quality" for
>> hardclock and statclock/profclock. User may affect that choice via
>> disabling unwanted drivers and/or via direct specification of wanted
>> ones. It is possible to change timers on-flight via sysctls:
>>
>> %sysctl kern.eventtimer.timer1=hpet
>> kern.eventtimer.timer1: i8254 -> HPET
>> %sysctl kern.eventtimer.timer2=hpet1
>> kern.eventtimer.timer2: NONE -> HPET1
>>
>> After every timer change, if two timers available, mid-layer
>> cross-checks them, and if one of them is not functional - replaces it.
>>
>> If there is no second timer available, or user specified to not use it -
>> mid-layer automatically increases rate of the first timer and divide
>> it's frequency to satisfy system needs as good as possible. User may
>> specify how fast he wish to run fist timer relative to hz by setting
>> kern.eventtimer.singlemul tunable/sysctl.
>>
>> When profiling is active, mid-layer automatically rises respective timer
>> frequency to about 8KHz (was 1KHz previously) and decreases it back on
>> profiling end.
>>
>> All above was tested on i386 and amd64. XEN was not affected and builds
>> fine. pc98 was slightly touched. It wasn't tested, but builds fine. It's
>> pc98/cbus/clock.c needs respective rewrite to use new features. Other
>> architectures are untouched, but if any of them may benefit from this
>> functionality - it should be possible to share most of the code.
>>
>> Latest patches can be found here:
>> http://people.freebsd.org/~mav/et.20100606.patch
>>
>> Known issues:
>>  - i8254 timer generates 18Hz interrupt rate when not used and not
>> disabled. I haven't found a way to disable it's interrupt source while
>> holding spinlock.
>>  - timer drivers code will need some more cleaning after interrupt
>> handler will be able to return both argument and frame same time.
>>
>> Feedback is very appreciated.
>>
>> --
>> Alexander Motin
>>
>
> This is excellent!
> I'll try this to apply this patch and  rewrite my old dynamic ticks code to
> fit this event timer API .
>
> Thank you, Alexander !
> --
> Tsuyoshi Ozawa
> <ozawa_at_t-oza.net>
>
> 2010/06/07 7:03 "Alexander Motin" <mav_at_freebsd.org>:
>
> Hi.
>
> Most of x86 systems now has at least 4 types of event timers: i8254,
> RTC, LAPIC and HPET. Respective code in kernel is very tangled, heavily
> hardcoded and absolutely not scalable. I have reimplemented it, trying
> to solve these issues.
>
> I did such things:
>  - created unified timer driver's API (sys/timeet.h, kernel/kern_et.c).
> It supports global and per-CPU timers, periodic and one-shot. Provides
> driver and consumer interfaces for choosing timers and operating them;
>  - cleaned existing x86 event timer driver's code and modified it for
> new API (x86/isa/atrtc.c, x86/isa/clock.c, x86/x86/local_apic.c). LAPIC
> timer is now per-CPU and supports both periodic and one-shot modes;
>  - extended HPET driver to support it's event timers in periodic and
> one-shot mode (dev/acpica/acpi_hpet.c). Support for per-CPU operation
> and FSB interrupts planned for later;
>  - written mostly machine-independent mid-layer for managing any present
> timers to provide clocks needed for kernel (x86/x86/timeevents.c). It
> supports both global and per-CPU timers. Now it supports only periodic
> mode, but one-shot mode support planned for later.
>
> All this stuff deeply configurable via both loader tunables on boot and
> sysctls in real time:
>
> %sysctl kern.eventtimer
> kern.eventtimer.choice: LAPIC(500) HPET(400) HPET1(390) HPET2(390)
> i8254(100) RTC(0)
> kern.eventtimer.et.LAPIC.flags: 7
> kern.eventtimer.et.LAPIC.frequency: 99752386
> kern.eventtimer.et.LAPIC.quality: 500
> kern.eventtimer.et.HPET.flags: 3
> kern.eventtimer.et.HPET.frequency: 14318180
> kern.eventtimer.et.HPET.quality: 400
> kern.eventtimer.et.HPET1.flags: 3
> kern.eventtimer.et.HPET1.frequency: 14318180
> kern.eventtimer.et.HPET1.quality: 390
> kern.eventtimer.et.HPET2.flags: 3
> kern.eventtimer.et.HPET2.frequency: 14318180
> kern.eventtimer.et.HPET2.quality: 390
> kern.eventtimer.et.RTC.flags: 1
> kern.eventtimer.et.RTC.frequency: 32768
> kern.eventtimer.et.RTC.quality: 0
> kern.eventtimer.et.i8254.flags: 1
> kern.eventtimer.et.i8254.frequency: 1193182
> kern.eventtimer.et.i8254.quality: 100
> kern.eventtimer.timer2: NONE
> kern.eventtimer.timer1: i8254
> kern.eventtimer.singlemul: 2
>
> By default system chooses two timers with highest "quality" for
> hardclock and statclock/profclock. User may affect that choice via
> disabling unwanted drivers and/or via direct specification of wanted
> ones. It is possible to change timers on-flight via sysctls:
>
> %sysctl kern.eventtimer.timer1=hpet
> kern.eventtimer.timer1: i8254 -> HPET
> %sysctl kern.eventtimer.timer2=hpet1
> kern.eventtimer.timer2: NONE -> HPET1
>
> After every timer change, if two timers available, mid-layer
> cross-checks them, and if one of them is not functional - replaces it.
>
> If there is no second timer available, or user specified to not use it -
> mid-layer automatically increases rate of the first timer and divide
> it's frequency to satisfy system needs as good as possible. User may
> specify how fast he wish to run fist timer relative to hz by setting
> kern.eventtimer.singlemul tunable/sysctl.
>
> When profiling is active, mid-layer automatically rises respective timer
> frequency to about 8KHz (was 1KHz previously) and decreases it back on
> profiling end.
>
> All above was tested on i386 and amd64. XEN was not affected and builds
> fine. pc98 was slightly touched. It wasn't tested, but builds fine. It's
> pc98/cbus/clock.c needs respective rewrite to use new features. Other
> architectures are untouched, but if any of them may benefit from this
> functionality - it should be possible to share most of the code.
>
> Latest patches can be found here:
> http://people.freebsd.org/~mav/et.20100606.patch
>
> Known issues:
>  - i8254 timer generates 18Hz interrupt rate when not used and not
> disabled. I haven't found a way to disable it's interrupt source while
> holding spinlock.
>  - timer drivers code will need some more cleaning after interrupt
> handler will be able to return both argument and frame same time.
>
> Feedback is very appreciated.
>
> --
> Alexander Motin

This is an exciting advance for FreeBSD, and I'm looking forward to
testing the new infrastructure code.

Alexander, do you feel that the code is at a stage where meaningful
user testing can occur?

Tsuyoshi, will you be providing code to Alexander to integrate into
his patch(es), or will your development continue as a separate effort?

Thank you both so much for taking on this task!

-Brandon