Re: Possible Threading problem with -CURRENT / MySQL?

On Thu, 17 Jun 2004, Jon Noack wrote:

> > ULE seems to do a very good job of scheduling interactive tasks over
> > other workloads, resulting in a very "snappy" feel on my boxes,
> > despite heavy CPU load from background builds, etc.  The workload I
> > looked at had no real "interactive" component, although it was a
> > latency-centric RPC test, so timely hand-off as well as high
> > throughput would be important.  I know that Jeff's measurement work
> > on ULE had a substantial focus on deadlines -- whether or not ULE was
> > timely in scheduling tasks, etc, and that he demonstrated that it was
> > much stronger than most other available schedulers in this area.
> 
> Well, I switched to 4BSD and noticed this right away.  I occasionally
> get sound hiccups under heavy i/o with ULE [1], but with 4BSD it's a bit
> ridiculous.  There's a PR open about it (can't remember which off the
> top of my head), but sound will skip and slow down (play at half speed
> or slower).  I know the sound locking is not all it should be, so this
> may not be a representative test.  Regardless, my workstations are back
> on ULE... 

Hrm.  So, I'm not really a scheduler guy, but I have some ideas about how
to investigate what's going on.  If you feel like playing with kernel
tracing facilities, there are some really neat things you can do with
ktr(4).  It allows you took hook context switch and interrupt delivery
events and dump a trace to userspace. 

One possible cause of the symptoms you're seeing is poor interrupt
handling response time -- possibly a failure to schedule the ithread in a
timely manner.  It would be quite interesting to compare the latency in
scheduling (and running) the ithread across ULE and 4BSD and see how well
each is managing to do in getting it running quickly.  I'm not sure
there's a tutorial on how to get KTR(4) running -- I found the man page
helpful, but a little confusing.  What you'll want to do, assuming you're
willing to get into this, is turn on KTR tracing with KTR_INTR and
KTR_PROC.  You'll need to set the flags in the kernel compile, and then
set them using a sysctl.  You can use ktrdump to dump a record trace --
I'd use a number of flags present to add timestamps, cpu numbers if SMP,
etc.  With some post-processing, it should be possible to generate a
distribution of "interrupt time" to "running interrupt handler". 

It would be useful to demonstrate to oneself that ithreads are running
quickly, preempting things they need to, etc.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research