ULE 2.0

Hello everyone,

After a considerable vacation from ULE I have come back to address some 
long standing concerns.  I felt that the old double-queue mechanism caused 
very unnatural behavior and have finally come up with something I'm happy 
to replace it with.  I've been working on this off and on for several 
months now.  Some details are below.  More are at:
http://jeffr-tech.livejournal.com/3729.html

The version now in CVS(1.172) should restore ULE's earlier interactive 
performance under load.  I have tested with a make -j128 kernel while 
using mozilla and while playing a dvd.  Neither ever skip for 
me.  nice now has a more gradual effect than before.  It no longer allows 
the total starvation of processes.  ULE should also be very slightly 
faster on UP as compared to before.  SMP behavior should have changed very 
little although I did simplify some small parts of these algorithms.  In 
general, non-interactive tasks are scheduled much more intelligently 
although this may not be apparent under most workloads.

I'm hoping for the following types of feedback from anyone interested in 
testing:

1)  Is the response to nice levels as you would hope?  I think nice +20 
may not inhibit the nice'd thread enough at the moment.
2)  Is the interactive performance satisfactory?
3)  Is there any performance degredation for your common tasks?
4)  Does the cpu estimator give reasonable results?  See %cpu in top.  It 
is expected that there will be periods where summing up all threads will 
yield slightly over 100% cpu.

Any and all feedback is welcome.  Please make sure any problem reports are 
sent to jroberson_at_chesapeake.net in the to line so I see them more 
quickly.

Thanks,
Jeff

---------- Forwarded message ----------
Date: Thu, 4 Jan 2007 08:56:25 +0000 (UTC)
From: Jeff Roberson <jeff_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sys/kern sched_ule.c

jeff        2007-01-04 08:56:25 UTC

   FreeBSD src repository

   Modified files:
     sys/kern             sched_ule.c
   Log:
   ULE 2.0:
    - Remove the double queue mechanism for timeshare threads.  It was slow
      due to excess cache lines in play, caused suboptimal scheduling behavior
      with niced and other non-interactive processes, complicated priority
      lending, etc.
    - Use a circular queue with a floating starting index for timeshare threads.
      Enforces fairness by moving the insertion point closer to threads with
      worse priorities over time.
    - Give interactive timeshare threads real-time user-space priorities and
      place them on the realtime/ithd queue.
    - Select non-interactive timeshare thread priorities based on their cpu
      utilization over the last 10 seconds combined with the nice value.  This
      gives us more sane priorities and behavior in a loaded system as
      compared to the old method of using the interactivity score.  The
      interactive score quickly hit a ceiling if threads were non-interactive
      and penalized new hog threads.
    - Use one slice size for all threads.  The slice is not currently
      dynamically set to adjust scheduling behavior of different threads.
    - Add some new sysctls for scheduling parameters.

   Bug fixes/Clean up:
    - Fix zeroing of td_sched after initialization in sched_fork_thread() caused
      by recent ksegrp removal.
    - Fix KSE interactivity issues related to frequent forking and exiting of
      kse threads.  We simply disable the penalty for thread creation and exit
      for kse threads.
    - Cleanup the cpu estimator by using tickincr here as well.  Keep ticks and
      ltick/ftick in the same frequency.  Previously ticks were stathz and
      others were hz.
    - Lots of new and updated comments.
    - Many many others.

   Tested on:      up x86/amd64, 8way amd64.

   Revision  Changes    Path
   1.172     +332 -412  src/sys/kern/sched_ule.c