New scheduler interface patch

(CC'd to current)

Discussing the patch at:
http://www.freebsd.org/~julian/f.diff
or available to committers as p4 branch "nsched2"

AS discussed before there were several "goals" in this change.

1/ remove the KSE structure from general view. (if not delete it entirely)
  (I admit it was a mistake to make it visible in the first place.. we learn)
2/ clean up thread and ksegrp shutdown.
3/ Make the libthr interface behave according to the rules that
   threads were supposed to behave by (I am working on documenting this)
    i.e all threads under a single KSEGRP are limitted by
      the ksegrp's concurrency and may be handled in some
      special ways by schedulers.  (e.g. affinity etc).
      Threads under different KSEGRPs are treated as completely
      separate from each other by the schedulers and not restricted
      in any way.. Threads in different processes are treated separatly
      by the scheduler because they are in different ksegrps. the schedulers
      should only dimmly be aware of the concept of a process, but keenly aware
      of kergrps and threads.
  More on this later.
4/ Make the schedulers a bit more independent from each other and, in
a related goal:
5/ move all fields in the thread, and ksegrp structures that were
scheduler internal implementation dependent into the scheduler specific
extensions for those structures.. thus runqueue hooks etc are no longer
generally visible. This allows development of schedulers that use
completely different internal structures, such as puting a thread on
multiple queues at once.

There are a 6th and 7th goal for this particular patch:
6/ No changes to functionality of either sheduler (as far as possible)
7/ Comments, field-names, structure names etc. should be left as much alone
as possible, even if generally making them inaccurate, in order to make
the diff itself as small as possible. This should make it possible for you
or others to prove to yourself that the schedulers are basically unalterred
in a functional manner, without drowning the diff in non-functional
nomenclature and comment changes. Optimisations which can now be made
are NOT included in these diffs for the reason given above.
I also used macros to achieve some of this.

My intention is that after the FUNCTIONAL change has been committed and we
all agree that functionality has not changed, that a completely separate
pass should be made to fix all that stuff and to clean it up..
The aim is to have a working schedule(s) at all times..



Method:
phase 1.
I first made a set of diffs that absteacted all the current KSE handling
to a set of sched_calls, and manipulated the number of KSEs via a
sched_set_concurrency() call. (and some variantes of it.. init_concurrency..
reset_concurrency() etc.) The KSE structires were moved in their entirety
to the sched_{4bsd,ule}.c files to ENSURE that nothing else was accessing
them. Since these new sched_ methods were common to both schedulers I added
them to kern_switch.c (at the bottom). This file was then INCLUDED into both
schedulers so it could pick up their local definitions of the KSE structure
etc.
  (these diffs are available on request)

Phase 2
Once that was done and nothing outside the schedulers could see the KSE 
sructures I then merged the KSE sructure and the td_sched structure.
The resulting structure is the td_sched structure however the fields from
the kse still have their old names and cod estill refers to KSEs and compiles 
due to the macro  "#define kse td_sched" .
This meant that every thread has a KSE at all times. I could then remove
all the kse_alloc() etc. stuff as they are allocated as part of the thread.
Some fields moved from the ksegrp to the kg_sched extension. I added
counters to the ksegrp structure to implement the concurrency calls.

Nothing outside the schedulers was changed in this
phase as all relevent code had already been isolated to the schedulers
in phase 1.

Phase 3 (Not yet done)
After committing and testing, a pass over the code to rename items,correct 
comments, make new optimisations etc. will be made.

Phase 4
Experiment with radically different and revolutionary schedulers.



problems:
* There may be an interration between the concurrency counters and the
preemption code that was just re-enabled. A preemptionmay cause the ksegrp to 
lose track of the real status. I need to check this in the next day.
This could lead to a threaded process 'freezing'.


* Load Average under ULE is completely whacked.. I hope you (jeffr) can see what 
I've done wrong here.

* I think I need to look at the new libthr interface code and make an alternate
thr_create() that allows libthr to create all its threads under a single ksegrp,
with a concurrency of NCPU (the suggested maximum for a ksegrp concurrency).
The old thr_create was a strange hybrid.. All threads scheduled independently
with no cuncurrency limit (i.e able to flood the run queues) but all onder one
KSEGRP. Performance testing will be needed to check which way to go.