Re: cpuset and affinity implementation

On Mon, Feb 25, 2008 at 03:12:00PM -1000, Jeff Roberson wrote:
> 
> On Mon, 25 Feb 2008, Brooks Davis wrote:
> 
>> On Sun, Feb 24, 2008 at 05:38:37PM -1000, Jeff Roberson wrote:
>>> Hello,
>>> 
>>> I have implemented a new api similar to processors sets on solaris.  This
>>> allows you to assign processes to sets of cpus and dynamically change 
>>> those
>>> sets.  This is useful for provisioning purposes to add and remove cpu
>>> resources for a particular process or group of processes.  This new
>>> facility also supports binding secific threads to specific cpus which 
>>> some
>>> applications may want to do.  At some point in the future this will be
>>> integrated with jail so you can restrict the cpus any jail is allowed to
>>> use.
>>> 
>>> This api should not be considered final and the 'cpuset' tool is quite
>>> rough.  This also only works with ULE and is unfortunately intertwined 
>>> with
>>> a big ULE patch I've been working on.  The set management code is generic
>>> but 4BSD doesn't contain the hooks to actually constrain threads.
>> 
>> I took a look at the patch this morning.  The API looks like it's
>> capable of doing what I need, at least at a first pass.  It looks like I
>> should be able to implement the semantics currently employed by the Sun
>> Grid Engine scheduler on Irix systems.
>> 
>> The one thing I noticed that I found worrying was the recursion in
>> cpuset_(test)update().  It wasn't immediately clear to me there there
>> is anything to would prevent an arbitrarily deep hierarchy from being
>> created and blowing the kernel stack.  I'm I missing something?
> 
> Yes, presently it can never be more than 3 levels deep.  Once we have jails 
> the max would be 6 levels, unless you can make jails within jails.
> 
> There is presently now way for the user to create a cpuset that is a subset 
> of another set.  So the three cpu sets are:
> 
> 1)  Root set - immutable, all cpus, may be root of jail in which case root 
> outside of the jail can change the set.
> 2)  cpuset - the set this process is a member of.
> 3)  mask - the anonymous set that is applied to an individual thread.

OK.  That makes sense.  It would be useful from my perspective if
creating a root set (or an otherwise inescapable set) was not explicitly
tied to jails.  I could see doing this either by extending the syscalls
or by introducing a more fine grained light weight virtualization as was
discussed in Milan.  This would probably want to be a privileged operation
regardless.

> Did you look at the userland tool at all?  I think this needs the most 
> improvement.  I basically just made something that would allow me to pass 
> every possible parameter to the api.  Not exactly engineered for usability.

I glanced at it, but haven't really thought about what should/shouldn't be
there much.  I'll try to do that in the next day or so.

-- Brooks