Re: MAXCPU preparations

On Wed, Sep 29, 2010 at 1:54 PM, Robert N. M. Watson
<rwatson_at_freebsd.org> wrote:
>
> On 29 Sep 2010, at 12:49, John Baldwin wrote:
>
>> On Tuesday, September 28, 2010 6:24:32 pm Robert N. M. Watson wrote:
>>>
>>> On 28 Sep 2010, at 19:40, Sean Bruno wrote:
>>>
>>>>> If you go fully dynamic you should use mp_maxid + 1 rather than maxcpus.
>>>>
>>>> I assume that mp_maxid is the new kern.smp.maxcpus?  Can you inject some
>>>> history here so I can understand why one is "better" than the other?
>>>
>>> So, unlike maxcpus, mp_maxid is in theory susceptible to races in a brave new world in which we support hotplug CPUs -- a brave new world we're
>> not yet ready for, however. If you do use mp_maxid, be aware you need to add one to get the size of the array -- maxcpus is the number of CPUs that
>> can be used, whereas mp_maxid is the highest CPU ID (counting from 0) that is used.
>>
>> Hmm, I'm not sure that is true.  My feeling on mp_maxid is that it is the
>> largest supported CPUID.  Platforms that support hotplug would need to set
>> mp_maxid such that it has room for hotpluggable CPUs.  You don't want to
>> go reallocate the UMA datastructures for every zone when a CPU is hotplugged
>> for example.
>
> Yes, we'll have to break one (or even both) of two current assumptions with the move to hotplug: contiguous in-use CPU IDs and mp_maxid representing the greatest possible CPU ID as a constant value. The former is guaranteed, but I wonder about the latter. On face value, you might say "oh, we know how many sockets there are", but if course, we don't know how many threads will arrive when a package is inserted.  For many subsystems, DPCPU will present a more than adequate answer for avoiding resizing, although not for low-level systems (perhaps such as UMA?). Likewise, on virtual machine platforms where hotplug actually might reflect a longer-term scheduling choice by the admin/hypervisor (i.e., resource partitioning), it may be harder to reason about what the future holds.
>

As a historical note, when AIX added hotplug CPU support, we kept the
MAXCPU define as the largest number of CPUs supported by the OS image.
At the time it was 256; as it shifted to 1024 there was a large
cleanup effort to eliminate as many statically sized arrays as
possible.

AIX also has an mp_maxid equivalent which changed when new higher core
numbers were used.  For various reasons, new CPUs were added only
while a single CPU was running, so any loop that looked like (for i =
0; i <= mp_maxid; i++) could get interrupted by an IPI (the handler
spun until the core was added by the coordinating CPU) and find that
it had a stale mp_maxid value.  This was not a bug in 99% of the uses
of the code, since whatever it was doing with each CPU was either not
atomic (e.g. summing per-cpu counters) or was some kind of
initializing work which was also done by the new CPU before it was
fully awake.

I don't necessarily recommend this as an architected plan for adding
CPUs, but I wanted to offer the historical note that it could work.

Also, CPU id's may not be contiguous with hotplug.  Even if they are
contiguous on boot, there is no reason to assume CPUs are offlined
from highest ID down.  For reasons of cache sharing, the best
performance may be obtained by picking a specific CPU to offline.  It
also may be that it makes more sense to reserve CPU ids so that e.g.
CPUs N*T through N*(2T-1) are all HMT threads on the same core, (for
T-way HMT).  In this case CPU IDs become sparse if HMT is completely
disabled, and the best performance for the box overall is probably
obtained by offlining T3 and T2 for each core while keeping T0 and T1
active.

But discontiguous IDs shouldn't be a problem as all the code I've seen
checks CPU_ABSENT(i) in the loop.

Cheers,
matthew