Re: Fwd: UMA initialization failure with 48 core ARM64

2015-05-22 16:36 GMT+02:00 Konstantin Belousov <kostikbel_at_gmail.com>:

> On Fri, May 22, 2015 at 04:14:13PM +0200, Micha?? Stanek wrote:
> > Success! I am finally able to boot 48 cores. I have been trying out
> > different values given to uma_prealloc() and uma_zone_reserve(). It
> started
> > working when I increased the parameter in uma_prealloc() 32 times and
> left
> > uma_zone_reserve() as it was originally. UMA_BOOT_PAGES also needs to be
> > set to 512.
> >
> > Thank you very much for your help. Do you know how the value to
> > uma_prealloc() should scale with the number of CPUs? If it is not
> obvious,
> > then maybe for now we should make a #define with a value to multiply
> > BT_MAXALLOC by, with a comment that a higher number is required on
> > platforms with many CPUs. What do you think the final fix should look
> like?
>
> I suspect it is not only the number of CPUs which makes the play.  Note
> that the number of tags is already scaled with the number of CPUs.
>
> It is also the question of how much the given architecture needs to
> allocate
> before the normal uma/vmem mechanisms start working.  My quess is that
> arm64
> performes more kva_alloc()s on early stages than other architectures.
>

I am forwarding the result of my conversation with Konstantin Belousov.
With his help I was able to boot 48 cores on an arm64 platform. I needed to
set UMA_BOOT_PAGES=512 and increase the parameter given to uma_prealloc()
in vmem_startup() 32 times (giving 32 * BT_MAXALLOC). It looks like this
should be made configurable to avoid running out of space for initial
allocations on some platforms. In our case, the panic happened still in
SI_SUB_VM sysinit.

Best regards,
Michal Stanek