Re: [RFC] Start SMP subsystem earlier

On 5 January 2015 at 06:08, Hans Petter Selasky <hps_at_selasky.org> wrote:
> On 01/05/15 14:43, Konstantin Belousov wrote:
>>
>> On Mon, Jan 05, 2015 at 02:18:17PM +0100, Hans Petter Selasky wrote:
>>>
>>> Hi,
>>>
>>> There is a limitiation on the number of interrupt vectors available when
>>> only a single processor is running. To have more interrupts available we
>>> need to start SMP earlier when building a monotolith kernel and not
>>> loading drivers as modules. The driver in question is a network driver
>>> and because it cannot be started after SI_SUB_ROOT_CONF due to PXE
>>> support I see no other option than to move SI_SUB_SMP earlier.
>>>
>>> Suggested patch:
>>>
>>>> Index: sys/kernel.h
>>>> ===================================================================
>>>> --- sys/kernel.h        (revision 276691)
>>>> +++ sys/kernel.h        (working copy)
>>>> _at__at_ -152,6 +152,7 _at__at_
>>>>         SI_SUB_KPROF            = 0x9000000,    /* kernel profiling*/
>>>>         SI_SUB_KICK_SCHEDULER   = 0xa000000,    /* start the timeout
>>>> events*/
>>>>         SI_SUB_INT_CONFIG_HOOKS = 0xa800000,    /* Interrupts enabled
>>>> config */
>>>> +       SI_SUB_SMP              = 0xa850000,    /* start the APs*/
>>>>         SI_SUB_ROOT_CONF        = 0xb000000,    /* Find root devices */
>>>>         SI_SUB_DUMP_CONF        = 0xb200000,    /* Find dump devices */
>>>>         SI_SUB_RAID             = 0xb380000,    /* Configure GEOM
>>>> classes */
>>>> _at__at_ -165,7 +166,6 _at__at_
>>>>         SI_SUB_KTHREAD_BUF      = 0xea00000,    /* buffer daemon*/
>>>>         SI_SUB_KTHREAD_UPDATE   = 0xec00000,    /* update daemon*/
>>>>         SI_SUB_KTHREAD_IDLE     = 0xee00000,    /* idle procs*/
>>>> -       SI_SUB_SMP              = 0xf000000,    /* start the APs*/
>>>>         SI_SUB_RACCTD           = 0xf100000,    /* start racctd*/
>>>>         SI_SUB_LAST             = 0xfffffff     /* final initialization
>>>> */
>>>>   };
>>
>> Did you inspected all reordered sysinit routines and ensured that the
>> reordering is safe ?  At very least, SUB_SMP starts event timers,
>> while KTHREAD_IDLE is about configuring some hardware which might
>> be required/not ready for that.
>
>
> Hi,
>
> I did not inspect everything myself yet regarding this change. That's why
> I'm sending this e-mail out. The problem is simply that the total number of
> interrupts appears to be limited by "APIC_NUM_IOINTS" and "NUM_IO_INTS"
> which is per CPU from what I understand. Until SMP is activated the newbus
> code is simply distributing the IRQ vectors on the available IRQs, then when
> SMP is up it is re-shuffling them all.
>
> I was initially thinking that a hack might be possible, like using RF_SHARED
> for the IRQ resource, but then noticed that we were using MSI interrupts,
> which are not allocated in the same manner.
>
> The other issue is that the IRQs should be functional too, so that PXE boot
> can work.
>

I'm also starting to see increasing amounts of wifi hardware that
expects interrupts to be up and working during probe/attach.

(I think i915kms has the same issue too, no?)




-adrian