Re: Enabling NUMA in BIOS stop booting FreeBSD

On Sun, Dec 11, 2016 at 10:45:59PM +0300, Slawa Olhovchenkov wrote:
> On Sun, Dec 11, 2016 at 09:26:56PM +0200, Konstantin Belousov wrote:
> 
> > On Sun, Dec 11, 2016 at 10:16:26PM +0300, Slawa Olhovchenkov wrote:
> > > On Sun, Dec 11, 2016 at 09:21:11PM +0300, Slawa Olhovchenkov wrote:
> > > 
> > > > On Sat, Nov 26, 2016 at 05:57:47PM +0200, Konstantin Belousov wrote:
> > > > 
> > > > > On Sat, Nov 26, 2016 at 12:21:24PM +0300, Slawa Olhovchenkov wrote:
> > > > > > I am try to enable NUMA in bios and can't boot FreeBSD.
> > > > > > Boot stoped after next messages:
> > > > > > 
> > > > > > ===
> > > > > > Booting...
> > > > > > KDB: debugger backends: ddb
> > > > > > KDB: current backend: ddb
> > > > > So at least the hammer_time() has a chance to initialize the console.
> > > > > Do you have serial console ?  Set the loader tunable debug.late_console
> > > > > to 1 and see if any NMI reaction appear.
> > > > > 
> > > > > > ===
> > > > > > 
> > > > > > This is verbose boot.
> > > > > > No reaction to ~^B, NMI.
> > > > > > 
> > > > > > Same for head and 10.3-RELEASE.
> > > > > > 
> > > > > > Hardware is Supermicro X10DRi, Dual E5-2650v4, 256GB RAM.
> > > > > Is there a BIOS option for 'on-chip cluster' or 'HPC computing' ?
> > > > > What if you try to frob it ?
> > > > > 
> > > > > > 
> > > > > > On slight different hardware
> > > > > > (Supermicro X10DRi w/ old BIOS, Dual E5-2640v3, 128GB RAM)
> > > > > > 10.3 boot ok w/ BIOS NUMA enabled.
> > > > > 
> > > > > I think the only way to debug this is to add printf() lines to hammer_time()
> > > > > to see where does it break.  Note that amd64_kdb_init() call succeeded,
> > > > > so you can start bisect the code from there.
> > > > > 
> > > > 
> > > > Hang in next two lines:
> > > > 
> > > >         msgbufinit(msgbufp, msgbufsize);
> > > > 	fpuinit();
> > 
> > Can you show the verbose dmesg up to the failure point ?
> > In particular, the SMAP lines should be relevant.
> 
> KDB: debugger backends: ddb
> KDB: current backend: ddb
> exit from kdb_init
> KDB: enter: Boot flags requested debugger
> [ thread pid 0 tid 0 ]
> Stopped at      0xffffffff805361eb = kdb_enter+0x3b:    movq
> $0,0xffffffff80dcef20 = kdb_why
> 
> No SMAP print, boot_verbose enabled.
The log above shows that you used boot -d. What are the pristine boot
messages, with debug.late_console set to 0, of course ?

Hm, might be also show the output of the 'smap' and 'memmap' output from
the loader.  If any of them worked, could be useful to see the same output
with the NUMA option disabled as well.