Re: Enabling NUMA in BIOS stop booting FreeBSD

From: Slawa Olhovchenkov <slw_at_zxy.spb.ru>
Date: Sun, 11 Dec 2016 23:47:09 +0300
On Sun, Dec 11, 2016 at 10:06:54PM +0200, Konstantin Belousov wrote:

> On Sun, Dec 11, 2016 at 10:45:59PM +0300, Slawa Olhovchenkov wrote:
> > On Sun, Dec 11, 2016 at 09:26:56PM +0200, Konstantin Belousov wrote:
> > 
> > > On Sun, Dec 11, 2016 at 10:16:26PM +0300, Slawa Olhovchenkov wrote:
> > > > On Sun, Dec 11, 2016 at 09:21:11PM +0300, Slawa Olhovchenkov wrote:
> > > > 
> > > > > On Sat, Nov 26, 2016 at 05:57:47PM +0200, Konstantin Belousov wrote:
> > > > > 
> > > > > > On Sat, Nov 26, 2016 at 12:21:24PM +0300, Slawa Olhovchenkov wrote:
> > > > > > > I am try to enable NUMA in bios and can't boot FreeBSD.
> > > > > > > Boot stoped after next messages:
> > > > > > > 
> > > > > > > ===
> > > > > > > Booting...
> > > > > > > KDB: debugger backends: ddb
> > > > > > > KDB: current backend: ddb
> > > > > > So at least the hammer_time() has a chance to initialize the console.
> > > > > > Do you have serial console ?  Set the loader tunable debug.late_console
> > > > > > to 1 and see if any NMI reaction appear.
> > > > > > 
> > > > > > > ===
> > > > > > > 
> > > > > > > This is verbose boot.
> > > > > > > No reaction to ~^B, NMI.
> > > > > > > 
> > > > > > > Same for head and 10.3-RELEASE.
> > > > > > > 
> > > > > > > Hardware is Supermicro X10DRi, Dual E5-2650v4, 256GB RAM.
> > > > > > Is there a BIOS option for 'on-chip cluster' or 'HPC computing' ?
> > > > > > What if you try to frob it ?
> > > > > > 
> > > > > > > 
> > > > > > > On slight different hardware
> > > > > > > (Supermicro X10DRi w/ old BIOS, Dual E5-2640v3, 128GB RAM)
> > > > > > > 10.3 boot ok w/ BIOS NUMA enabled.
> > > > > > 
> > > > > > I think the only way to debug this is to add printf() lines to hammer_time()
> > > > > > to see where does it break.  Note that amd64_kdb_init() call succeeded,
> > > > > > so you can start bisect the code from there.
> > > > > > 
> > > > > 
> > > > > Hang in next two lines:
> > > > > 
> > > > >         msgbufinit(msgbufp, msgbufsize);
> > > > > 	fpuinit();
> > > 
> > > Can you show the verbose dmesg up to the failure point ?
> > > In particular, the SMAP lines should be relevant.
> > 
> > KDB: debugger backends: ddb
> > KDB: current backend: ddb
> > exit from kdb_init
> > KDB: enter: Boot flags requested debugger
> > [ thread pid 0 tid 0 ]
> > Stopped at      0xffffffff805361eb = kdb_enter+0x3b:    movq
> > $0,0xffffffff80dcef20 = kdb_why
> > 
> > No SMAP print, boot_verbose enabled.
> The log above shows that you used boot -d. What are the pristine boot
> messages, with debug.late_console set to 0, of course ?

This is stable/11, no debug.late_console.
With ANSI ESC, captured from SOL:

ESC[01;00HType '?' for a /boot/kernel.VSTREAM/opensolaris.ko size 0xcb10 at 0x13d3000
ESC[01;00HOK smap                                                                         /boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x13e0000
ESC[02;00HSMAP type=02 base=000000000009ESC[01;00HSMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01                 ESC[02;00HSMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01                 ESC[03;00Hcan't find 'if_ixgbe'
ESC[01;00H/boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x144a000
ESC[01;00HSMAP type=02 base=00000000000e/boot/kernel.VSTREAM/ukbd.ko size 0xe280 at 0x1460000
loading required moduleESC[01;00H 'usb'^MSMAP type=01 base=000000007916b000 len=0000000000936000 attr
=01                 ESC[02;00H/boot/kernel.VSTREAM/usb.ko size 0x45d40 at 0x146f000
ESC[01;00HSMAP type=04 base=0000000079aa1000 len=000000/boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x14b5000
ESC[01;00HSMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01                 ESC[02;00HSMAP type=01 ba/boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x14c0000
ESC[01;00HSMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01                 /boot/kernel.VSTREAM/sfxge.ko ESC[02;00HSMAP type=02 base=000000007c000000 len=0000000014000000 attr=01                 size 0x1a8ee0 at 0x14c3000
ESC[03;00HSMAP type=02 base=00000000ff000000 len=0000000001000000 attr/boot/kernel.VSTREAM/uhci.ko size 0xd448 at 0x166c000
ESC[01;00HSMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01                 /boot/kernel.VSTREAM/ohci.ko size 0xc900 at 0x167a000
ESC[02;00HOK memmap                                                                       ESC[03;00Hmemmap not foun/boot/kernel.VSTREAM/ehci.ko size 0xfb60 at 0x1687000
ESC[01;00HOK memmap                                                   /boot/kernel.VSTREAM/xhci.ko size 0x11010 at 0x1697000
ESC[01;00Hmemmap not found              /boot/kernel.VSTREAM/if_ix.ko size 0x51358 at 0x16a9000^MESC[01;00HOK boot                                      
                              ESC[01;00HOK boot                       /boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x16fb000
Booting...
ESC[01;00H8+0x8+0xe9bdc]                                                                  KDB: debugger backends: ddb
KDB: current backend: ddb
exit from kdb_init
KDB: enter: Boot flags requested debugger
[ thread pid 0 tid 0 ]

> Hm, might be also show the output of the 'smap' and 'memmap' output from
> the loader.  If any of them worked, could be useful to see the same output
> with the NUMA option disabled as well.

NUMA disabled:
OK smap
SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
SMAP type=02 base=000000007916b000 len=000000000093a000 attr=01
SMAP type=04 base=0000000079aa5000 len=0000000000505000 attr=01
SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
OK memmap
memmap not found

NUMA enabled:
OK smap
SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
SMAP type=02 base=000000007916b000 len=0000000000936000 attr=01
SMAP type=04 base=0000000079aa1000 len=0000000000509000 attr=01
SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
OK memmap
memmap not found
Received on Sun Dec 11 2016 - 19:47:14 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:09 UTC