Re: Enabling NUMA in BIOS stop booting FreeBSD

From: Slawa Olhovchenkov <slw_at_zxy.spb.ru>
Date: Mon, 12 Dec 2016 00:15:53 +0300
On Sun, Dec 11, 2016 at 11:47:09PM +0300, Slawa Olhovchenkov wrote:

> On Sun, Dec 11, 2016 at 10:06:54PM +0200, Konstantin Belousov wrote:
> 
> > On Sun, Dec 11, 2016 at 10:45:59PM +0300, Slawa Olhovchenkov wrote:
> > > On Sun, Dec 11, 2016 at 09:26:56PM +0200, Konstantin Belousov wrote:
> > > 
> > > > On Sun, Dec 11, 2016 at 10:16:26PM +0300, Slawa Olhovchenkov wrote:
> > > > > On Sun, Dec 11, 2016 at 09:21:11PM +0300, Slawa Olhovchenkov wrote:
> > > > > 
> > > > > > On Sat, Nov 26, 2016 at 05:57:47PM +0200, Konstantin Belousov wrote:
> > > > > > 
> > > > > > > On Sat, Nov 26, 2016 at 12:21:24PM +0300, Slawa Olhovchenkov wrote:
> > > > > > > > I am try to enable NUMA in bios and can't boot FreeBSD.
> > > > > > > > Boot stoped after next messages:
> > > > > > > > 
> > > > > > > > ===
> > > > > > > > Booting...
> > > > > > > > KDB: debugger backends: ddb
> > > > > > > > KDB: current backend: ddb
> > > > > > > So at least the hammer_time() has a chance to initialize the console.
> > > > > > > Do you have serial console ?  Set the loader tunable debug.late_console
> > > > > > > to 1 and see if any NMI reaction appear.
> > > > > > > 
> > > > > > > > ===
> > > > > > > > 
> > > > > > > > This is verbose boot.
> > > > > > > > No reaction to ~^B, NMI.
> > > > > > > > 
> > > > > > > > Same for head and 10.3-RELEASE.
> > > > > > > > 
> > > > > > > > Hardware is Supermicro X10DRi, Dual E5-2650v4, 256GB RAM.
> > > > > > > Is there a BIOS option for 'on-chip cluster' or 'HPC computing' ?
> > > > > > > What if you try to frob it ?
> > > > > > > 
> > > > > > > > 
> > > > > > > > On slight different hardware
> > > > > > > > (Supermicro X10DRi w/ old BIOS, Dual E5-2640v3, 128GB RAM)
> > > > > > > > 10.3 boot ok w/ BIOS NUMA enabled.
> > > > > > > 
> > > > > > > I think the only way to debug this is to add printf() lines to hammer_time()
> > > > > > > to see where does it break.  Note that amd64_kdb_init() call succeeded,
> > > > > > > so you can start bisect the code from there.
> > > > > > > 
> > > > > > 
> > > > > > Hang in next two lines:
> > > > > > 
> > > > > >         msgbufinit(msgbufp, msgbufsize);
> > > > > > 	fpuinit();
> > > > 
> > > > Can you show the verbose dmesg up to the failure point ?
> > > > In particular, the SMAP lines should be relevant.
> > > 
> > > KDB: debugger backends: ddb
> > > KDB: current backend: ddb
> > > exit from kdb_init
> > > KDB: enter: Boot flags requested debugger
> > > [ thread pid 0 tid 0 ]
> > > Stopped at      0xffffffff805361eb = kdb_enter+0x3b:    movq
> > > $0,0xffffffff80dcef20 = kdb_why
> > > 
> > > No SMAP print, boot_verbose enabled.
> > The log above shows that you used boot -d. What are the pristine boot
> > messages, with debug.late_console set to 0, of course ?
> 
> This is stable/11, no debug.late_console.

Booting HEAD:

panic: pmap_mapdev_attr: too many preinit mappings
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff80535197 at ??+0
#1 0xffffffff804eb0f2 at ??+0
#2 0xffffffff804eaf63 at ??+0
#3 0xffffffff807b5995 at ??+0
#4 0xffffffff808479ca at ??+0
#5 0xffffffff804079ea at ??+0
#6 0xffffffff8040bb44 at ??+0
#7 0xffffffff8047e178 at ??+0
#8 0xffffffff807a47c3 at ??+0
#9 0xffffffff8028f0a4 at ??+0
Uptime: 1s


> With ANSI ESC, captured from SOL:
> 
> ESC[01;00HType '?' for a /boot/kernel.VSTREAM/opensolaris.ko size 0xcb10 at 0x13d3000
> ESC[01;00HOK smap                                                                         /boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x13e0000
> ESC[02;00HSMAP type=02 base=000000000009ESC[01;00HSMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01                 ESC[02;00HSMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01                 ESC[03;00Hcan't find 'if_ixgbe'
> ESC[01;00H/boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x144a000
> ESC[01;00HSMAP type=02 base=00000000000e/boot/kernel.VSTREAM/ukbd.ko size 0xe280 at 0x1460000
> loading required moduleESC[01;00H 'usb'^MSMAP type=01 base=000000007916b000 len=0000000000936000 attr
> =01                 ESC[02;00H/boot/kernel.VSTREAM/usb.ko size 0x45d40 at 0x146f000
> ESC[01;00HSMAP type=04 base=0000000079aa1000 len=000000/boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x14b5000
> ESC[01;00HSMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01                 ESC[02;00HSMAP type=01 ba/boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x14c0000
> ESC[01;00HSMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01                 /boot/kernel.VSTREAM/sfxge.ko ESC[02;00HSMAP type=02 base=000000007c000000 len=0000000014000000 attr=01                 size 0x1a8ee0 at 0x14c3000
> ESC[03;00HSMAP type=02 base=00000000ff000000 len=0000000001000000 attr/boot/kernel.VSTREAM/uhci.ko size 0xd448 at 0x166c000
> ESC[01;00HSMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01                 /boot/kernel.VSTREAM/ohci.ko size 0xc900 at 0x167a000
> ESC[02;00HOK memmap                                                                       ESC[03;00Hmemmap not foun/boot/kernel.VSTREAM/ehci.ko size 0xfb60 at 0x1687000
> ESC[01;00HOK memmap                                                   /boot/kernel.VSTREAM/xhci.ko size 0x11010 at 0x1697000
> ESC[01;00Hmemmap not found              /boot/kernel.VSTREAM/if_ix.ko size 0x51358 at 0x16a9000^MESC[01;00HOK boot                                      
>                               ESC[01;00HOK boot                       /boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x16fb000
> Booting...
> ESC[01;00H8+0x8+0xe9bdc]                                                                  KDB: debugger backends: ddb
> KDB: current backend: ddb
> exit from kdb_init
> KDB: enter: Boot flags requested debugger
> [ thread pid 0 tid 0 ]
> 
> > Hm, might be also show the output of the 'smap' and 'memmap' output from
> > the loader.  If any of them worked, could be useful to see the same output
> > with the NUMA option disabled as well.
> 
> NUMA disabled:
> OK smap
> SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
> SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
> SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
> SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
> SMAP type=02 base=000000007916b000 len=000000000093a000 attr=01
> SMAP type=04 base=0000000079aa5000 len=0000000000505000 attr=01
> SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
> SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
> SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
> SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
> SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
> OK memmap
> memmap not found
> 
> NUMA enabled:
> OK smap
> SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
> SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
> SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
> SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
> SMAP type=02 base=000000007916b000 len=0000000000936000 attr=01
> SMAP type=04 base=0000000079aa1000 len=0000000000509000 attr=01
> SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
> SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
> SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
> SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
> SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
> OK memmap
> memmap not found
Received on Sun Dec 11 2016 - 20:15:55 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:09 UTC