Re: Enabling NUMA in BIOS stop booting FreeBSD

On Thu, Dec 15, 2016 at 04:16:24PM +0300, Slawa Olhovchenkov wrote:
> On Thu, Dec 15, 2016 at 02:33:30PM +0200, Konstantin Belousov wrote:
> 
> > On Thu, Dec 15, 2016 at 01:51:18PM +0300, Slawa Olhovchenkov wrote:
> > > On Wed, Dec 14, 2016 at 09:03:49PM +0200, Konstantin Belousov wrote:
> > > 
> > > > So my opinion did not changed, this sounds like firmware problem.
> > > > I do not see how can I drill into it more.
> > > 
> > > I am don't know how it related. msgbufp mapped different with and w/o
> > > memory test:
> > > 
> > > w/o memory test, hang:
> > > msgbufp=0xfffff8207ff00000 pa_indx=7 phys_avail[pa_indx]=207ff00000
> > > 
> > > w/ memory test, boot:
> > > msgbufp=0xfffff8203ff00000 pa_indx=7 phys_avail[pa_indx]=203ff00000
> > Interesting.
> > 
> > Can you show me
> > - the output of the smap command from the loader (yes, I know it was already
> >   shown, I want it in the same mail as the data below for convenience);
> 
> OK set hw.memtest.tests=1
> OK smap
> SMAP type=01 base=0000000000000000 len=0000000000099c00 attr=01
> SMAP type=02 base=0000000000099c00 len=0000000000006400 attr=01
> SMAP type=02 base=00000000000e0000 len=0000000000020000 attr=01
> SMAP type=01 base=0000000000100000 len=000000007906b000 attr=01
> SMAP type=02 base=000000007916b000 len=0000000000936000 attr=01
> SMAP type=04 base=0000000079aa1000 len=0000000000509000 attr=01
> SMAP type=02 base=0000000079faa000 len=0000000002056000 attr=01
> SMAP type=01 base=0000000100000000 len=0000001f80000000 attr=01
> SMAP type=02 base=000000007c000000 len=0000000014000000 attr=01
> SMAP type=02 base=00000000fed1c000 len=0000000000029000 attr=01
> SMAP type=02 base=00000000ff000000 len=0000000001000000 attr=01
> 
> > - the output of sysctl machdep.smap after the succesfull boot with the
> >   memtest enabled.
> 
> machdep.smap:
> SMAP type=01, xattr=01, base=0000000000000000, len=0000000000099c00
> SMAP type=02, xattr=01, base=0000000000099c00, len=0000000000006400
> SMAP type=02, xattr=01, base=00000000000e0000, len=0000000000020000
> SMAP type=01, xattr=01, base=0000000000100000, len=000000007906b000
> SMAP type=02, xattr=01, base=000000007916b000, len=0000000000936000
> SMAP type=04, xattr=01, base=0000000079aa1000, len=0000000000509000
> SMAP type=02, xattr=01, base=0000000079faa000, len=0000000002056000
> SMAP type=01, xattr=01, base=0000000100000000, len=0000001f80000000
> SMAP type=02, xattr=01, base=000000007c000000, len=0000000014000000
> SMAP type=02, xattr=01, base=00000000fed1c000, len=0000000000029000
> SMAP type=02, xattr=01, base=00000000ff000000, len=0000000001000000
> 
> > Possibly, the dmesg of the boot (with late_console=0) with this and only
> > this patch applied against stock HEAD.  This might be long.
> 
> Do you need all (262144?) lines?
> 
> Testing system
> memory........................................................................................................................pb 0x2040000000
> pb 0x2040001000
> pb 0x2040002000
> pb 0x2040003000
> pb 0x2040004000
> pb 0x2040005000
> pb 0x2040006000
> [...]
> pb 0x207ffff000
> 
> > diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c
> > index 682307f5fe4..072c8d76acf 100644
> > --- a/sys/amd64/amd64/machdep.c
> > +++ b/sys/amd64/amd64/machdep.c
> > _at__at_ -1400,6 +1400,7 _at__at_ getmemsize(caddr_t kmdp, u_int64_t first)
> >  			 */
> >  			*(int *)ptr = tmp;
> >  
> > +if (page_bad) printf("pb 0x%lx\n", pa);
> >  skip_memtest:
> >  			/*
> >  			 * Adjust array of valid/good pages.
> 
> PS: memtest86 hung at test 128-130G (server have 128G installed).
Well, the physical memory is 128G, but it is not mapped contiguously into
the address space accessible to the processors.  E.g. in the SMAPs you
posted above, there are several holes (type 2) used for PCIe config
window, PCI BARs, APICs, and other i/o register pages.  Intel chipsets
allow to remap the RAM hidden by the io pages, which is probably not
done correctly by BIOS.

The SMAP clearly reports segment 0x100000000-0x2080000000 as populated
by RAM, this is 4G-130G.  Very primitive memory test in kernel does
not like all pages starting at 129G.  Possibly important detail is that
kernel memory test only touches first 4 bytes on each page.  So if BIOS
erronously mapped any io registers into that range, memory test might
luckily avoid touching anything critical, but still noting that the
page does not behave as RAM.

Update BIOS, and if the issue persists, contact supermicro. This
interesting detail adds even more evidence that BIOS is problematic.