Re: Enabling NUMA in BIOS stop booting FreeBSD

From: Slawa Olhovchenkov <slw_at_zxy.spb.ru>
Date: Wed, 14 Dec 2016 18:26:27 +0300
On Wed, Dec 14, 2016 at 03:13:36PM +0300, Slawa Olhovchenkov wrote:

> On Wed, Dec 14, 2016 at 01:39:27PM +0200, Konstantin Belousov wrote:
> 
> > In other words, it is almost certainly the hang and not a fault causing
> > hang. This means that the machine is not compliant with the IA32
> > architecture, in particular, the region reported as normal memory by
> > E820 BIOS service does not behave as normal memory.
> > 
> > Since regardless of the option setting, the memory map is same, and
> > bootstrap page table only depend on the memory map, we use the same page
> > table when hanging and when operating correctly. We do not fault or hang
> > when the option is turned off, which together with the improved early
> > fault handling in the patch, makes it almost certain that the problem is
> > in hardware configuration and not in our early setup.
> > 
> > Of course, the most puzzling part is that memory test makes the hang
> > go away, while repeating memory test operation only on the msgbuf region
> > does not. msgbuf is special in that it is located at TOHM (top of high
> > memory). It spans 128KB from below it to the last byte of the last
> > physical segment.
> > 
> > The only ideas I have right now is that there is either a bug in the
> > Caching Agent/Home agent/IMC configuration in BIOS, in which case there
> > is nothing OS can do to mitigate it.  Or it might be that the memory
> > map reported by CMS is wrong (you said that you use legacy boot, right
> > ?).  This is not too surprising if true, because non-EFI boot code path
> > definitely get less and less testing.
> > 
> > For the later case (potential bug in CMS), could you switch to EFI boot
> > mode and see whether the issue magically healths itself ?  You could boot
> > from USB stick in EFI mode without reinstalling for test.
> 
> I can't boot from USB stick -- this is remote DC and IPMI allow only
> CDROM emulation.
> 
> OK, I am boot in UEFI 12.0 snapshot ISO.
> Boot ok.

Sorry. Overload bu work and test wrong combination (NUMA=ON,
interleave=OFF)

snapshot iso don't boot with NUMA=ON interleave=ON

For test hardware setup (NUMA+interleave), what ISO I can try to boot?

PS: memmaps:

NUMA=ON interleave=OFF
OK memmap
                   Type     Physical      Virtual   #Pages Attr
       BootServicesCode 000000000000 000000000000 00000008 UC WC WT WB
     ConventionalMemory 000000008000 000000000000 00000027 UC WC WT WB
       BootServicesData 00000002f000 000000000000 00000011 UC WC WT WB
       BootServicesCode 000000040000 000000000000 00000060 UC WC WT WB
     ConventionalMemory 000000100000 000000000000 000660a3 UC WC WT WB
       BootServicesData 0000661a3000 000000000000 00000080 UC WC WT WB
     ConventionalMemory 000066223000 000000000000 000076b8 UC WC WT WB
             LoaderData 00006d8db000 000000000000 00008000 UC WC WT WB
             LoaderCode 0000758db000 000000000000 00000070 UC WC WT WB
       BootServicesData 00007594b000 000000000000 00003220 UC WC WT WB
     ConventionalMemory 000078b6b000 000000000000 0000028e UC WC WT WB
       BootServicesCode 000078df9000 000000000000 00000372 UC WC WT WB
               Reserved 00007916b000 000000000000 00000817 UC WC WT WB
     ConventionalMemory 000079982000 000000000000 0000011f UC WC WT WB
          ACPIMemoryNVS 000079aa1000 000000000000 00000509 UC WC WT WB
    RuntimeServicesData 000079faa000 000000000000 00001dbd UC WC WT WB
    RuntimeServicesCode 00007bd67000 000000000000 00000061 UC WC WT WB
       BootServicesData 00007bdc8000 000000000000 00000001 UC WC WT WB
    RuntimeServicesData 00007bdc9000 000000000000 00000086 UC WC WT WB
       BootServicesData 00007be4f000 000000000000 000001b1 UC WC WT WB
     ConventionalMemory 000100000000 000000000000 01f80000 UC WC WT WB
               Reserved 00007c000000 000000000000 00004000
         MemoryMappedIO 000080000000 000000000000 00010000 UC
         MemoryMappedIO 0000fed1c000 000000000000 00000029 UC
         MemoryMappedIO 0000ff000000 000000000000 00001000 UC

NUMA=ON interleave=ON
                   Type     Physical      Virtual   #Pages Attr
       BootServicesCode 000000000000 000000000000 00000008 UC WC WT WB
     ConventionalMemory 000000008000 000000000000 00000027 UC WC WT WB
       BootServicesData 00000002f000 000000000000 00000011 UC WC WT WB
       BootServicesCode 000000040000 000000000000 00000060 UC WC WT WB
     ConventionalMemory 000000100000 000000000000 000660a3 UC WC WT WB
       BootServicesData 0000661a3000 000000000000 00000080 UC WC WT WB
     ConventionalMemory 000066223000 000000000000 000076b8 UC WC WT WB
             LoaderData 00006d8db000 000000000000 00008000 UC WC WT WB
             LoaderCode 0000758db000 000000000000 00000070 UC WC WT WB
       BootServicesData 00007594b000 000000000000 00003220 UC WC WT WB
     ConventionalMemory 000078b6b000 000000000000 0000028e UC WC WT WB
       BootServicesCode 000078df9000 000000000000 00000372 UC WC WT WB
               Reserved 00007916b000 000000000000 00000817 UC WC WT WB
     ConventionalMemory 000079982000 000000000000 0000011f UC WC WT WB
          ACPIMemoryNVS 000079aa1000 000000000000 00000509 UC WC WT WB
    RuntimeServicesData 000079faa000 000000000000 00001dbd UC WC WT WB
    RuntimeServicesCode 00007bd67000 000000000000 00000061 UC WC WT WB
       BootServicesData 00007bdc8000 000000000000 00000001 UC WC WT WB
    RuntimeServicesData 00007bdc9000 000000000000 00000086 UC WC WT WB
       BootServicesData 00007be4f000 000000000000 000001b1 UC WC WT WB
     ConventionalMemory 000100000000 000000000000 01f80000 UC WC WT WB
               Reserved 00007c000000 000000000000 00004000
         MemoryMappedIO 000080000000 000000000000 00010000 UC
         MemoryMappedIO 0000fed1c000 000000000000 00000029 UC
         MemoryMappedIO 0000ff000000 000000000000 00001000 UC

NUMA=OFF interleave=ON
                   Type     Physical      Virtual   #Pages Attr
       BootServicesCode 000000000000 000000000000 00000008 UC WC WT WB
     ConventionalMemory 000000008000 000000000000 00000027 UC WC WT WB
       BootServicesData 00000002f000 000000000000 00000011 UC WC WT WB
       BootServicesCode 000000040000 000000000000 00000060 UC WC WT WB
     ConventionalMemory 000000100000 000000000000 000660a3 UC WC WT WB
       BootServicesData 0000661a3000 000000000000 00000080 UC WC WT WB
     ConventionalMemory 000066223000 000000000000 000076b8 UC WC WT WB
             LoaderData 00006d8db000 000000000000 00008000 UC WC WT WB
             LoaderCode 0000758db000 000000000000 00000070 UC WC WT WB
       BootServicesData 00007594b000 000000000000 00003220 UC WC WT WB
     ConventionalMemory 000078b6b000 000000000000 0000028e UC WC WT WB
       BootServicesCode 000078df9000 000000000000 00000372 UC WC WT WB
               Reserved 00007916b000 000000000000 00000817 UC WC WT WB
     ConventionalMemory 000079982000 000000000000 00000123 UC WC WT WB
          ACPIMemoryNVS 000079aa5000 000000000000 00000505 UC WC WT WB
    RuntimeServicesData 000079faa000 000000000000 00001dbd UC WC WT WB
    RuntimeServicesCode 00007bd67000 000000000000 00000061 UC WC WT WB
       BootServicesData 00007bdc8000 000000000000 00000001 UC WC WT WB
    RuntimeServicesData 00007bdc9000 000000000000 00000086 UC WC WT WB
       BootServicesData 00007be4f000 000000000000 000001b1 UC WC WT WB
     ConventionalMemory 000100000000 000000000000 01f80000 UC WC WT WB
               Reserved 00007c000000 000000000000 00004000
         MemoryMappedIO 000080000000 000000000000 00010000 UC
         MemoryMappedIO 0000fed1c000 000000000000 00000029 UC
         MemoryMappedIO 0000ff000000 000000000000 00001000 UC
Received on Wed Dec 14 2016 - 14:26:31 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:09 UTC