Re: kernel panic with greater that 8 GB of memory

On Wed, 1 Dec 2004, David O'Brien wrote:

> I've seen S2882 BIOS's that don't say "ccNUMA" and my S2885 K8W Thunder
> certainly doesn't.  It's BIOS knobs and explanation are:
>   Interleaving allows memory accesses to be spread out over BANKS on the
>   same node, or across NODES, decreasing access contention.
>    o "Bank Interleaving"
>    o "Node Interleaving"
> On every Operon BIOS (AMI or Phoenix), the SRAT enabling and Node
> Interleaving are related -- they are mutually exclusive by design as Node
> Interleaving lets the BIOS setup the physical memory topology vs.
> enabling the SRAT which then puts it in the OS's court.
> Enabling the SRAT doesn't need to and shouldn't disable Bank
> Interleaving.

Yes, I do indeed stand corrected. Writing emails late at night while tired
is something I should probably cut down on. The S2882 I see most often is
_not_ a 2.03 BIOS; the 2.03 BIOS refused to even POST with our RAID 
controller. (Predictably, Tyan's response was 'don't run that BIOS then.')
Unfortunately, the S4882 hell is still quite fresh in my mind; the Phoenix
6.0 Ref on those is mutual exclusive. No joke; if SRAT is enabled, actual
Bank Interleave is disabled. (The BIOS seems to consider Node/Bank as 
equal. Not like the S4882 doesn't have enough quirks anyways. :P) Coupled
with the 'DDR 400MHz Memmory Detected' line, you can probably guess my
opinion of Tyan's BIOS guys at this point.

> Nor in the AMI reference BIOS :-(, which is a shame as more information
> might better clear things up.

Indeed; also better note-taking on my part, but those are on a dead 
machine right now. ;(

> Actually the S2882 uses a AMI BIOS, as does all of Tyan's 2P Opteron
> boards, except the S2885 variant Tyan makes for the Fujitsu Celsius V810.
> I like the Fujitsu Celsius V810 Phoenix better, wished it was used on all
> S2885's.

Yeah, I thought they might have used Phoenix because I haven't seen the 
S2882 rebooted in a good long time. I'd take _anything_ over Tyan's BIOS;
it has been nothing but trouble in various ways since day one.

> How do you want to control the "memory hole"?  It is mandatory in order
> to map in memory mapped I/O devices.

On the S4882, you have limited control over the PCI Memory Hole. You can
set it to Auto, or to a specific size. Auto will size it based on enabled
devices in the bios, setting it to a specific value will create a hole of
that size. One of those way-too-fine-tuning things.

> Correct, the "memory hole" must be below the 4GB mark.
> http://www.amd.com/us-en/assets/content_type/DownloadableAssets/RichBrunnerClusterWorldpresFINAL.pdf
> slide 25 is AMD's picture for this.  The "hole" is of course an area
> where memory mapped PCI I/O, APIC, APG GART, etc.. are mapped into the
> memory address space.  Thus RAM cannot be addressed using (mapped into)
> those memory locations.

Making me wish I had a way to view PDFs right now. Looking at the 
presented memory regions:

Physical memory chuck(s):
0x0000000000001000 - 0000000000009bfff, 634880 bytes (155 pages)
0x000000000082d000 - 000000000fbfeffff, 4219219968 bytes (1030083 pages)
0x0000000100000000 - 000000003e1fcffff, 12381388800 bytes (3022800 pages)

1000-9bfff is the 640K region. Then a significant gap between 640K
and the 4023MB region. Followed by another large gap between the 
4023MB region and the 11807MB region. My hex conversion with 
numbers that large, frankly, sucks, but it looks to me as though 
the PCI hole is 512MB (e.g.; all onboard devices enabled,) and 
starting at 4024MB. Which would put the PCI Memory Hole past the
4GB limit. Only explanation for behavior like that, that I can 
think of, is yet another BIOS bug.

-Ketrien I. Saihr-Kenchedra
  "And I'm going back to bed."