Re: kernel panic with greater that 8 GB of memory

On Wed, 1 Dec 2004, David O'Brien wrote:

> There is no 'ccNUMA' setting in the BIOS.  A multi-processor Opteron is a
> NUMA architecture machine regardless of any BIOS settings.  I.E. there is
> no way to disable that a MP Opteron is a NUMA machine.
> The setting of interest is should the BIOS round-robin interleave
> physical addresses across the NUMA nodes[*].  The reference AMI BIOS
> refers to this as "Node Interleaving".  It should be "DISABLED".  Or if
> the BIOS speaks of the SRAT table, it should be "ENABLED".  While FreeBSD
> doesn't use the SRAT table (and cannot until ACPI 3.0 BIOS's); turning on
> the SRAT turns off node interleaving.

David, trying very hard to be nice, the S2882's a 'Special Case.'
Where special should be taken as 'short bus.' VERY short bus.

On the S2882/S2885, and even the S4882, the BIOS specifically says, and I
quote: 'ccNUMA Support.' No joke. The beauty is that ccNUMA is, you got
it, SRAT Table Control, which _disables_ interleave completely. Beautiful,
huh? The only way to enable interleave on the S2882/S2885 is to
specifically turn 'ccNUMA' off; otherwise it's SRAT only, with no 
interleave. And no, it's not mentioned anywhere on the S2882s. Only on 
the S4882, which uses a Phoenix 6.0 reference. (Which Tyan, predictably,
did not remove the reference lines _or_ fix the typos on.) The S2882s
also use a Phoenix, I believe. So in order to enable Interleave, your
only method is that switch.

What got me peering curiously here is the placement of the hole; there 
appears to be a 512MB hole, starting at 4024MB. It certainly is similar
to the behavior of 2882 I've looked at; hole is about 1/3rd through 
memory. On the 2.03 BIOS, still no way to control the PCI hole; that 
feature is reserved for the S4882 apparently, and even then only size.
If the hole _is_ at 4024MB, that would put it past 4096MB and the 4GB
limit; on the S2882 the bulk of the peripherals are on the PCI32 bus, not
a PCI64 bus. Specifically, disk, fxp(4), bge(4), ACPI, and USB.
Presuming I am reading the hole correctly, and bearing in mind that I do
not have access to my copy of the PCI2.3 spec, ISTR that the entirety of 
the PCI memory hole must be within the first 4GB of system memory. (I'd 
appreciate it if someone could sanity check that.) Presuming that it does
need to be within the first 4GB of system memory, I'm wondering if what's
being seen here is not a BIOS bug. These boards have a real bad track 
record with regards to them, so it would not be at all surprising, at 
least to me.

-ksaihr