Re: cardbus memory allocation problem

On Monday, May 02, 2011 10:29:56 pm Michael Butler wrote:
> I've stared at this for a (long) while but haven't come to any
> reasonable conclusion as to why it does what it does or how to fix it :-(
> 
> Specifically, the BIOS in this machine doesn't set up a memory window
> for the cardbus controller nor does it properly configure the PCI bridge
> to route to the correct buses. BSD tries but allocates memory from the
> wrong space.
> 
> My question is - how to get PCI-cardbus bridge to allocate memory inside
> the window of the parent PCI-PCI bridge? .. the bus tree looks like ..
> 
> imb_at_toshi:/home/imb> sudo lspci -t
> -[0000:00]-+-00.0
>            +-02.0
>            +-02.1
>            +-1b.0
>            +-1c.0-[02]--
>            +-1c.1-[03-04]--
>            +-1c.2-[05-06]----00.0
>            +-1d.0
>            +-1d.1
>            +-1d.2
>            +-1d.3
>            +-1d.7
>            +-1e.0-[07]--+-06.0
>            |            +-06.1
>            |            +-06.2
>            |            +-06.3
>            |            \-08.0
>            +-1f.0
>            +-1f.2
>            \-1f.3
> 
> I've annotated the verbose dmesg below to highlight the issues ..
> 
> pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pcib4:   domain            0
> pcib4:   secondary bus     7
> pcib4:   subordinate bus   7
>  *** subordinate bus needs to be '9' so as to include both '8' & '9'
>  *** for the PCI-cardbus bridge

I have WIP patches to fix this but they aren't ready yet.
 
> pcib4:   I/O decode        0x4000-0x4fff
> pcib4:   memory decode     0xf0900000-0xf09fffff
>  *** this memory widow is what I expected all children to allocate from
> 
> pcib4:   no prefetched decode
> pcib4:   Subtractively decoded bridge.

It's a subtractive bridge, so the resources do not have to be allocated from 
the window.  That said, I'm committing the last of my patches to HEAD today to 
rework how PCI-PCI bridges handle I/O windows to support growing windows, etc. 
and the new PCI-PCI bridge driver will attempt to grow the memory window to 
allocate a new range before falling back to depending on the subtractive 
decode.

> cbb0: <PCI-CardBus Bridge> at device 6.0 on pci7
> pcib4: cbb0 requested memory range 0x0-0xffffffff: good
>  *** what appears to be a "wildcard" alloc request
> 
> cbb0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xbf670000
>  *** but which isn't constrained to be within the parent bridge's space

Yes, it is a subtractive bridge, so that should be fine.  The problem may be 
that the bf670000 address may not be decoded by the parent Host-PCI bridge.  
There are some tunable hacks you can try to force this address higher, but I 
am also working on other patches (before the bus numbering ones) to query ACPI 
for the list of valid decoded ranges of PCI addresses for Host-PCI bridges and 
to restrict PCI allocations to coming from those ranges.

You can try increasing hw.acpi.host_mem_start or hw.cbb.memory_start loader
tunables.  (There should be sysctl's with the current values I think.)

-- 
John Baldwin