Re: PCI PF memory decode disable when sizing VF BARs

On Tue, May 5, 2015 at 9:17 AM, Eric Badger <eric_badger_at_dell.com> wrote:

> Hi Ryan and -current,
>
> During IOV config, when setting up VF bars, several calls are made to
> 'pci_read_bar' (in sys/dev/pci/pci.c) in order to size VF BARs, which
> causes memory decoding to be turned off temporarily for the PF associated
> with those VFs. I'm finding that this can interfere with an already running
> PF.
>

Ouch.  That's a nasty bug.  Thanks for tracking this down.


> 1. Check the value of the 'reg' arg to 'pci_read_bar' and, if it is
> outside a standard BAR range, don't disable memory decoding. This is
> simple, but feels a little hackish and may have consequences I'm missing.
>
2. Pass some flag/context through such that pci_read_bar knows it is
> configuring VF BARs (we might instead disable VF MSE in this case, if it is
> enabled). It would be necessary to carry this flag/context through several
> function calls before reaching pci_read_bar, which might end up being ugly.
>
3. Rearrange the calls so that VF BARs are sized when the PF is not yet
> running, and that info saved until VFs are created. Probably it would be
> done when the PF BARs are sized for any device supporting IOV, even if that
> device never creates VFs.
>

I don't think that #3 is possible.  Unfortunately the BAR is sized again
when the BAR is allocated so it's difficult so it wouldn't be enough to
just size the BAR during attach.  I would have to reserve the memory space
during attach, but that might reserve physical address space that other
devices need to function.

Actually, that problem will prevent #2 from being easy to implement too.
We'd have to add an extra flag to pci_alloc_multi_resource.  I think that
#1 is the best option.  There's already a precedent for something similar
has it has a special base for the device ROM BAR.

I haven't had a chance to test this yet, but I believe that this patch will
solve the problem:

https://people.freebsd.org/~rstone/patches/iov_bar.diff