Re: PCI PF memory decode disable when sizing VF BARs

On Tuesday, May 05, 2015 07:15:21 PM Ryan Stone wrote:
> On Tue, May 5, 2015 at 9:17 AM, Eric Badger <eric_badger_at_dell.com> wrote:
> 
> > Hi Ryan and -current,
> >
> > During IOV config, when setting up VF bars, several calls are made to
> > 'pci_read_bar' (in sys/dev/pci/pci.c) in order to size VF BARs, which
> > causes memory decoding to be turned off temporarily for the PF associated
> > with those VFs. I'm finding that this can interfere with an already running
> > PF.
> >
> 
> Ouch.  That's a nasty bug.  Thanks for tracking this down.
> 
> 
> > 1. Check the value of the 'reg' arg to 'pci_read_bar' and, if it is
> > outside a standard BAR range, don't disable memory decoding. This is
> > simple, but feels a little hackish and may have consequences I'm missing.
> >
> 2. Pass some flag/context through such that pci_read_bar knows it is
> > configuring VF BARs (we might instead disable VF MSE in this case, if it is
> > enabled). It would be necessary to carry this flag/context through several
> > function calls before reaching pci_read_bar, which might end up being ugly.
> >
> 3. Rearrange the calls so that VF BARs are sized when the PF is not yet
> > running, and that info saved until VFs are created. Probably it would be
> > done when the PF BARs are sized for any device supporting IOV, even if that
> > device never creates VFs.
> >
> 
> I don't think that #3 is possible.  Unfortunately the BAR is sized again
> when the BAR is allocated so it's difficult so it wouldn't be enough to
> just size the BAR during attach.  I would have to reserve the memory space
> during attach, but that might reserve physical address space that other
> devices need to function.
> 
> Actually, that problem will prevent #2 from being easy to implement too.
> We'd have to add an extra flag to pci_alloc_multi_resource.  I think that
> #1 is the best option.  There's already a precedent for something similar
> has it has a special base for the device ROM BAR.
> 
> I haven't had a chance to test this yet, but I believe that this patch will
> solve the problem:
> 
> https://people.freebsd.org/~rstone/patches/iov_bar.diff

There are some devices with BARs in non-standard locations. :(  If there is
a flag to just disable the VF bar decoding, then ideally we should just be
doing that and leaving the global decoding flag alone while sizing the VF
BAR.

-- 
John Baldwin