Re: accessing a PCIe register from userspace through kmem or other ways ?

On Tuesday, April 05, 2016 07:21:29 PM Konstantin Belousov wrote:
> On Tue, Apr 05, 2016 at 08:55:54AM -0700, John Baldwin wrote:
> > Mostly I do not have experience with MGTDEVICE, though I was planning to
> > look at it as a way to implement this.  Two things though: 1) there may
> > not be a cdev to associate with, and 2) I know of at least one device driver
> > that would use this in addition to using this for a general "map this BAR"
> > ioctl on /dev/pci.
> So /dev/pci is the natural cdev to place the functionality.
> An ioctl on /dev/pci may mmap BAR and return the base address.
> 
> > There are other cases in the past where I used OBJT_SG
> > but would have preferred to use a variant that used managed pages so that
> > I could invidate any existing mappings.  In particular what I want to do
> > is invalidate an object so that any future uses fail.
> > 
> > Alternatively, it might be nice to hook a destructor call into a VM object
> > so that I could know when the object is no longer in use (knowing that all
> > its mappings have been destroyed).  When using OBJT_SG objects as aliases
> > for other things (memory allocated via contigmalloc or bus_dma or for
> > resources like PCI BARs), I could keep a reference count on the original
> > "thing" that I increment when creating an OBJT_SG object to return from
> > something like d_mmap_single() or the /dev/pci ioctl and drop the reference
> > count in the destructor hook for that object.
> This is in essence how GEM objects + MGTDEVICE mappings work for i915.
> 
> The only bottleneck in the API arrangement is that d_mmap_single() only
> gets the offset as the identifying data to construct the mapping.
> For /dev/pci, the offset parameter would need to encode d:b:s:f and BAR
> index.

For the ioctl I planned to either 1) call vm_mmap_object() or the like directly
and return the virtual address to the user, or 2) return the mmap offset to use
from the ioctl that the user would then supply to mmap() and d_mmap_single would
use to find the object created by the ioctl.  1) is probably simpler and is more
what I was leaning towards.  Still, I want to be able to handle invalidations
either by pinning the BAR while the object is mapped, or being able to invalidate
the object.  Given that you can eject a hotplug PCI device, I think explicit
invalidation is the better route in this case.  I would create a VM object for
each BAR on the first mmap request and save a reference to it in the PCI bus
ivars.  If the BAR is ever cleared I would be able to find the object and
invalidate it ensuring any programs that then tried to access it would get a
page fault instead of accessing some other random thing.

-- 
John Baldwin