Re: accessing a PCIe register from userspace through kmem or other ways ?

On Tuesday, April 05, 2016 09:14:31 AM Konstantin Belousov wrote:
> On Mon, Apr 04, 2016 at 03:45:07PM -0700, John Baldwin wrote:
> > However, another question is how to deal with systems that do bus address
> > translation (like the arm64 ThunderX boxes) where the values in the PCI
> > BAR are not CPU physical addresses.  To do this properly we may need some
> > sort of bus method akin to my bus_map_resource() WIP but one that instead
> > returns a suitable 'struct sglist' for a given 'struct resource *' that
> > can be used with OBJT_SG to build a VM object to use for the mapping.
> Is there any documentation on the ThunderX PCI access mechanisms ?

The Host-PCI (or really some other bus -> PCI) bridges apply a fixed offset
to PCI addresses, so if your BAR has address X written to it, the CPU has
to use an address of X + Y (where Y is the PCI window base address) to access
it.  The bridge device decodes the access to address X + Y and generates a
PCI transaction to address X.  It's not clear to me how DMA works for these
devices (e.g. if all DMA requests from PCI devices have to use an IOMMU built
into the bridge).

> > (At some point I do think I would like a variant of OBJT_SG that used
> > managed pages so that mappings could be revoked when whatever is backing
> > the sglist is disabled (e.g. the device is ejected or the BAR is
> > relocated, etc.).)
> Why cannot you use MGTDEVICE pager already ? Driver would need to
> maintain its private list of pages, and from this PoV, a convenience
> wrapper around MGTDEVICE which unifies operations on sg lists could be
> useful. Still, it could be that the wrapper appear to be single-purpose.

Mostly I do not have experience with MGTDEVICE, though I was planning to
look at it as a way to implement this.  Two things though: 1) there may
not be a cdev to associate with, and 2) I know of at least one device driver
that would use this in addition to using this for a general "map this BAR"
ioctl on /dev/pci.  There are other cases in the past where I used OBJT_SG
but would have preferred to use a variant that used managed pages so that
I could invidate any existing mappings.  In particular what I want to do
is invalidate an object so that any future uses fail.

Alternatively, it might be nice to hook a destructor call into a VM object
so that I could know when the object is no longer in use (knowing that all
its mappings have been destroyed).  When using OBJT_SG objects as aliases
for other things (memory allocated via contigmalloc or bus_dma or for
resources like PCI BARs), I could keep a reference count on the original
"thing" that I increment when creating an OBJT_SG object to return from
something like d_mmap_single() or the /dev/pci ioctl and drop the reference
count in the destructor hook for that object.

-- 
John Baldwin