Re: IOMMU for GPUs

On Sun, Aug 05, 2018 at 10:34:56AM +0100, Johannes Lundberg wrote:
> Hi
> 
> First I have to say I don't know much when it comes to virtual GPUs and
> IOMMU. I'm trying to figure out what we have and what is missing in regards
> to sharing the GPU to virtual guests on Intel and AMD and things like the
> amdkfd driver (for Radeon open compute).
> 
> Looking at the state of IOMMU in FreeBSD there seem to be (what I guess is)
> a general driver for Intel at /usr/src/sys/x86/iommu/.
This is indeed a generic driver, which also provides busdma
implementation using DMAR. Its main use right now is to debug other
drivers. It is also helpful if some device has very restrictive DMA
alignment, contiguous or location requirements which cannot be satisfied
by VM subsystem easily, esp. under the load. In this case its behaviour
might be better than the usual bounce busdma.

Also the Intel driver knows about significant set of the DMAR erratas in
the existing chipsets, although I did not updated the list recently.

There is no similar driver for AMD IOMMU.

> 
> Then there's the support for AMD's IOMMU in bhyve at
> /usr/src/sys/amd64/vmm/io/iommu.c.
Bhyve has both Intel DMAR and AMD IOMMU simple drivers, only providing
for the PCI pass-through. Intention is to switch to x86/iommu DMAR
driver eventually, at least on Intel. I did not found time to work on
this still.

> 
> Without looking too much into the details my guess is we have an iommu
> driver for Intel that Intel's i915/gvt can use but for AMD there's only the
> specific implementation for bhyve and no general driver that other clients
> can use... Or?
In-tree code in dev/drm2 does not work with GPU IOMMU turned on.  GTT code
for Intel/GEM assumes that physical == GTT address.  I strongly suspect
that dev/drm2/ati does the same for TTM buffers.

> 
> If anyone wants to work on this, it's up for grabs :)
> We'll probably have to add some glue in linuxkpi as well.
As far as I know (would be glad to appear wrong) linuxkpi does not
wrap Linux DMA KPI into our busdma, if such wrapping is ever possible.
More probably, DMA interfaces should be implemented from scratch and
perhaps use existing Intel DMAR driver as one of the substrates.