Re: vm_fault on boot with NVMe/nda

Daniel Nebdal wrote:
> Hi. I have a 12-ALPHA9 / r339331 amd64 system (a HPE ProLiant ML30 G9),
> with a Kingston NVMe SSD ("KINGSTON SKC1000480G") on a PCIe card.
> 
> By default, it shows up as /dev/nvd0, and this is how I installed the
> system. It has a single large UFS2 (with SJ and TRIM support) partition
> mounted as /. (There's also a few other partitions on it that should be
> irrelevant for this.) This works, but it does sometimes slow down for
> minutes at the time with disturbing queue lengths in gstat; on the order of
> tens of thousands. As I understand it, this is due to how TRIM operations
> take precedence over everything else when using nvd ?
> 
> Looking around, I noticed the nda driver for NVMe-through-CAM. To test it,
> I added hw.nvme.use_nvd=0 to loader.conf. On one level, this works: The
> drive shows up as /dev/nda0 . On the other hand, trying to mount nda0p2 as
> / floods the console with "vm_fault: pager read error, pid 1 (init)", and
> never finishes booting.
> 
> What is more interesting is that if I boot from the drive, but mount an
> alpha9 usb stick as /, I can then mount the nda device just fine, and the
> very minimal testing I did (using bin/cat and COPYRIGHT on the NVMe drive)
> seems to work.
> 
> So - is nda meant to be bootable, or am I a bit over-eager in trying to do
> so?
> If not, is there anything smart I can do to get better performance out of
> nvd?
> (Or have I just overlooked something obvious?)
> 
> Dmesg from a normal nvd boot here:
> https://openbenchmarking.org/system/1810159-RA-SSD30089593/SSD/dmesg

FWIW, I set hw.nvme.use_nvd=0 in the installer, got 12-ALPHA8 installed
on nda0, and it's happily booting from it (using ZFS, though), so it's
certainly meant to be bootable.