vm_fault on boot with NVMe/nda

From: Daniel Nebdal <dnebdal_at_gmail.com>
Date: Mon, 15 Oct 2018 16:31:38 +0200
Hi. I have a 12-ALPHA9 / r339331 amd64 system (a HPE ProLiant ML30 G9),
with a Kingston NVMe SSD ("KINGSTON SKC1000480G") on a PCIe card.

By default, it shows up as /dev/nvd0, and this is how I installed the
system. It has a single large UFS2 (with SJ and TRIM support) partition
mounted as /. (There's also a few other partitions on it that should be
irrelevant for this.) This works, but it does sometimes slow down for
minutes at the time with disturbing queue lengths in gstat; on the order of
tens of thousands. As I understand it, this is due to how TRIM operations
take precedence over everything else when using nvd ?

Looking around, I noticed the nda driver for NVMe-through-CAM. To test it,
I added hw.nvme.use_nvd=0 to loader.conf. On one level, this works: The
drive shows up as /dev/nda0 . On the other hand, trying to mount nda0p2 as
/ floods the console with "vm_fault: pager read error, pid 1 (init)", and
never finishes booting.

What is more interesting is that if I boot from the drive, but mount an
alpha9 usb stick as /, I can then mount the nda device just fine, and the
very minimal testing I did (using bin/cat and COPYRIGHT on the NVMe drive)
seems to work.

So - is nda meant to be bootable, or am I a bit over-eager in trying to do
so?
If not, is there anything smart I can do to get better performance out of
nvd?
(Or have I just overlooked something obvious?)

Dmesg from a normal nvd boot here:
https://openbenchmarking.org/system/1810159-RA-SSD30089593/SSD/dmesg

-- 
Daniel Nebdal
Received on Mon Oct 15 2018 - 12:32:17 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:18 UTC