Re: vm_fault on boot with NVMe/nda

At Netflix we have our OCA firmware based on FreeBSD -current (we take a
snapshot every 5 weeks or so). We've been booting thousands of machines off
nda for over a year... It absolutely works and is one of the things that
lets us deliver the content we do...

I have some patches in my queue waiting for the freeze to lift that do much
better trim shaping to the drives. You might also want to turn on
vfs.ffs.dotrimcons=1 which is a new feature that eliminates many of the
BIO_DELETE requests that come down from UFS that are turned into trims. nvd
has no queueing policy at all: it shot-guns all requests to the drive w/o
collapsing or any moderation at all... This isn't so good for most drives
out there today...

Warner

On Mon, Oct 15, 2018 at 8:38 AM Yuri Pankov <yuripv_at_yuripv.net> wrote:

> Daniel Nebdal wrote:
> > Hi. I have a 12-ALPHA9 / r339331 amd64 system (a HPE ProLiant ML30 G9),
> > with a Kingston NVMe SSD ("KINGSTON SKC1000480G") on a PCIe card.
> >
> > By default, it shows up as /dev/nvd0, and this is how I installed the
> > system. It has a single large UFS2 (with SJ and TRIM support) partition
> > mounted as /. (There's also a few other partitions on it that should be
> > irrelevant for this.) This works, but it does sometimes slow down for
> > minutes at the time with disturbing queue lengths in gstat; on the order
> of
> > tens of thousands. As I understand it, this is due to how TRIM operations
> > take precedence over everything else when using nvd ?
> >
> > Looking around, I noticed the nda driver for NVMe-through-CAM. To test
> it,
> > I added hw.nvme.use_nvd=0 to loader.conf. On one level, this works: The
> > drive shows up as /dev/nda0 . On the other hand, trying to mount nda0p2
> as
> > / floods the console with "vm_fault: pager read error, pid 1 (init)", and
> > never finishes booting.
> >
> > What is more interesting is that if I boot from the drive, but mount an
> > alpha9 usb stick as /, I can then mount the nda device just fine, and the
> > very minimal testing I did (using bin/cat and COPYRIGHT on the NVMe
> drive)
> > seems to work.
> >
> > So - is nda meant to be bootable, or am I a bit over-eager in trying to
> do
> > so?
> > If not, is there anything smart I can do to get better performance out of
> > nvd?
> > (Or have I just overlooked something obvious?)
> >
> > Dmesg from a normal nvd boot here:
> > https://openbenchmarking.org/system/1810159-RA-SSD30089593/SSD/dmesg
>
> FWIW, I set hw.nvme.use_nvd=0 in the installer, got 12-ALPHA8 installed
> on nda0, and it's happily booting from it (using ZFS, though), so it's
> certainly meant to be bootable.
>
>