Re: 13.0 RC4 might be delayed

On Sun, Mar 28, 2021 at 10:36 PM Gleb Popov <arrowd_at_freebsd.org> wrote:

> On Mon, Mar 29, 2021 at 4:37 AM David G Lawrence via freebsd-current <
> freebsd-current_at_freebsd.org> wrote:
>
> > > > On 27/03/21 06:04, David G Lawrence via freebsd-current wrote:
> > > >>> On Fri, Mar 26, 2021 at 1:01 PM Graham Perrin <
> > grahamperrin_at_gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>>> On 26/03/2021 03:40, The Doctor via freebsd-current wrote:
> > > >>>>> ??? if people are having issues with ports like ???
> > > >>>>
> > > >>>> If I'm not mistaken:
> > > >>>>
> > > >>>> * 13.0-RC3 seems to be troublesome, as a guest machine, with
> > > >>>> emulators/virtualbox-ose 6.1.18 as the host
> > > >>>>
> > > >>>> * no such trouble with 12.0-RELEASE-p5 as a guest.
> > > >>>>
> > > >>>> I hope to refine the bug report this weekend.
> > > >>>>
> > > >>>
> > > >>> Had nothing but frequent guest lockups on 6.1.18 with my Win7
> > system.
> > > >>> That
> > > >>> was right after 6.1.18 was put into ports. Fell back to legacy (v5)
> > and
> > > >>> will try again shortly to see if it's any better.
> > > >>
> > > >> Kevin,
> > > >>
> > > >> ?????? Make sure you have these options in your /etc/sysctl.conf :
> > > >>
> > > >> vfs.aio.max_buf_aio=8192
> > > >> vfs.aio.max_aio_queue_per_proc=65536
> > > >> vfs.aio.max_aio_per_proc=8192
> > > >> vfs.aio.max_aio_queue=65536
> > > >>
> > > >> ?????? ...otherwise the guest I/O will random hang in VirtualBox.
> > This
> > > >> issue was
> > > >> mitigated in a late 5.x VirtualBox by patching to not use AIO, but
> > the
> > > >> issue
> > > >> came back in 6.x when that patch wasn't carried forward.
> > > >
> > > > Sorry I lost that patch. Can you point me to the patch? Maybe it can
> > be
> > > > easily ported.
> > > >
> > >
> > > I found the relevant commit. Please give me some time for testing and
> > > I'll put this patch back in the tree.
> >
> >    If you're going to put that patch back in, then AIO should probably be
> > made an option in the port config, as shutting AIO off by default will
> > have a significant performance impact. Without AIO, all guest IO will
> > be become synchronous.
> >
>
> Are you sure about that? Without AIO, VBox uses a generic POSIX backend,
> which is based on pthread, I think.
>

We should also consider changing the defaults.

vfs.aio.max_buf_aio: this is the maximum number of buffered AIO requests
per process.  Buffered AIO requests are only used when directing AIO to
device nodes, not files, and only for devices that don't support unmapped
I/O.  Most devices do support unmapped I/O, including all GEOM devices.
For devices that do support unmapped I/O, the number of AIO requests per
process is unlimited.  So this knob isn't very important.  However, it is
more important on powerpc and mips, where unmapped I/O isn't always
possible.  16 is probably pretty reasonable for mips.

vfs.aio.max_aio_queue_per_proc: this is the maximum queued aio requests per
process.  This applies to all AIO requests, whether to files or devices.
So it ought to be large.  If your program is too unsophisticated to handle
EAGAIN, then it must be very large.  Otherwise, a few multiples of
max(vfs.aio.max_aio_per_proc, your SSD's queue depth) is probably
sufficient.

vfs.aio.max_aio_per_proc: this is the max number of active aio requests in
the slow path (for I/O to files, or other cases like misaligned I/O to
disks).  Setting this too low won't cause programs to fail, but it could
hurt performance.  Setting it higher than vfs.aio.max_aio_procs probably
won't have any benefit.

vfs.aio.max_aio_queue: like max_aio_per_proc, but global instead of
per-process.  Doesn't need to be more than a few multiples of
max_aio_per_proc.

Finally, I see that emulators/virtualbox-ose's pkg-message advises checking
for the AIO kernel module.  That advice is obsolete.  AIO is nowadays
builtin to the kernel and always enabled.  There is no kernel module any
longer.

Actually, the defaults don't look unreasonable to me, for an amd64 system
with disk, file, or zvol-backed VMs.  Does virtualbox properly handle
EAGAIN as returned by aio_write, aio_read, and lio_listio?  If not, raising
these limits is a poor substitute for fixing virtualbox.  If so, then I'm
really curious.  If anybody could tell me which limit actually solves the
problem, I would like to know.

-Alan