Re: The future of ZFS in FreeBSD

On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper <yaneurabeya_at_gmail.com> wrote:
>
> Hello Matthew,
>
> I appreciate the long write up, as someone who still uses FreeBSD ZFS on my NAS box and knowing some of the history with ZFS on *Solaris, etc. Something like this was bound to happen with post the Oracle buyout.
>
> > On Dec 18, 2018, at 10:49 PM, Matthew Macy <mmacy_at_freebsd.org> wrote:
> >
> > The sources for FreeBSD's ZFS support are currently taken directly
> > from Illumos with local ifdefs to support the peculiarities of FreeBSD
> > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> > has regularly pulled changes from Illumos and tried to push back any
> > bug fixes and new features done in the context of FreeBSD. In the past
> > few years the vast majority of new development in ZFS has taken place
> > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> > that they will be moving to ZoL
> > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> > that there will be little to no net new development of Illumos. While
> > working through the git history of ZoL I have also discovered that
> > many races and locking bugs have been fixed in ZoL and never made it
> > back to Illumos and thus FreeBSD. This state of affairs has led to a
> > general agreement among the stakeholders that I have spoken to that it
> > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> > has graciously encouraged me to add FreeBSD support directly to ZoL
> > https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> > shared code base.
> >
> > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> > Before it can be committed some additional functionality needs to be
> > added to the FreeBSD opencrypto framework. These can be found at
> > https://reviews.freebsd.org/D18520
> >
> > This port will provide FreeBSD users with multi modifier protection,
> > project quotas, encrypted datasets, allocation classes, vectorized
> > raidz, vectorized checksums, and various command line improvements.
> >
> > Before ZoF can be merged back in to ZoL several steps need to be taken:
> > - Integrate FreeBSD support into ZoL CI
> > - Have most of the ZFS test suite passing
> > - Complete additional QA testing at iX
>
> Can you please describe the testing process that will be employed to verify the sanity of the ZoL on FreeBSD port? Should other large companies who use ZFS on FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as a whole) collaborate to better suss out issues with the ZoL port?

The ZFS test suite itself provides ~80% coverage
https://codecov.io/gh/zfsonlinux/zfs/branch/master - FreeBSD currently
lacks equivalent gcov support, but presumably it would provide
comparable coverage here. Andrew Turner has some form of kernel gcov
support that he uses with syzkaller. However, I believe that it isn't
sufficient for this purpose. ZoL requires that coverage only increase
over time. If a change would decrease total coverage new tests have to
be included with the change. There is a command called ztest which
runs in user space that quickly covers a large percentage of the code
base where it doesn't integrate directly with geom or VFS (disk access
is emulated with files, kernel threads are emulated with pthreads,
etc). ztest has been broken in HEAD for the past 2 years (it triggers
an assertion failure), whereas it is not broken in ZoF. Joe Maloney is
the head of QA at iX and can likely give you more detail on their
approach to testing. This e-mail was an implicit request to Panzura to
do whatever testing that they can.

>
> > We at iX Systems need to port ZoL's EC2 CI scripts to work with
> > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> > Being integrated in to their CI will mean that, among other things,
> > most integration issues will be caught before a PR is merged upstream
> > rather than many months later when it is MFVed into FreeBSD. I’m
> > hoping to submit the PR to ZoL some time in January.
>
> Could you please better describe this, in particular provide pointers to any scripts that might be executed with the ZoL port?

The ZFS test suite is in ZoL under tests, it makes some Linux specific
assumptions about paths to binaries like ksh etc and it appears to
require some additional configuration options. I started fixing things
before I decided to punt and ask that a porter at iX fix it so that I
could focus on areas that make more sense for me to spend my time on.
If it doesn't get handled I will of course go back to it. The CI
scripts are not something I've looked at. My intention is again that
the people who would be responsible for keeping them up at iX would
close the loop. It's close to Christmas time right now and folks are
inevitably preoccupied with family matters. I sent this mail in
advance of resolving these details to give any outside stakeholders a
heads up as far out as possible. "Shadow" stakeholders coming out of
the wood work to cry foul only after disruptive changes have been made
- in spite of advance notice having been given on the mailing list(s)
- has been an issue in the past. In this instance I intend to keep
posting updates to the list so that no one can claim that sufficient
effort was not made.


> > This port will make it easy for end users on a range of releases to
> > run the latest version of ZFS. Nonetheless, transitioning away from an
> > Illumos based ZFS is not likely to be entirely seamless. The
> > stakeholders I’ve spoken to all agree that this is the best path
> > forward but some degree of effort needs to be made to accommodate
> > downstream consumers. The current plan is to import ZoF and unhook the
> > older Illumos based sources from the build on April 15th or two months
> > after iX systems QA deems ZoF stable - which ever comes later. The
> > Illumos based sources will be removed some time later - but well
> > before 13. This will give users a 3 month period during which both the
> > port and legacy Illumos based ZFS will be available to users. Pools
> > should interoperate between ZoF and legacy provided the user does not
> > enable any features available only in ZoF. We will try to accommodate
> > any downstream consumers in the event that they need that date pushed
> > back. We ask that any downstream consumers who are particularly
> > sensitive to changes start testing the port when it is formally
> > announced and report back any issues they have. I will do my best to
> > ensure that this message is communicated to all those who it may
> > concern. However, I can’t ensure that everyone reads these lists. That
> > is the responsibility of -CURRENT users.
>
>
> As a thought: is there a way for the ZoL port to run in parallel with the non-ZoL port, kind of like in an audit/read-only/verify-only mode?

Only one zfs kernel module can be loaded at a time. The command line
utilities can't coexist due to libzfs differences. The closest one
could come to this would be to switch between boot environments with
the different kmods and utilities and run zdb from one on the pools
that had been managed by the other.

-M