Re: MPSAFE VFS -- List of upcoming actions

On Fri, Sep 21, 2012 at 1:22 AM, Attilio Rao <attilio_at_freebsd.org> wrote:
> On Wed, Sep 19, 2012 at 3:48 AM, Attilio Rao <attilio_at_freebsd.org> wrote:
>> On Fri, Jul 13, 2012 at 12:18 AM, Attilio Rao <attilio_at_freebsd.org> wrote:
>>> 2012/7/4 Attilio Rao <attilio_at_freebsd.org>:
>>>> 2012/6/29 Attilio Rao <attilio_at_freebsd.org>:
>>>>> As already published several times, according to the following plan:
>>>>> http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS
>>>>>
>>>>
>>>> I still haven't heard from Vivien or Edward, anyway as NTFS is
>>>> basically only used RO these days (also the mount_ntfs code just
>>>> permits RO mounting) I stripped all the uncomplete/bogus write support
>>>> with the following patch:
>>>> http://www.freebsd.org/~attilio/ntfs_remove_write.patch
>>>>
>>>> This is an attempt to make the code smaller and possibly just focus on
>>>> the locking that really matter (as read-only filesystem).
>>>> On some points of the patch I'm a bit less sure as we could easily
>>>> take into account also write for things like vaccess() arguments, and
>>>> make easier to re-add correct write support at some point in the
>>>> future, but still force RO, even if the approach used in the patch is
>>>> more correct IMHO.
>>>> As an added bonus this patch cleans some dirty code in the mount
>>>> operation and fixes a bug as vfs_mountedfrom() is called before real
>>>> mounting is completed and can still fail.
>>>
>>> A quick update on this.
>>> It looks like NTFS won't be completed for this GSoC thus I seriously
>>> need to find an alternative to not loose the NTFS support entirely.
>>>
>>> I tried to look into the NTFS implementation right now and it is
>>> really a poor support. As Peter has also verified, it can deadlock in
>>> no-time, it compeltely violates VFS rules, etc. IMHO it deserves a
>>> complete rewrite if we would still support in-kernel NTFS. I also
>>> tried to look at the NetBSD implementation. Their code is someway
>>> similar to our, but they used very complicated (and very dirty) code
>>> to do the locking. Even if I don't know well enough NetBSD VFS, I have
>>> the impression not all the races are correctly handled. Definitively,
>>> not something I would like to port.
>>>
>>> Considering all that the only viable option would be meaning an
>>> userland filesystem implementation. My preferred choice would be to
>>> import PUFFS and librefuse on top of it but honestly it requires a lot
>>> of time to be completed, time which I don't currently have as in 2
>>> months Giant must be gone by the VFS.
>>>
>>> I then decided to switch to gnn's rewamp of FUSE patches. You can find
>>> his initial e-mail here:
>>> http://lists.freebsd.org/pipermail/freebsd-fs/2012-March/013876.html
>>>
>>> I've precisely got the second version of George's patch and created
>>> this dolphin branch:
>>> svn://svn.freebsd.org/base/projects/fuse
>>>
>>> I'm fixing low hanging fruit for the moment (see r238411 for example)
>>> and I still have to make a throughful review.
>>> However my idea is to commit the support once:
>>> - ntfs-3g is well stress-tested and proves to be bug-free
>>> - there is no major/big technical issue pending after the reviews
>>
>> In the last weeks Peter, Florian, Gustau and I have been working in
>> stabilizing fuse support. In the specific, Peter has worked hard on
>> producing several utilities to nit stress-test fuse and in particular
>> ntfs, Florian has improved fuse related ports (as explained later) and
>> Gustau has done sparse testing. I feel moderately satisfied by the
>> level of stability of fuse now to propose to wider usage, in
>> particular given the huge amount of complaints I'm hearing around
>> about occasional fuse users.
>>
>> The final target of the project is to completely import into base the
>> content of fusefs-kmod starting from earlier posted patches by George.
>> So far, we took care only of importing in the fuse branch the kernel
>> part, so that fusefs-kmod userland part is still needed to be
>> installed from ports, but I was studying the mount_fusefs licensing
>> before to process with the import for the userland bits of it.
>>
>> The fixing has been happening here:
>> svn://svn.freebsd.org/base/projects/fuse/
>>
>> which is essentially an HEAD branch + fuse kernel components. In order
>> to get fuse, please compile a kernel from this branch with FUSE option
>> or simply build and load fuse module.
>> Alternatively, a kernel patch that should work with HEAD_at_240684 is here:
>> http://www.freebsd.org/~attilio/fuse_import/fuse_240684.patch
>>
>> I guess the patch can easilly apply to all FreeBSD branches, really,
>> but it is not tested to anything else different then -CURRENT.
>>
>> As said you still need currently to build fusefs-kmod port. However
>> you need these further patches, to be put in the fusefs-kmod/files/
>> directory::
>> http://www.freebsd.org/~attilio/fuse_import/patch-Makefile
>> http://www.freebsd.org/~attilio/fuse_import/patch-mount_fusefs__mount_fusefs2.c
>>
>> They both disable the old kernel building/linking and import new
>> functionality to let the new kernel support work well in presence of
>> many consumers.
>>
>> In addition to fusefs-kmod, Bryan and Florian have also updated
>> fusefs-lib and fusefs-ntfs ports. For instance, please refer to this
>> e-mail:
>> http://lists.freebsd.org/pipermail/freebsd-ports/2012-August/077950.html
>>
>> Even if this work is someway independent by the fusefs-kmod import, I
>> warmly suggest to all of you to use their patches (and this what we
>> have been testing so far too.
>
> So, after Bryan and Florian ports update, I've also committed userland
> part of fusefs-kmod and now the project branch fully mirrors
> functionality of fusefs-kmod. The code in projects/fuse, infact, will
> also install mount_fusefs as part of the fuse support.

I've committed the FUSE support into base as r241519.

Few things I would like to stress out at the present time:
- The import of the fusefs in base, along with update of fusefs-ntfs
and fusefs-libs ports (which happened by Florian and Brian), will
bring us a fairly good ntfs (and possibly smbfs too, which is
currently untested, as far as I know) support. In particular,
in-kernel ntfs is so fragile that I don't understand how people cannot
easily experience deadlocks, memory leaks and panics at the present
time.
- In -CURRENT, you don't need to install fusefs-kmod anymore. It would
be good if a port committer (Florian, Brian?) will update the port in
order to forbid install of fusefs-kmod in -CURRENT at the present
time. I didn't bump the __FreeBSD_version because it is really
unnecessary (port can just detect 10xxxx version and be ok with not
installing the port) but once I will MFC it, I will bump the version
in order to make the port not installable in STABLE branches
- It would be good if someone could give a sweep to the style(9) of
the fuse code. I tried to reduce the changes to the gnn's proposed
patches at the minimum, thus I didn't fix style before to commit it,
but there are several style violations that should be addressed. This
could be a work for junior developers willing to take over.
- There are still a couple of bugs, but they are infrequent and minor
enough that they can be addressed after the committing of the
infrastructure. gnn has agreed to take care of them in the long term
and I will feed him as soon as possible with the details to reproduce
and fix them. I will also help directly with fuse code when I will
have available time.

Next step for the VFS deorbit project now include the disconnect of
non-MPSAFE filesystems, including NTFS and SMBFS that can be now used
as FUSE modules instead.

Thanks,
Attilio


-- 
Peace can only be achieved by understanding - A. Einstein