Re: MPSAFE VFS -- List of upcoming actions

On Wed, Sep 19, 2012 at 12:30 AM, Attilio Rao <attilio_at_freebsd.org> wrote:
> On Wed, Sep 19, 2012 at 4:47 AM, Kevin Oberman <kob6558_at_gmail.com> wrote:
>> On Tue, Sep 18, 2012 at 7:48 PM, Attilio Rao <attilio_at_freebsd.org> wrote:
>>> On Fri, Jul 13, 2012 at 12:18 AM, Attilio Rao <attilio_at_freebsd.org> wrote:
>>>> 2012/7/4 Attilio Rao <attilio_at_freebsd.org>:
>>>>> 2012/6/29 Attilio Rao <attilio_at_freebsd.org>:
>>>>>> As already published several times, according to the following plan:
>>>>>> http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS
>>>>>>
>>>>>
>>>>> I still haven't heard from Vivien or Edward, anyway as NTFS is
>>>>> basically only used RO these days (also the mount_ntfs code just
>>>>> permits RO mounting) I stripped all the uncomplete/bogus write support
>>>>> with the following patch:
>>>>> http://www.freebsd.org/~attilio/ntfs_remove_write.patch
>>>>>
>>>>> This is an attempt to make the code smaller and possibly just focus on
>>>>> the locking that really matter (as read-only filesystem).
>>>>> On some points of the patch I'm a bit less sure as we could easily
>>>>> take into account also write for things like vaccess() arguments, and
>>>>> make easier to re-add correct write support at some point in the
>>>>> future, but still force RO, even if the approach used in the patch is
>>>>> more correct IMHO.
>>>>> As an added bonus this patch cleans some dirty code in the mount
>>>>> operation and fixes a bug as vfs_mountedfrom() is called before real
>>>>> mounting is completed and can still fail.
>>>>
>>>> A quick update on this.
>>>> It looks like NTFS won't be completed for this GSoC thus I seriously
>>>> need to find an alternative to not loose the NTFS support entirely.
>>>>
>>>> I tried to look into the NTFS implementation right now and it is
>>>> really a poor support. As Peter has also verified, it can deadlock in
>>>> no-time, it compeltely violates VFS rules, etc. IMHO it deserves a
>>>> complete rewrite if we would still support in-kernel NTFS. I also
>>>> tried to look at the NetBSD implementation. Their code is someway
>>>> similar to our, but they used very complicated (and very dirty) code
>>>> to do the locking. Even if I don't know well enough NetBSD VFS, I have
>>>> the impression not all the races are correctly handled. Definitively,
>>>> not something I would like to port.
>>>>
>>>> Considering all that the only viable option would be meaning an
>>>> userland filesystem implementation. My preferred choice would be to
>>>> import PUFFS and librefuse on top of it but honestly it requires a lot
>>>> of time to be completed, time which I don't currently have as in 2
>>>> months Giant must be gone by the VFS.
>>>>
>>>> I then decided to switch to gnn's rewamp of FUSE patches. You can find
>>>> his initial e-mail here:
>>>> http://lists.freebsd.org/pipermail/freebsd-fs/2012-March/013876.html
>>>>
>>>> I've precisely got the second version of George's patch and created
>>>> this dolphin branch:
>>>> svn://svn.freebsd.org/base/projects/fuse
>>>>
>>>> I'm fixing low hanging fruit for the moment (see r238411 for example)
>>>> and I still have to make a throughful review.
>>>> However my idea is to commit the support once:
>>>> - ntfs-3g is well stress-tested and proves to be bug-free
>>>> - there is no major/big technical issue pending after the reviews
>>>
>>> In the last weeks Peter, Florian, Gustau and I have been working in
>>> stabilizing fuse support. In the specific, Peter has worked hard on
>>> producing several utilities to nit stress-test fuse and in particular
>>> ntfs, Florian has improved fuse related ports (as explained later) and
>>> Gustau has done sparse testing. I feel moderately satisfied by the
>>> level of stability of fuse now to propose to wider usage, in
>>> particular given the huge amount of complaints I'm hearing around
>>> about occasional fuse users.
>>>
>>> The final target of the project is to completely import into base the
>>> content of fusefs-kmod starting from earlier posted patches by George.
>>> So far, we took care only of importing in the fuse branch the kernel
>>> part, so that fusefs-kmod userland part is still needed to be
>>> installed from ports, but I was studying the mount_fusefs licensing
>>> before to process with the import for the userland bits of it.
>>>
>>> The fixing has been happening here:
>>> svn://svn.freebsd.org/base/projects/fuse/
>>>
>>> which is essentially an HEAD branch + fuse kernel components. In order
>>> to get fuse, please compile a kernel from this branch with FUSE option
>>> or simply build and load fuse module.
>>> Alternatively, a kernel patch that should work with HEAD_at_240684 is here:
>>> http://www.freebsd.org/~attilio/fuse_import/fuse_240684.patch
>>>
>>> I guess the patch can easilly apply to all FreeBSD branches, really,
>>> but it is not tested to anything else different then -CURRENT.
>>>
>>> As said you still need currently to build fusefs-kmod port. However
>>> you need these further patches, to be put in the fusefs-kmod/files/
>>> directory::
>>> http://www.freebsd.org/~attilio/fuse_import/patch-Makefile
>>> http://www.freebsd.org/~attilio/fuse_import/patch-mount_fusefs__mount_fusefs2.c
>>>
>>> They both disable the old kernel building/linking and import new
>>> functionality to let the new kernel support work well in presence of
>>> many consumers.
>>>
>>> In addition to fusefs-kmod, Bryan and Florian have also updated
>>> fusefs-lib and fusefs-ntfs ports. For instance, please refer to this
>>> e-mail:
>>> http://lists.freebsd.org/pipermail/freebsd-ports/2012-August/077950.html
>>>
>>> Even if this work is someway independent by the fusefs-kmod import, I
>>> warmly suggest to all of you to use their patches (and this what we
>>> have been testing so far too).
>>>
>>> At this point what I'm looking for are reviews and further testing.
>>> I would like to spend some words on what you should expect from this work:
>>> *Fuse is far from being perfect*.
>>> I cannot stress this enough. Peter stress-tests could break also Fuse
>>> on Linux generally and by Fuse authors admissions the modules can
>>> never guarantee to be completely starvation-free. However, they tend
>>> to be designed in a way that sleeps can be at least interrupted
>>> easily, making at least easy to recover from deadlocks. This is mostly
>>> retained also in FreeBSD, for what I can tell. Also, sometimes fuse
>>> seems to leave a small amount of hidden files, when it find references
>>> on files it wants to delete. This happens also under Linux and it is
>>> part of FUSE design, not much we can do.
>>> However, if deadlocks can be someway tollerated, things you should
>>> really pay attention are dumps of fuse modules (like ntfs-3g binary)
>>> and kernel panics. They must not happen and if they do they need to be
>>> fixed promptly.
>>> However, the good new is that ntfs seems doing exceptionally good.
>>> Florian could use ntfs as a backend for postgresql test. I think this
>>> is by far a big improvement if compared to current in-kernel ntfs
>>> which is completely torned.
>>>
>>> So far we have almost entirely tested only ntfs-3g. I know Gustau also
>>> used other modules like sshfs and George used GlusterFS with his older
>>> patches, but I encourage you to test as many modules as you want, as
>>> they may expose different bugs. Of course, I don't plan to spend much
>>> more time on FUSE, but I can occasionally look at bugs as they fall in
>>> the filesystems category and I'm always interested in keeping a good
>>> open eye on such issues.
>>>
>>> A few operational informations:
>>> - In the next days I will import the userland bits of fusefs-kmod to
>>> the fuse project branch making the port obsolete. When this happens I
>>> will make this clear to the user of this thread.
>>> - If no major bug is remained by the early October, I will commit this
>>> to -CURRENT
>>> - I expect Bryan and Florian to commit libfuse and ntfs updates soon.
>>> They can do independently from the fusefs-kmod retiral, but I would
>>> prefer their patches to go on first.
>>> - After that I will handover fusefs maintainership to gnn as agreed in
>>> precedence but I will be around helping with analysis and fixing,
>>> depending on time availability
>>>
>>> In the end I have really 2 minor questions:
>>> - One is about importing the mount_fusefs userland bits. I don't think
>>> we need a vendor import at all because they were developed by a
>>> FreeBSD GSoC student and kept in his git repo (or someone else's).
>>> Anyway, i'd just commit as new files once I do a good sweep. I hope
>>> nobody objects to that.
>>> - Another one is: fusefs-kmod right now is only amd64/i386 specific. I
>>> have no idea why as it has not any MD specific code. However I'm sure
>>> it has not been tested on other arches so far. Anyway I left it usable
>>> by all the arches. I think this is the correct choice. If someone
>>> objects with valid argument I can bring it back to be usable only on
>>> i386 and amd64.
>>>
>>> That's all, for any question please don't hesitate to contact me and
>>> the other people involved in this work.
>>
>> Attilio (and the crew),
>>
>> Thanks for working on fusefs-ntfs. It's been increasingly worrying to
>> me that we might lose it and I really depend on it. I really hope to
>> be able to use rsync to update files without killing my system some
>> day.
>>
>> I tried the new fusefs-libs and fusefs-ntfs ports from Florian and
>> Bryan, but ran into trouble as I could no longer build the kmod after
>> installing the updated fusefs-libs. It had an unresolved symbol:
>> cc -O2 -pipe -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE
>> -nostdinc  -I../include -I. -I_at_ -I_at_/contrib/altq -finline-limit=8000
>> --param inline-unit-growth=100 --param large-function-growth=1000
>> -fno-common  -fno-omit-frame-pointer  -mcmodel=kernel -mno-red-zone
>> -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables
>> -ffreestanding -fstack-protector -std=iso9899:1999 -fstack-protector
>> -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes
>> -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef
>> -Wno-pointer-sign -fformat-extensions  -Wmissing-include-dirs
>> -fdiagnostics-show-option   -c fuse_vnops.c
>> fuse_vnops.c: In function 'create_filehandle':
>> fuse_vnops.c:1586: error: 'struct fuse_open_in' has no member named 'mode'
>> *** [fuse_vnops.o] Error code 1
>>
>> This was on amd64 9-Stable r239879 until/unless this issue is
>> resolved, please keep the existing port available and/or mark the new
>> one to not install on pre-10 systems.
>
> If you follow the rule I described in this e-mail, the fusefs-kmod
> kernel part won't be build anymore, so you won't run into this.
> If it is build yet, please let me know because there is a bug in the 2
> patches I posted for fusefs-kmod port.

Attilo,

I assumed that your new kernel module was only tested/working with
current, so I did not try to use it. I was only referring to the use
of the updated of fusefs-libs and fusefs-ntfs that Florian and Bryan
provided. I had tested these on 9-stable and found that after
installing the updated fusefs-libs, the old fusefs-kmod port would no
longer compile.

Today Florian sent me a one line patch to fuse-modue/fuse-vnops.c in
the current fusefs-kmod port which appears to have fixed the problem.
It compiled fine and it is currently running on the system on which I
am typing this. I have done a bit of light testing and it works to
this point. I'll do some heavier testing later today. So it looks like
this there is probably no issue with Florian committing the new
fusefs-libs and fusefs-ntfs ports for those of us not running current.

If I get enough time, I'll look into applying the patches to the
kernel module on 9-stableand see how that does, but I have my day job
and contractors working in the house, so I won't make any promises.

Thanks again to you and all of the others who contributed to this. May
not be perfect, but it is a huge win over the kernel NTFS code,
especially or those of us who need to actually write a file now and
then.
-- 
R. Kevin Oberman, Network Engineer
E-mail: kob6558_at_gmail.com