Re: Should we simply disallow ZFS on FreeBSD/i386?

On Sun, Jan 06, 2008 at 01:56:59PM -0800, Maxim Sobolev wrote:

  Gary Corcoran wrote:
>> Perhaps the 7.0 release notes should include a note to the effect that
>> ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due
  
  By watching this and other threads on ZFS and reading Sun's own design 
  papers I am getting strong impression that this should be even more strong 
  than strong NOT RECOMMENDED. Perhaps ZFS should BE DISALLOWED to run on 
  i386 at all (unless one does some manual source code tweaking or something 
  like this, and hence can ask no official support from the project).

I feel like stating my opinion on this, noting that I am usually stubborn enough to 
squeeze alot of value out of any rough product, while avoiding complaining about 
continuing problems if I am not prepared to put in appropriate effort to solve them.

A summary of my opinion on this matter is that some i386 FreeBSD servers do have a 
place running zfs in a useful role, but some dedication and patience from the 
administrator is usually required, and the effort to tune at least kmem is nearly 
required on ALL hardware platforms, not just i386.  I think kmem shortages from zfs 
are simply more touchy on i386, and with enough ram and slightly more tuning than 
amd64 the kmem can most likely be tuned away, but this does not do anything for 
other zfs problems such as zil deadlocks and other deadlocks.  I think doing 
something to prevent FreeBSD/i386 users from using zfs will just rule out a portion 
of the people having problems, and admins who take a little time to tune zfs AND use 
it more than just lightly may continue to have problems, and will just come back to 
the lists.

I have zfs on at least 4 systems presently, each one tuned to where I no longer
receive kmem panics at least based on their expected system load.  2 of them
are i386 and I would be quite dismayed to upgrade RELENG_7 to find ZFS has 
been disabled for me (although since I read the mailing lists I would expect 
it and deal appropriately).  It would be a tradeoff between breaking a limited
amount of existing setups versus somehow limiting the influx of new zfs users
who _may_ encounter zfs problems (of course you will only hear from the people
who complain).  

The amount of kmem required for a particular workload on any one machine can vary 
alot.  Believe it or not, it is one of my AMD64 systems that I had to increase kmem 
to 1.6G to prevent kmem panics (it does some heavy nightly rsyncs); versus just 
having kmem set to 1G on a i386 system that constantly serves out files to the 
internet with various rsyncs running through the day.  I don't think its exactly 
fair to punish all i386 users by disabling functionality, but I'm not the one that 
has to support all the users so I can understand the caution.  I'm certainly in 
favor of more dire warnings where appropriate.  Perhaps even a simple runtime 
warning when creating or importing a zpool?  I still think problems with running zfs 
on i386 will be the tip of that iceberg.

Certainly light use of zfs on an i386 might never see a crash.  But I think this 
might be a new frontier for a FreeBSD release to include functionality that is 
blatently labeled experimental and considered unstable yet will have wide appeal and 
interest; almost anyone can use a filesystem on FreeBSD, its not like it is just an 
unstable network driver affecting a small portion of users.
  
  I believe that 95% of hardware today that realistically is capable of 
  running ZFS is also capable of running 64bit code, so that potential ZFS 
  users are far better off switching to FreeBSD/amd64 and help 
  testing/improving that architecture than fighting architectural limitations 
  of already dying i386. And we are as a project are better off too, by 
  spending out limited resources on something that has future.

One of my busiest zfs servers is a dual Xeon 2.0ghz (i386 only) with 2 gigs
of ram (and I plan to add more 'just cuz').  I don't have any spare amd64 
systems I could replace it with at this time, yet I feel the hardware is 
up to the job if configured as best I can.  I have plenty of spares of this
server type since it is older yet not useless.  It pushes out data constantly
to the internet, boots off of zfs, and I believe has never crashed from a 
kmem panic thanks to the tuning I have set below, gathered from wiki, mailing
lists, and experience:

vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_max="104857600"
vm.kmem_size_max="1G"
vfs.zfs.zil_disable="1"

I will not pretend it is perfectly stable (it has hung every couple of weeks)
but it is not due to a kmem panic.  zil was disabled because it caused deadlocks.
I think what I face now is some other kind of deadlock, which I will try harder
to debug the next time it happens, although I have to balance time spent with 
benefit.  It has only happened twice since I disabled zil.  This server is doing 
something useful with zfs yet I can put up with occasional downtime without much 
complaint from users.  I am using it as a semi-production testbed to kick the tires 
on zfs and use the experience to boost my zfs skills and hopefully help out the 
whole FreeBSD community with my results when possible. 

    From my own experience FreeBSD/amd64 is quite mature for running most if 
  not all of the server tasks today and ZFS is first and foremost a server 
  FS. The only place where FreeBSD/i386 beats FreeBSD/amd64 is desktop, due 
  to binary drivers and such, but ZFS is almost useless there. So that by 
  simply officially disallowing ZFS on FreeBSD/i386 we could win by a great 
  margin.
  
  Just my CAD0.02.
  
  -Maxim
  _______________________________________________
  freebsd-current_at_freebsd.org mailing list
  http://lists.freebsd.org/mailman/listinfo/freebsd-current
  To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"