Re: ZFS honesty

Scott Long wrote:
> Kris Kennaway wrote:
>> Ivan Voras wrote:
>>> Kris Kennaway wrote:
>>>> Ivan Voras wrote:
>>>>> Robert Watson wrote:
>>>>>
>>>>>> I'm not sure if anyone has mentioned this yet in the thread, but 
>>>>>> another thing worth taking into account in considering the 
>>>>>> stability of ZFS is whether or not Sun considers it a production 
>>>>>> feature in Solaris.  Last I heard, it was still considered an 
>>>>>> experimental feature there as well.
>>>>>
>>>>> Last I heard, rsync didn't crash Solaris on ZFS :)
>>>>
>>>> [Citation needed]
>>>
>>> I can't provide citation about a thing that doesn't happen - you 
>>> don't hear things like "oh and yesterday I ran rsync on my Solaris 
>>> with ZFS and *it didn't crash*!" often.
>>>
>>> But, with some grains of salt taken, consider this Google results:
>>>
>>> * searching for "rsync crash solaris zfs": 790 results, most of them 
>>> obviously irrelevant
>>> * searching for "rsync crash freebsd zfs": 10,800 results; a small 
>>> number of the results is from this thread, some are duplicates, but 
>>> it's a large number in any case.
>>>
>>> I feel that the number of Solaris+ZFS installations worldwide is 
>>> larger than that of FreeBSD+ZFS and they've had ZFS longer.
>>
>> Almost all Solaris systems are 64 bit.
>>
>> Kris
> 
> So, let's be honest here.  ZFS is simply unreliable on FreeBSD/i386.
> There are things that you can do mitigate the problems, and in certain
> well controlled environments you might be able to make it work well
> enough for your needs.  But as a general rule, don't expect it to work
> reliably, period.  This is backed up by Sun's own recommendation to not
> run it on 32-bit Solaris.
> 
> But let's also be honest about ZFS in the 64-bit world.  There is ample
> evidence that ZFS basically wants to grow unbounded in proportion to the
> workload that you give it.  Indeed, even Sun recommends basically
> throwing more RAM at most problems.  Again, tuning is often needed, and
> I think it's fair to say that it can't be expected to work on arbitrary
> workloads out of the box.
> 
> Now, what about the other problems that have been reported in this
> thread by Ivan and others?  I don't think that it can be said that the
> only problem that ZFS has is with memory.  Unfortunately, it looks like
> these "other" problems aren't well quantified, so I think that they are
> being unfairly dismissed.  But at the same time, maybe these other
> problems are rare and unique enough that they represent very special
> cases that won't be encountered by most people.  But it also tells me
> that ZFS is still immature, at least in FreeBSD.
> 
> The universal need for tuning combined with the poorly understood
> problem reports tells me that administrators considering ZFS should
> expect to spend a fair amount of timing testing and tuning.  Don't
> expect it to work out of the box for your situation.  That's not to
> say that it's useless; there are certainly many people who can attest to
> it working well for them.  Just be prepared to spend time and possibly
> money making it work, and be willing to provide good problem reports for
> any non-memory related problems that you encounter.

To be clear, in this thread I have been mostly restricting myself to 
discussion of kmem problems only, although I have also noted that there 
are known ZFS bugs including bugs that are unfixed even in solaris (the 
ZIL low memory deadlock is one of them).  Indeed, pjd has a long list of 
bug reports from me :)

I agree with the rest of this summary.

Kris