Re: ZFS honesty

Kris Kennaway wrote:
> Scott Long wrote:
>> Kris Kennaway wrote:
>>> Ivan Voras wrote:
>>>> Kris Kennaway wrote:
>>>>> Ivan Voras wrote:
>>>>>> Robert Watson wrote:
>>>>>>
>>>>>>> I'm not sure if anyone has mentioned this yet in the thread, but 
>>>>>>> another thing worth taking into account in considering the 
>>>>>>> stability of ZFS is whether or not Sun considers it a production 
>>>>>>> feature in Solaris.  Last I heard, it was still considered an 
>>>>>>> experimental feature there as well.
>>>>>>
>>>>>> Last I heard, rsync didn't crash Solaris on ZFS :)
>>>>>
>>>>> [Citation needed]
>>>>
>>>> I can't provide citation about a thing that doesn't happen - you 
>>>> don't hear things like "oh and yesterday I ran rsync on my Solaris 
>>>> with ZFS and *it didn't crash*!" often.
>>>>
>>>> But, with some grains of salt taken, consider this Google results:
>>>>
>>>> * searching for "rsync crash solaris zfs": 790 results, most of them 
>>>> obviously irrelevant
>>>> * searching for "rsync crash freebsd zfs": 10,800 results; a small 
>>>> number of the results is from this thread, some are duplicates, but 
>>>> it's a large number in any case.
>>>>
>>>> I feel that the number of Solaris+ZFS installations worldwide is 
>>>> larger than that of FreeBSD+ZFS and they've had ZFS longer.
>>>
>>> Almost all Solaris systems are 64 bit.
>>>
>>> Kris
>>
>> So, let's be honest here.  ZFS is simply unreliable on FreeBSD/i386.
>> There are things that you can do mitigate the problems, and in certain
>> well controlled environments you might be able to make it work well
>> enough for your needs.  But as a general rule, don't expect it to work
>> reliably, period.  This is backed up by Sun's own recommendation to not
>> run it on 32-bit Solaris.
>>
>> But let's also be honest about ZFS in the 64-bit world.  There is ample
>> evidence that ZFS basically wants to grow unbounded in proportion to the
>> workload that you give it.  Indeed, even Sun recommends basically
>> throwing more RAM at most problems.  Again, tuning is often needed, and
>> I think it's fair to say that it can't be expected to work on arbitrary
>> workloads out of the box.
>>
>> Now, what about the other problems that have been reported in this
>> thread by Ivan and others?  I don't think that it can be said that the
>> only problem that ZFS has is with memory.  Unfortunately, it looks like
>> these "other" problems aren't well quantified, so I think that they are
>> being unfairly dismissed.  But at the same time, maybe these other
>> problems are rare and unique enough that they represent very special
>> cases that won't be encountered by most people.  But it also tells me
>> that ZFS is still immature, at least in FreeBSD.
>>
>> The universal need for tuning combined with the poorly understood
>> problem reports tells me that administrators considering ZFS should
>> expect to spend a fair amount of timing testing and tuning.  Don't
>> expect it to work out of the box for your situation.  That's not to
>> say that it's useless; there are certainly many people who can attest to
>> it working well for them.  Just be prepared to spend time and possibly
>> money making it work, and be willing to provide good problem reports for
>> any non-memory related problems that you encounter.
> 
> To be clear, in this thread I have been mostly restricting myself to 
> discussion of kmem problems only, although I have also noted that there 
> are known ZFS bugs including bugs that are unfixed even in solaris (the 
> ZIL low memory deadlock is one of them).  Indeed, pjd has a long list of 
> bug reports from me :)
> 
> I agree with the rest of this summary.
> 
> Kris

I guess what makes me mad about ZFS is that it's all-or-nothing; either
it works, or it crashes.  It doesn't automatically recognize limits and
make adjustments or sacrifices when it reaches those limits, it just
crashes.  Wanting multiple gigabytes of RAM for caching in order to
optimize performance is great, but crashing when it doesn't get those
multiple gigabytes of RAM is not so great, and it leaves a bad taste in
my mouth about ZFS in general.

Scott