Re: When will ZFS become stable?

Kris Kennaway wrote:
> Ivan Voras wrote:
>> Kris Kennaway wrote:
>>
>>> No, clearly it is not enough 
>>
>> This looks like we're constantly chasing the "right amount". Does it 
>> depend so much on CPU and IO speed that there's never a generally 
>> sufficient "right amount"? So when CPU and drive speed increase, the 
>> new amount will always be some bigger value?
> 
> It depends on your workload, which in turn depends on your hardware. The 
> harder you can drive ZFS the more memory it will require.

As a user, I would expect the above to mean "to continue running quickly".
If it has to slow to a crawl for a moment, due to inadequate memory in
your system, then that's just tough cookies.  But crashing (panicing)
is not really acceptable for most people (maybe except a developer).
Again from a user perspective, if ZFS needs "tuning" to run at full speed,
or even at all, I would expect *it* to be able to do a few simple calculations
and do the tuning itself!  :-)  (even if, in worst case, it requires a clean
shutdown and reboot for the new values to take effect)

The above is not meant as a criticism of the current explicitly-labeled
"experimental" code.  Rather, it is what I would hope we might be able
to see sometime over the next year...

>>> (and you claimed previously to have done more tuning than this). 
>>
>> Where? What else is there except kmem tuning (including KVA_PAGES)? 
>> IIRC Pawel said all other suggested tunings don't do much.
> 
> Tuning is an interactive process.  If 512MB is not enough kmem_map, then 
> increase it.  Repeat as necessary.
> 
>>> I have it set to 600MB on the i386 system with a 1.5GB KVA.  Both 
>>> were necessary.
>>
>> My point is that the fact that such things are necessary (1.5 GB KVA 
>> os a lot on i386) mean that there are serious problems which aren't 
>> getting fixed since ZFS was imported (that's over 6 months ago).
> 
> ZFS is a memory hog.  There is nothing that can really be done about 
> this, and it is just not a good fit on i386 because of limitations of 
> the hardware architecture.  Note that Sun does not recommend using ZFS 
> on a 32-bit system either, for the same reasons.  It is unlikely this 
> can really be fixed, although mitigation strategies like the vm_kern.c 
> patch are possible.

Perhaps the 7.0 release notes should include a note to the effect that
ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due
to the likelihood of panics.  I say this because it sure sounds like
"out of the box" that is what you're most likely to end up with, and
even with manual "corrections" you may still have panics.  So why not
just be upfront about it and tell people that, at least at this time,
ZFS is only recommended for 64-bit systems, with a minimum of "N" (2?)
GB of memory?  If you were already planning something like this for
the release notes, my apologies.

>> I see you've added to http://wiki.freebsd.org/ZFSTuningGuide; can you 
>> please add the values that work for you to it (especially for 
>> KVA_PAGES since the exact kernel configuration line is never spelled 
>> out in the document; and say for which hardware are the values known 
>> to work)?
> 
> OK.
> 
>>> ZFS already tells you up front that it's experimental code and likely 
>>> to have problems.  
>>
>> I know it's experimental, but requiring users to perform so much 
>> tuning just to get it work without crashing will mean it will get a 
>> bad reputation early on. Do you (or anyone) know what are the reasons 
>> for not having vm.kmem_size to 512 MB by default?
> 
> Increasing vm.kmem_size.max to 512MB by default has other implications, 
> but it is something that should be considered.
> 
>  > Better yet, why not
>> increase both vm.kmem_size and KVA_PAGES to (the equivalent of) 640 MB 
>> or 768 MB by default for 7.0?
> 
> That is answered in the tuning guide.  Tuning KVA_PAGES by default is 
> not appropriate.
> 
>>  >Users of 7.0-RELEASE should not have unrealistic
>>  > expectations.
>>
>> As I've said at the first post of this thread: I'm interested in if 
>> it's ever going to be stable for 7.x.
> 
> This was in reply to a comment you made about the vm_kern.c patch 
> affecting users of 7.0-RELEASE.
> 
> Anyway, to sum up, ZFS has known bugs, some of which are unresolved by 
> the authors, and it is difficult to make it work on i386.  It is likely 
> that the bugs will be fixed over time (obviously), but amd64 will always 
> be a better choice than i386 for using ZFS because you will not be 
> continually bumping up against the hardware limitations.

BTW, I am a happy user of ZFS on a 2GB Core2Duo 64-bit system.  I never
did any "tuning", it "just worked" for my light-duty file serving needs.
This was from the (I believe) May 2007 snapshot.

Gary