Re: Help me select hardware and software options for very large server

On 24/01/2009, at 1:30 PM, Terry Kennedy wrote:

>  I'm pretty sure I need ZFS, since even with the 2TB partitions I  
> have now,
> taking snapshots for dump or doing a fsck take approximately forever  
> 8-)
> I'll be using the harware RAID 6 on the 3Ware controller, so I'd  
> only be
> using ZFS to get filesystems larger than 2TB.

If you do RAID on the hardware card you get a few downsides:

* if your 3ware card dies you'll need a spare of the same model and  
same BIOS to replace it with. Otherwise you risk your disks not being  
properly detected.

* you don't get the advantage of ZFS check-summing and auto-repairing  
the data. There are some very nice features in ZFS for doing this on  
the fly and periodically. You'll want to compare that against what the  
hardware card gives you.

* personally I've found the 3ware utilities much more cumbersome to  
use than ZFS tools

* you can't use RAIDz which has some nice features

You'll want to do some tests and see how it performs in different  
configurations using the type of data and access you plan on  
implementing. Another person's benchmarks may not necessarily apply to  
your specific setup, so try it and see. The 3ware cards are still very  
good to use even if you don't use them in RAID mode.


> I've been following the ZFS discussions on -current and -stable, and I
> think that while it isn't quite ready yet, it probably will be ready  
> in
> a few months, being available around the same time I get this hardware
> asssembled. I recall reading that there will be an import of newer ZFS
> code in the near future.

That new code is already in Current. Whether it will be ported to 7 is  
not yet known. On the other hand a great many people are running  
7.0/7.1 ZFS in production very successfully with properly tuning.


>  I think this system may have the most storage ever configured on a
> FreeBSD system, and it is probably up near the top in terms of CPU and
> memory.

I doubt it. 8 core systems are very common these days. I've seen  
benchmarking with anything up to 64 cores (which I believe is the  
current FreeBSD limit). As for memory, there was a recent thread with  
someone installing 64Gb also claiming theirs was the biggest, but  
others insisting that it certainly wasn't. As for storage, I know Sun  
sell systems into the petabyte range which I assume these days are  
powered by ZFS.

The above is not to belittle your project; more to reassure you that  
you aren't anywhere near the limits and you should have no troubles if  
you tune things properly.

Ari Maniatis



-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A