Help me select hardware and software options for very large server

  [I posted the following message to freebsd-questions, as I thought it
woule be the most appropriate list. As it has received no replies in two
weeks, I'm trying freebsd-current.]

--------

  [I decided to ask this question here as it overlaps -hardware, -current,
and a couple other lists. I'd be glad to redirect the conversation to a
list that's a better fit, if anyone would care to suggest one.]

  I'm in the process of planning the hardware and software for the second
generation of my RAIDzilla file servers (see http://www.tmk.com/raidzilla
for the current generation, in production for 4+ years).

  I expect that what I'm planning is probably "off the scale" in terms of
processing and storage capacity, and I'd like to find out and address any
issues before spending lots of money. Here's what I'm thinking of:

o Chassis - CI Design SR316 (same model as current chassis, except i2c link
  between RAID controller and front panel
o Motherboard - Intel S5000PSLSATAR
o CPU - 2x Intel Xeon E5450 BX80574E5450P
p Remote management - Intel Remote Management Module 2 - AXXRM2
o Memory - 16GB - 8x Kingston KVR667D2D4F5/2GI
o RAID controller - 3Ware 9650SE-16ML w/ BBU-MODULE-04
o Drives - 16x 2TB drives [not mentioning manufacturer yet]
o Cables - 4x multi-lane SATA cables
o DVD-ROM drive
o Auxiliary slot fan next to BBU card
o Adaptec AHA-39160 (for Quantum Superloader 3 tape drive)

  So much for the hardware. On the software front:

o FreeBSD 8.x?
o amd64 architecture
o MBR+UFS2 for operating system partitions (hard partition in controller)
o GPT+ZFS for data partitions
o Multiple 8TB data partitions (separate 8TB controller partitions or one
  big partition divided with GPT?)

  I looked at "Large data storage in FreeBSD", but that seems to be a stale
page from 2005 or so: http://www.freebsd.org/projects/bigdisk/index.html

  I'm pretty sure I need ZFS, since even with the 2TB partitions I have now,
taking snapshots for dump or doing a fsck take approximately forever 8-)
I'll be using the harware RAID 6 on the 3Ware controller, so I'd only be
using ZFS to get filesystems larger than 2TB.

  I've been following the ZFS discussions on -current and -stable, and I
think that while it isn't quite ready yet, it probably will be ready in
a few months, being available around the same time I get this hardware
asssembled. I recall reading that there will be an import of newer ZFS 
code in the near future.

  Similarly, the ports collection seems to be moving along nicely with
amd64 support.

  I think this system may have the most storage ever configured on a
FreeBSD system, and it is probably up near the top in terms of CPU and
memory. Once I have it assembled I'd be glad to let any FreeBSD devel-
opers test and stress it if that would help improve FreeBSD on that
type of configuration.

In the meantime, any suggestions regarding the hardware or software con-
figuration would be welcomed.

        Terry Kennedy             http://www.tmk.com
        terry_at_tmk.com             New York, NY USA