Re: Help me select hardware and software options for very large server

On Fri, Jan 23, 2009 at 6:30 PM, Terry Kennedy <terry_at_tmk.com> wrote:
>  I'm in the process of planning the hardware and software for the second
> generation of my RAIDzilla file servers (see http://www.tmk.com/raidzilla
> for the current generation, in production for 4+ years).
>
>  I expect that what I'm planning is probably "off the scale" in terms of
> processing and storage capacity, and I'd like to find out and address any
> issues before spending lots of money. Here's what I'm thinking of:
>
> o Chassis - CI Design SR316 (same model as current chassis, except i2c link
>  between RAID controller and front panel
> o Motherboard - Intel S5000PSLSATAR
> o CPU - 2x Intel Xeon E5450 BX80574E5450P
> p Remote management - Intel Remote Management Module 2 - AXXRM2
> o Memory - 16GB - 8x Kingston KVR667D2D4F5/2GI
> o RAID controller - 3Ware 9650SE-16ML w/ BBU-MODULE-04
> o Drives - 16x 2TB drives [not mentioning manufacturer yet]
> o Cables - 4x multi-lane SATA cables
> o DVD-ROM drive
> o Auxiliary slot fan next to BBU card
> o Adaptec AHA-39160 (for Quantum Superloader 3 tape drive)
>
>  So much for the hardware. On the software front:
>
> o FreeBSD 8.x?
> o amd64 architecture
> o MBR+UFS2 for operating system partitions (hard partition in controller)
> o GPT+ZFS for data partitions
> o Multiple 8TB data partitions (separate 8TB controller partitions or one
>  big partition divided with GPT?)

We did something similar for our off-site, automated backups box.

Hardware (2 identical boxes):
  - Chenbro 5U chassis with 24 hot-swap drive bays
  - 1350 W 4-way redundant PSU
  - Tyan h2000M motherboard
  - 2x Opteron 2200-series CPUs (dual-core) _at_ 2.8 GHz
  - 8 GB DDR2-800 SDRAM
  - 3Ware 9650SE-12ML PCIe RAID controller
  - 3Ware 9550SX-12ML PCI-X RAID controller
  - 12x Seagate Barracude 7200.11 500 GB SATA drives
  - 12x WD 500 GB SATA drives
  - Intel Pro/1000MT 4-port gigabit NIC

One box has 2x 2 GB CompactFlash in IDE adapters, the other has 2x 2
GB USB flash drives.

Software:
  - 64-bit FreeBSD 7.1-RELEASE (started with 7.1-STABLE from August 08)
  - UFS for / partition, using gmirror across the two CF/USB drives
  - ZFS for everything else
  - uses rsync and ssh to backup 83 remote servers everynight,
creating ZFS snapshots every night
  - uses rsync to transfer "snapshots" between the two servers

The drives on each of the RAID controllers are configured as "Single
Disk Array", so they appear as 24 separate drives to the OS, but still
benefit from the controller's disk cache, management interface, and so
on (as compared to JBOD where it acts like nothing more than a SATA
controller).

The drives on one box are configured as 1 large 24-drive raidz2 in ZFS
(this box also has 12x 400 GB drives).

The drives on the other box are configured as 2 separate 11-drive
raidz2 arrays, with 2 hot spares.

The usable space on box boxes is 9 TB.

I'm experimenting with different ways to allocated the drives in ZFS,
to get the best performance.  The recommendations of the Solaris folks
seems to be to use raidz arrays of fewer than 10 disks.  Depending on
the cost, for the next storage box, I may use 3x 3Ware 9550SX RAID
controllers and use 3x raidz2 arrays of 8 disks each, to see how that
compares.

Other than a bit of kernel tuning back in August/September, these
boxes have been running nice and smooth.  Just waiting for either the
release of FreeBSD 8.0 or an MFC of ZFS v13 to 7-STABLE to get support
for auto-rebuild using hot spares.

Also waiting for a pair of CF-to-SATA controllers to come in so that I
can remove the USB flash drives from the one box.  They're just too
unreliable in my testing.  The CF-to-IDE adapters work really well,
but there's only a single IDE controller on the motherboard, so it's
doing gmirror across master and slave devices on the same controller,
which isn't the fastest way of doing things.

-- 
Freddie Cash
fjwcash_at_gmail.com