Re: Help me select hardware and software options for very large server

> We did something similar for our off-site, automated backups box.
>
> One box has 2x 2 GB CompactFlash in IDE adapters, the other has 2x 2
> GB USB flash drives.

  I assume this was for FreeBSD itself? I was concerned about write
cycles (and, to a lesser extent, storage capacity) on CF or USB
media. I haven't seen much degradation due to seeks when the FreeBSD
space is a separate logical drive on the AMCC controller. That also
gets me inherent RAID 6 protection (assuming I carve it from the main
batch of drives).

>   - 64-bit FreeBSD 7.1-RELEASE (started with 7.1-STABLE from August 08)
>   - UFS for / partition, using gmirror across the two CF/USB drives
>   - ZFS for everything else

  This seems to mirror what I'll be doing. It is good to hear that this
has been working well for you.

>   - uses rsync and ssh to backup 83 remote servers everynight,
> creating ZFS snapshots every night
>   - uses rsync to transfer "snapshots" between the two servers

  In my case, the data will originate on one of the servers (instead of
being backups of other servers) and will be synchronized with an off-
site server (same hardware) via either 1Gbit or 10Gbit Ethernet (right
now I have regular Gigabit hardware), but all I'd need is new add-in
cards and an upgrade to the switches at each end of the fiber. The
synchonization is currently dones nightly via rdiff-backup. But that
package can occasionally loose its marbles, even with < 2TB of data,
so I may have to consider alternatives. 

> The drives on each of the RAID controllers are configured as "Single
> Disk Array", so they appear as 24 separate drives to the OS, but still
> benefit from the controller's disk cache, management interface, and so
> on (as compared to JBOD where it acts like nothing more than a SATA
> controller).

  Hmmm. I was planning to use the hardware RAID 6 on the AMCC, for a
number of reasons: 1) that gives me front-panel indications of broken
RAID sets, controller-hosted rebuild, and so forth. 2) I'd be using
fewer ZFS features (basically, just large partitions and snapshots)
so if anything went wrong, I'd have a larger pool of expertise to draw
on to fix things (all AMCC users, rather than all FreeBSD ZFS users).

  Did you consider this option and reject it? If so, can you tell me
why?

> The drives on one box are configured as 1 large 24-drive raidz2 in ZFS
> (this box also has 12x 400 GB drives).

> The drives on the other box are configured as 2 separate 11-drive
> raidz2 arrays, with 2 hot spares.

> The usable space on box boxes is 9 TB.

  So a single ZFS partition of 8TB would be manageable without long
delays for backup snapshots?

> Other than a bit of kernel tuning back in August/September, these
> boxes have been running nice and smooth.  Just waiting for either the
> release of FreeBSD 8.0 or an MFC of ZFS v13 to 7-STABLE to get support
> for auto-rebuild using hot spares.

  That's good to hear. What sort of tuning was involved (if it is still
needed)?

	Thanks,
        Terry Kennedy             http://www.tmk.com
        terry_at_tmk.com             New York, NY USA