SIIS/CAM driver performance

I sent this to mav earlier but thought it would be of wider interest...

My first attempt at measuring raw throughput with the SIIS driver
(Silicon Image) got 875 MByte/sec with no tuning or changes to an old
dual Opteron 244 (1.8GHz single core) Tyan K8W motherboard using two
3124 controllers and ten drives in four port-multiplier enclosures.  I
used 128KB blocks in the sequential reads since that's what ZFS seems to
do by default.  This same combo peaked at 375 MByte/sec with the ATA
driver with my (badly done) hacks.  SIIS appears stable.

SIIS should clearly do better than 1 GByte/sec with modern hardware. 
One unknown is how well PCI-Express cards with the 3124 perform: the
3124 is a PCI-X part, and PCI-Express cards using the 3124 use the Intel
41110 Serial to Parallel PCI bridge which may add some overhead.

mav has thrown down the GB/s gauntlet and we'll see which filesystems
can pick it up. :-)

This is exciting because it allows FreeBSD to be the OS of choice for
cheap storage servers or perhaps even high-speed data capture.  A 50 TB
server should price well under $10k, a price/performance point hard to
hit any other way.

Issues:

The Silicon Image 3132 is a native 2-port PCI-Express controller. but it
appears to have problems as I am not aware of any driver on any OS
capable of coaxing more than 150MByte/sec total out of it.  The chip
works but is very slow.  Use the 3124 for performance.

The SIIS driver doesn't support staggered spin-up yet.  Anyone wanting
to build a 50 TB storage array with SIIS needs to keep in mind that
spin-up power spike for now.