Re: writing to 3ware escalade uses up 100% cpu time in system.

Doug White wrote:
> On Fri, 9 Jan 2004, Klaus Robert Suetterlin wrote:
> 
> 
>>Hello,
>>
>>I'm using a 3ware 8506-4LP controller with 4 250GB harddrives in
>>Raid 0.  All space is in a single slice, separated into 1GB swap and
>>the rest into twed0s1d which is mounted under /space...
>>
>>When I do
>>    ``dd if=/dev/zero of=/space/test bs=1000000 count=10000''
>>the twed0 maxes out at 70MB/sec and (according to systat) 70% usage.
>>Unfortunately my system goes to 100% cpu usage at the same time.
> 
> 
> What speed of bus is the 3ware attached to?
> 
> You might run 'systat -vmstat 2' and check the amount of CPU going to
> interrupts. I suspect you're maxing out on the interrupt handler.  Also
> look at the # of interrupts to the twe.
> 

The interrupt handler for the driver is actually pretty lean; most real 
work is offloaded to a taskqueue.  However, the driver as a whole is not
locked, so it's going to have contention on the Giant mutex.  He might
well get some better performance just my making the interrupt handler
be INTR_MPSAFE.  Unfortunately, this requires a bit of work and I don't
have any hardware to test on.

Another cheap hack to help performance would be to remove the 
INTR_ENTROPY flag from the call to bus_setup_intr() in twe_freebsd.c.
This will affect the quality of your entropy a bit, but it would be an
interesting test.

One issue that might be at play here is that the 3ware hardware has some
limits on DMA alignment for data buffers.  If the buffers don't meet
those limits, then the driver has to manually allocate and copy
temporary buffers.  The driver doesn't seem to keep any stats on this,
but it would be quite useful if it did.

Other tests to try would include doing I/O directly to the device, and
turning off softupdates, especially if you're doing lots of file
operations.

Scott