(long) Re: writing to 3ware escalade uses up 100% cpu time in system.

Hi,

I'll try to provide more information on the system I use and on the
problem I see along the line of three emails I recieved so far.

My system is a P3-933 Compaq Proliant ML350, with 512MB of Ram.
Using software raid (i.e. ccd) on 4 scsi-2 or ultra ata-100 disks
I can get 80MByte/sec easily, without any significant cpu usage.
Unfortunately ultra ata cable length and scsi prices are too limiting
for my storage needs (1GByte in 4 Disks), thus I opted for a serial
ata raid controller.

"Subhro" <subhro_at_fusemail.com> asked:
> Can you post this?
> 
> sysctl -a |grep dma
    vfs.nfs.iodmaxidle: 120
    vfs.nfs.iodmax: 20
    hw.ata.ata_dma: 1
    hw.ata.atapi_dma: 0

But I do not see the point here.  twe(4) does not list any sysctl
variables for dma, so I'd expect to find none.

``sysctl -a|grep twe'' gives
    kern.disks: twed0 ad0
    twe commands   255    16K     16K      255  64
    hw.twe0.driver_version: 1.50.00.000


Doug White <dwhite_at_gumbysoft.com> wrote:
> What speed of bus is the 3ware attached to?

Unfortunately my system only has a 33MHz 64Bit PCI Slot.  The Adapter
I use supports 66MHz.

> You might run 'systat -vmstat 2' and check the amount of CPU going to
> interrupts. I suspect you're maxing out on the interrupt handler.  Also
> look at the # of interrupts to the twe.

According to ``systat -vmstat 2'' my cpu is used 70-90% by System
(the exact number depends on the amount of free memory), 2% by
Interrupt.  The number of interrupts in twe is approx. 270.  The
twe is serving 570 tps with 128KB/t for a grand total of 70MB/s and
is 75% busy.

I get a total of 500 Interrupts and the cpu is at 8%-30% Idle.  Cpu
usage depends on the ammount of free memory.  In the beginning (free
memory available) cpu is at approx. 30% Idle, after the first few
GByte of transferred data cpu is at 10% Idle.

> What kind of workload are you expecting to put on? Its very, very rare
> that you would be getting large sequential writes like those you are
> getting with dd.

I expect linear writes of 200 GByte at 10 - 80 MByte / sec.

Scott Long comments:
> The interrupt handler for the driver is actually pretty lean; most real
> work is offloaded to a taskqueue.  However, the driver as a whole is not
> locked, so it's going to have contention on the Giant mutex.  He might
> well get some better performance just my making the interrupt handler
> be INTR_MPSAFE.  Unfortunately, this requires a bit of work and I don't
> have any hardware to test on.

Sorry I had no time and no expertise to investigate this.

> Another cheap hack to help performance would be to remove the
> INTR_ENTROPY flag from the call to bus_setup_intr() in twe_freebsd.c.
> This will affect the quality of your entropy a bit, but it would be an
> interesting test.

I will do this today, if I find out what it means :).

> One issue that might be at play here is that the 3ware hardware has some
> limits on DMA alignment for data buffers.  If the buffers don't meet
> those limits, then the driver has to manually allocate and copy
> temporary buffers.  The driver doesn't seem to keep any stats on this,
> but it would be quite useful if it did.

According to twe(4) this is only a problem when userland tries to
access the raw media... I guess this is because kernel buffercache
uses aligned buffers only?

> Other tests to try would include doing I/O directly to the device, and
> turning off softupdates, especially if you're doing lots of file
> operations.

Huh?  I don't understand how this would help with throughput.


Regards, Robert S.