Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i

On Wed, 22 Jun 2016 08:58:08 +0200
Borja Marcos <borjam_at_sarenet.es> wrote:

> > On 22 Jun 2016, at 04:08, Jason Zhang <jasonzhang_at_cyphytech.com> wrote:
> > 
> > Mark,
> > 
> > Thanks
> > 
> > We have same RAID setting both on FreeBSD and CentOS including cache
> > setting.  In FreeBSD, I enabled the write cache but the performance is the
> > same.  
> > 
> > We don’t use ZFS or UFS, and test the performance on the RAW GEOM disk
> > “mfidx” exported by mfi driver.  We observed the “gstat” result and found
> > that the write latency is too high.  When we “dd" the disk with 8k, it is
> > lower than 1ms, but it is 6ms on 64kb write.  It seems that each single
> > write operation is very slow. But I don’t know whether it is a driver
> > problem or not.  
> 
> There is an option you can use (I do it all the time!) to make the card
> behave as a plain HBA so that the disks are handled by the “da” driver. 
> 
> Add this to /boot/loader.conf
> 
> hw.mfi.allow_cam_disk_passthrough=1
> mfip_load=“YES"
> 
> And do the tests accessing the disks as “da”. To avoid confusions, it’s
> better to make sure the disks are not part of a “jbod” or logical volume
> configuration.
> 
> 
> 
> 
> Borja.
[...]

How is this supposed to work when ALL disks (including boot device) are settled
with the mfi (in our case, it is a Fujitsu CP400i, based upon LSI3008 and
detected within FreeBSD 11-BETA and 12-CURRENT) controller itself?

I did not find any solution to force the CP400i into a mode making itself
acting as a HBA (we intend to use all drives with ZFS and let FreeBSD
kernel/ZFS control everything).

The boot device is a 256 GB Samsung SSD for enterprise use and putting the UEFI
load onto a EFI partition from 11-CURRENT-ALPHA4 is worse: dd takes up to
almost a minute to put the image onto the SSD. The SSD active LED is blinking
alle the time indicating activity. Caches are off. I tried to enable the cache
via the mfiutil command by 'mfiutil cache mfid0 enable', but it failed ... It
failed also on all other attached drives.

I didn't further go into more investigations right now, since the experience
with the EFI boot loader makes me suspect bad performance and that is harsh so
to speak. Glad to have found this thread anyway.

I cross post this also to CURRENT as it might be an issue with CURRENT ...

Kind regards,

Oliver Hartmann