Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking

On 04.09.2013 11:20, Johan Hendriks wrote:
> Alexander Motin wrote:
>> I would like to invite more people to review and test my patches for
>> improving CAM and GEOM scalability, that for last six months you could
>> see developing in project/camlock SVN branch. Full diff of that branch
>> against present head (r255131) can be found here:
>> http://people.freebsd.org/~mav/camlock_patches/camlock_20130902.patch
>>
>> Heavy CAM changes there were focused on reducing scope of SIM lock to
>> only protecting SIM internals, but not CAM core. That allows many
>> times reduce lock congestion, especially on heavily parallel request
>> submission with GEOM changes below. More detailed description of
>> changes you could see here earlier:
>> http://docs.freebsd.org/cgi/mid.cgi?520D4ADB.50209
>>
>> GEOM changes were focused on avoiding switching to GEOM up/down
>> threads in relatively simple setups where respective classes don't
>> require it (and were explicitly marked so). That allows save on
>> context switches and on systems with several HBAs and disks talk to
>> them concurrently (that is where CAM locking changes are handy). Such
>> classes were modified to support it: DEV, DISK, LABEL, MULTIPATH, NOP,
>> PART, RAID (partially), STRIPE, ZERO, VFS, ZFS::VDEV, ZFS::ZVOL and
>> some others. Requests to/from other classes will be queued to GEOM
>> threads same as before.
>>
>> Together that allows to double block subsystem performance on high (at
>> least 100-200K) IOPS benchmarks, allowing to reach up to a million
>> total IOPS, while keeping full compatibility with all major ABIs/KBIs.
>>
>> Since we are already in 10.0 release process and changes are quite
>> big, my plan is to wait and commit them to head branch after the
>> freeze end, and then merge to stable/10.  I hope the release process
>> will go on schedule to not delay this work for another six months.
>>
>> This work is sponsored by iXsystems, Inc.
>>
> Hello i would like to test this patch set.
> But how can i stress the machine, do you have a script or something i
> can use to make the system do heavy I/O on the disks!

For testing IOPS over RAW disks or different GEOM providers (to exclude 
FS influence) I am using such a trivial synthetic test:

#!/bin/sh

for disk in da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 
da14 da15
do
     for i in `jot 16`
     do
         dd if=/dev/$disk of=/dev/null bs=512 &
     done
done

iostat -zxw3 -c12 | grep total |tail -n 10 |cut -c 6-18

killall dd

For total IOPS measurement in above script I am using dirtily hacked 
iostat tool that prints summary values in addition to per-disk ones:
http://people.freebsd.org/~mav/camlock_patches/iostat.patch

BTW I've uploaded new patch with some more minor CAM changes:
http://people.freebsd.org/~mav/camlock_patches/camlock_20130904.patch

-- 
Alexander Motin