Re: low(er) disk performance with sched_4bsd then with sched_ule

Oliver Lehmann wrote:

> Hi,
> 
> I've a dual-cpu system (SMP) which runs with 2 PIII-850 MHz on an Intel
> Serverboard (440GX chipset). My diskspace is provided by a RAID-5
> containing 4 WD2500KS harddrives connected to a 3ware 9500S-4LP
> controller which runs at 32bit/33MHz.
> The system itself runs FreeBSD 6.0-BETA2.
> 
> with SCHED_4BSD in the kernel:
> 
> root_at_nudel olivleh1> dd if=/dev/zero of=/mnt/files/test.dd bs=64k count=32000
> 32000+0 records in
> 32000+0 records out
> 2097152000 bytes transferred in 44.136711 secs (47514913 bytes/sec)
> 
> with SCHED_ULE in the kernel:
> 
> root_at_nudel olivleh1> dd if=/dev/zero of=/mnt/files/test.dd bs=64k count=32000
> 32000+0 records in
> 32000+0 records out
> 2097152000 bytes transferred in 27.005334 secs (77656954 bytes/sec)
> 
> the scheduler is the only difference between both configs - everything
> else is the same. Kernel debugging is disabled, malloc.conf exists.
> 
> Filesystem                    Size    Used   Avail Capacity  Mounted on
> /dev/da0s1a                   496M    159M    297M    35%    /
> devfs                         1.0K    1.0K      0B   100%    /dev
> /dev/da0s1d                   496M     20M    436M     4%    /var
> /dev/da0s1e                   496M     22M    434M     5%    /tmp
> /dev/da0s1f                    19G    2.9G     15G    16%    /usr
> /dev/da0s1g                   671G    216G    455G    32%    /mnt/files
> 
> 
> 3ware device driver for 9000 series storage controllers, version: 3.60.00.017
> twa0: <3ware 9000 series Storage Controller> port 0x3400-0x34ff mem 0xf4102800-0xf41028ff,0xf4800000-0xf4ffffff irq 16 at device 16.0 on pci0
> twa0: [FAST]
> twa0: INFO: (0x15: 0x1300): Controller details:: 4 ports, Firmware FE9X 2.08.00.005, BIOS BE9X 2.03.01.052
> 
> da0 at twa0 bus 0 target 0 lun 0
> da0: <AMCC 9500S-4LP  DISK 2.08> Fixed Direct Access SCSI-3 device 
> da0: 100.000MB/s transfers
> da0: 715224MB (1464778752 512 byte sectors: 255H 63S/T 91178C)
> 
> 
> Any idea where the 30MB/sec drawback comes from and if I missed sth.?
> I mean why there are 30MB/s more or less is worth to think about imho.
> 

Indeed.  This definitely warrants much more testing and investigation. 
My best guess is that ULE is allowing the uio copies of the data between
kernel and userland to complete with fewer interruptions.  It could also
be better about keeping the threads from ping-ponging between CPUs.  Can
you retest with different block sizes, ranging from 4k to 1m?

Scott