Re: ZFS/RAIDZ and SAMBA: abyssimal performance

On Jan 4, 2013, at 2:52 PM, "O. Hartmann" <ohartman_at_zedat.fu-berlin.de> wrote:

> I use a small testing server. The hardware is most modern Intel hardware
> (i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1
> r245036M: Fri Jan  4 12:48:53 CET 2013.
> 
> The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives (WDC
> WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS RAIDZ:
> 
> ---
> root_at_gate [etc] zpool status
>  pool: ASGARD00
> state: ONLINE
>  scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec  1 20:59:44 2012
> config:
> 
>        NAME                                            STATE     READ
> WRITE CKSUM
>        ASGARD00                                        ONLINE       0
>   0     0
>          raidz1-0                                      ONLINE       0
>   0     0
>            gptid/1e716118-1492-11e2-b828-90f6526a24d6  ONLINE       0
>   0     0
>            gptid/294a6798-1492-11e2-b828-90f6526a24d6  ONLINE       0
>   0     0
>            gptid/30c813f8-1492-11e2-b828-90f6526a24d6  ONLINE       0
>   0     0
>        logs
>          ada0p1                                        ONLINE       0
>   0     0
>        cache
>          ada0p2                                        ONLINE       0
>   0     0
> 
> errors: No known data errors
> ---
> 
> The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB
> capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB.
> 
> I think its not the optimal setup using the very same SSD for both
> caching/L2ARC and ZIL, but without the cache device the performance
> doen't differ much at the moment. Luckliy, with ZFS I can change the
> arrangement as I like.
> 
> The ZFS volumes created on the pool named ASGARD00 are standard, only
> options sharenfs/sharesmb/checksum are set to yes. Everthing elese is
> set to the defaults.
> 
> In /boot/loader.conf I set the following parameters according to many
> (and confusing!) help and suggestions on the web:
> 
> # ZFS
> #vfs.zfs.cache_flush_disable=1
> #
> #vfs.zfs.write_limit_override=1073741824 # 1GB
> vfs.zfs.l2arc_noprefetch=0
> vfs.zfs.l2arc_headroom=6
> 
> The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same
> date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s
> sustained, but here are sometimes "spikes" I can watch with "systat -vm
> 1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives in
> RAIDZ). I still benchmark via iozone. Both server and client use JUMBO
> frames (MTU=6120), which gives better throughput compared to the
> standard MTU=1500.
> 
> The local performance on the server itself is slightly better, but
> iozone reports some strange numbers. The benchmark "writes" (using 4
> threads, 4k blocksizes, writing four times files of size 1G to the ZFS
> volume reports sometimes 150 MB/s throughput, and then 70 MB/s and
> re-writes is then 1/10 of the "write" throughput and according to the
> manual of iozone, re-write is considered to have higher values due to
> the lack of writing the meta data again. But I'm still testing this case.
> 
> Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I
> experience something that is simply described as "abyssimal"
> performance! From both a dedicated Windows 7 Pro client and a VirtualBox
> 4.2.6-client access to folders in a share, say my local home, can take
> ages! Opening files takes eons, if possible, in most cases windows
> reports "can not open ...". Copying files from Windows to the SAMBA
> share doesn't work or take ages, the throughput visible on the server
> side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a hiatus
> of several seconds.
> 
> Well, the SAMBA setup is straightforward, for two weeks now I have
> permutated nearly every parameter suggested on all the web's help sites
> and I simply took the well configuration from one of our lab's FreeBSD
> 9.1-STABLE SAMBA servers and changed the local settings for IP and
> domain names etc. The working server (FreeBSD 9.1-STABLE) in question
> has a single ZFS drive and is exporting this also via NFSv4. It doesn't
> have RAIDZ setup!
> 
> Before I start benchmarking further with iozone I need to know whether
> there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and SAMBA
> or whether I'm mislead and have overseen an important setup option.
> Before exposing all of my setups here I need to clearify.
> 
> I didn't find so far any issues on the web regarding SAMBA, NFSv4 and
> ZFS/RAIDZ.
> 
> Thanks in advance,
> Oliver
> 
> 



I experienced the same performance problem, then followed Jeremy Chadwick's advice and tuned variables a bit, I'm getting excellent performance now.


/boot/loader.conf

# Tune ZFS somewhat aye ?
vm.kmem_size="3072M"
vfs.zfs.arc_min="128M"
vfs.zfs.arc_max="2048M"

# Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout="5"



And network cards:
# Up a bit our intel cards parameters
hw.em.txd=4096
hw.em.rxd=4096
hw.em.tx_int_delay=512
hw.em.rx_int_delay=512
hw.em.tx_abs_int_delay=1024
hw.em.rx_abs_int_delay=1024



And this is under [global] in /usr/local/etc/smb.conf:
   min receivefile size = 16384
   aio read size = 16384
   aio write size = 16384
   aio write behind = yes