ZFS/RAIDZ and SAMBA: abyssimal performance

From: O. Hartmann <ohartman_at_zedat.fu-berlin.de>
Date: Fri, 04 Jan 2013 14:52:17 +0100
I use a small testing server. The hardware is most modern Intel hardware
(i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1
r245036M: Fri Jan  4 12:48:53 CET 2013.

The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives (WDC
WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS RAIDZ:

---
root_at_gate [etc] zpool status
  pool: ASGARD00
 state: ONLINE
  scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec  1 20:59:44 2012
config:

        NAME                                            STATE     READ
WRITE CKSUM
        ASGARD00                                        ONLINE       0
   0     0
          raidz1-0                                      ONLINE       0
   0     0
            gptid/1e716118-1492-11e2-b828-90f6526a24d6  ONLINE       0
   0     0
            gptid/294a6798-1492-11e2-b828-90f6526a24d6  ONLINE       0
   0     0
            gptid/30c813f8-1492-11e2-b828-90f6526a24d6  ONLINE       0
   0     0
        logs
          ada0p1                                        ONLINE       0
   0     0
        cache
          ada0p2                                        ONLINE       0
   0     0

errors: No known data errors
---

The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB
capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB.

I think its not the optimal setup using the very same SSD for both
caching/L2ARC and ZIL, but without the cache device the performance
doen't differ much at the moment. Luckliy, with ZFS I can change the
arrangement as I like.

The ZFS volumes created on the pool named ASGARD00 are standard, only
options sharenfs/sharesmb/checksum are set to yes. Everthing elese is
set to the defaults.

In /boot/loader.conf I set the following parameters according to many
(and confusing!) help and suggestions on the web:

# ZFS
#vfs.zfs.cache_flush_disable=1
#
#vfs.zfs.write_limit_override=1073741824 # 1GB
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_headroom=6

The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same
date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s
sustained, but here are sometimes "spikes" I can watch with "systat -vm
1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives in
RAIDZ). I still benchmark via iozone. Both server and client use JUMBO
frames (MTU=6120), which gives better throughput compared to the
standard MTU=1500.

The local performance on the server itself is slightly better, but
iozone reports some strange numbers. The benchmark "writes" (using 4
threads, 4k blocksizes, writing four times files of size 1G to the ZFS
volume reports sometimes 150 MB/s throughput, and then 70 MB/s and
re-writes is then 1/10 of the "write" throughput and according to the
manual of iozone, re-write is considered to have higher values due to
the lack of writing the meta data again. But I'm still testing this case.

Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I
experience something that is simply described as "abyssimal"
performance! From both a dedicated Windows 7 Pro client and a VirtualBox
4.2.6-client access to folders in a share, say my local home, can take
ages! Opening files takes eons, if possible, in most cases windows
reports "can not open ...". Copying files from Windows to the SAMBA
share doesn't work or take ages, the throughput visible on the server
side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a hiatus
of several seconds.

Well, the SAMBA setup is straightforward, for two weeks now I have
permutated nearly every parameter suggested on all the web's help sites
and I simply took the well configuration from one of our lab's FreeBSD
9.1-STABLE SAMBA servers and changed the local settings for IP and
domain names etc. The working server (FreeBSD 9.1-STABLE) in question
has a single ZFS drive and is exporting this also via NFSv4. It doesn't
have RAIDZ setup!

Before I start benchmarking further with iozone I need to know whether
there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and SAMBA
or whether I'm mislead and have overseen an important setup option.
Before exposing all of my setups here I need to clearify.

I didn't find so far any issues on the web regarding SAMBA, NFSv4 and
ZFS/RAIDZ.

Thanks in advance,
Oliver



Received on Fri Jan 04 2013 - 12:52:33 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:33 UTC