Re: samba+zfs

On Wed, Nov 9, 2011 at 1:39 PM, Garrett Cooper <yanegomi_at_gmail.com> wrote:
> On Nov 8, 2011, at 11:07 PM, "Daniel O'Connor" <doconnor_at_gsoft.com.au> wrote:
>
>>
>> On 09/11/2011, at 17:32, Garrett Cooper wrote
>>>> dd's of large files (spooled backups going to tape) to /dev/null are as slow as Samba.
>>>
>>>   - Dedupe?
>>
>> Nope.
>>
>>>   - Compression?
>>
>> On the mail spool & ports, but not on the tape spool.
>>
>>>   - How much RAM?
>>
>> 8GB.
>>>   - What debug options do you have enabled in the kernel?
>>
>> It is 8.2-GENERIC so.. no WITNESS (for example)
>
> Ok. 8.2 release or stable?
>
>>>   I've been noticing a slowdown in some respects with NFS/SMB, but I
>>> suspected it was because I have an re(4) based NIC. ZFS has also wired
>>> down a lot of my system memory for the L2ARC…
>>
>>
>> re isn't great but I wouldn't expect it to slow down over time.. Unless bounce buffers got used more and more or something.
>>
>> I have an em0 card in this system - but in any case it is slow locally (i.e. dd a large file with 64k block size).
>
>    Good point. Simple base cases help isolate the root cause. That
> being said, my disk speed(s) are a lot better than my network + samba
> speeds (zfs:store is mfid0 backed with write cache enabled; zfs:sac is
> a single ada(4) backed disk with write cache enabled -- err... it
> shouldn't be like that), but I suspect that's misconfiguration on my
> part:
>
> $ sysctl hw.model hw.physmem
> hw.model: Intel(R) Xeon(R) CPU           W3520  _at_ 2.67GHz
> hw.physmem: 12863094784
> $ sudo mfiutil show volumes
> mfi0 Volumes:
>  Id     Size    Level   Stripe  State   Cache   Name
>  mfid0 ( 1860G) RAID-6      64k OPTIMAL Enabled  <STORE>
> $ zpool status
>  pool: sac
>  state: ONLINE
>  scan: none requested
> config:
>
>        NAME        STATE     READ WRITE CKSUM
>        sac         ONLINE       0     0     0
>          ada0p3    ONLINE       0     0     0
>
> errors: No known data errors
>
>  pool: store
>  state: ONLINE
>  scan: scrub repaired 0 in 10h9m with 0 errors on Mon Nov  7 18:58:01 2011
> config:
>
>        NAME        STATE     READ WRITE CKSUM
>        store       ONLINE       0     0     0
>          mfid0p1   ONLINE       0     0     0
>
> errors: No known data errors
> $ zfs list -o name,mountpoint,atime,sync,compression,dedup
> NAME                         MOUNTPOINT                ATIME      SYNC
>  COMPRESS          DEDUP
> sac                          legacy                       on  standard
>      off            off
> sac/root                     /                            on  standard
>      off            off
> sac/scratch                  /scratch                     on  standard
>      off            off
> sac/scratch/freenas          /scratch/freenas            off  standard
>       on            off
> sac/scratch/freenas/FreeBSD  /scratch/freenas/FreeBSD    off  standard
>       on            off
> sac/usr                      /usr                         on  standard
>      off            off
> sac/var                      /var                         on  standard
>      off            off
> store                        /store                       on  standard
>      off            off
> store/freebsd                /store/freebsd               on  standard
>       on             on
> store/home                   /usr/home                    on  standard
>      off            off
> $ dd if=/dev/zero of=foo bs=1m count=1024
> 1024+0 records out
> 1073741824 bytes transferred in 13.426620 secs (79971119 bytes/sec)
> $ cd /store
> $ dd if=/dev/zero of=foo bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 7.565117 secs (141933276 bytes/sec)
> $ cat /usr/local/etc/smb.conf
> [global]
>        workgroup = WORKGROUP
>        server string = BAYONETTA
>        security = user
>        load printers = no
>        max log size = 50
>        preferred master = yes
>        local master = yes
>        socket options = SO_RCVBUF=16384 SO_SNDBUF=16384
>        nt acl support = yes
>        inherit acls = yes
>        map acl inherit = yes
>        aio read size = 16384
>        aio write size = 16384
>
> [scratch]
>        path = /scratch
>        writeable = yes
>
> [store]
>        path = /store
>        writeable = yes
> $
>
>    I'll have to:
>    1. Recheck what Windows 7 says when transferring out to my box
> with a large file.
>    2. Use nc to quickly measure network performance.
>    3. Try transferring over NFS with both my Macbook and setup
> FreeBSD or Linux on the other workstation for testing out NFS
> transfers (64kB rsize/wsize of course). Wash, rinse, repeat with
> samba.
>    The last I remember the transfer speeds were pitiful with samba36
> (somewhere around 45MBps to my 'store' zpool). I've been conservative
> with the socket settings, but it might be time to bump that up along
> with the mbuf cluster count (for some odd reason I haven't changed it
> from the system default), reboot, and retest.
> Thanks,
> -Garrett
>

I have found the source of my speed issues, and they are not tied to
samba or zfs.  The replacement drive that I put into my array (just
after I updated to stable/9 a few weeks ago) is hitting 100% busy in
gstat while only writing out1-3MB/s -- it's a WDC WD15EARS-00Z5B1 and
some quick googling has demonstrated that there are issues with drives
like it.