Re: samba+zfs

On Nov 8, 2011, at 11:07 PM, "Daniel O'Connor" <doconnor_at_gsoft.com.au> wrote:

>
> On 09/11/2011, at 17:32, Garrett Cooper wrote
>>> dd's of large files (spooled backups going to tape) to /dev/null are as slow as Samba.
>>
>>   - Dedupe?
>
> Nope.
>
>>   - Compression?
>
> On the mail spool & ports, but not on the tape spool.
>
>>   - How much RAM?
>
> 8GB.
>>   - What debug options do you have enabled in the kernel?
>
> It is 8.2-GENERIC so.. no WITNESS (for example)

Ok. 8.2 release or stable?

>>   I've been noticing a slowdown in some respects with NFS/SMB, but I
>> suspected it was because I have an re(4) based NIC. ZFS has also wired
>> down a lot of my system memory for the L2ARC…
>
>
> re isn't great but I wouldn't expect it to slow down over time.. Unless bounce buffers got used more and more or something.
>
> I have an em0 card in this system - but in any case it is slow locally (i.e. dd a large file with 64k block size).

    Good point. Simple base cases help isolate the root cause. That
being said, my disk speed(s) are a lot better than my network + samba
speeds (zfs:store is mfid0 backed with write cache enabled; zfs:sac is
a single ada(4) backed disk with write cache enabled -- err... it
shouldn't be like that), but I suspect that's misconfiguration on my
part:

$ sysctl hw.model hw.physmem
hw.model: Intel(R) Xeon(R) CPU           W3520  _at_ 2.67GHz
hw.physmem: 12863094784
$ sudo mfiutil show volumes
mfi0 Volumes:
  Id     Size    Level   Stripe  State   Cache   Name
 mfid0 ( 1860G) RAID-6      64k OPTIMAL Enabled  <STORE>
$ zpool status
  pool: sac
 state: ONLINE
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        sac         ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors

  pool: store
 state: ONLINE
 scan: scrub repaired 0 in 10h9m with 0 errors on Mon Nov  7 18:58:01 2011
config:

        NAME        STATE     READ WRITE CKSUM
        store       ONLINE       0     0     0
          mfid0p1   ONLINE       0     0     0

errors: No known data errors
$ zfs list -o name,mountpoint,atime,sync,compression,dedup
NAME                         MOUNTPOINT                ATIME      SYNC
 COMPRESS          DEDUP
sac                          legacy                       on  standard
      off            off
sac/root                     /                            on  standard
      off            off
sac/scratch                  /scratch                     on  standard
      off            off
sac/scratch/freenas          /scratch/freenas            off  standard
       on            off
sac/scratch/freenas/FreeBSD  /scratch/freenas/FreeBSD    off  standard
       on            off
sac/usr                      /usr                         on  standard
      off            off
sac/var                      /var                         on  standard
      off            off
store                        /store                       on  standard
      off            off
store/freebsd                /store/freebsd               on  standard
       on             on
store/home                   /usr/home                    on  standard
      off            off
$ dd if=/dev/zero of=foo bs=1m count=1024
1024+0 records out
1073741824 bytes transferred in 13.426620 secs (79971119 bytes/sec)
$ cd /store
$ dd if=/dev/zero of=foo bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 7.565117 secs (141933276 bytes/sec)
$ cat /usr/local/etc/smb.conf
[global]
        workgroup = WORKGROUP
        server string = BAYONETTA
        security = user
        load printers = no
        max log size = 50
        preferred master = yes
        local master = yes
        socket options = SO_RCVBUF=16384 SO_SNDBUF=16384
        nt acl support = yes
        inherit acls = yes
        map acl inherit = yes
        aio read size = 16384
        aio write size = 16384

[scratch]
        path = /scratch
        writeable = yes

[store]
        path = /store
        writeable = yes
$

    I'll have to:
    1. Recheck what Windows 7 says when transferring out to my box
with a large file.
    2. Use nc to quickly measure network performance.
    3. Try transferring over NFS with both my Macbook and setup
FreeBSD or Linux on the other workstation for testing out NFS
transfers (64kB rsize/wsize of course). Wash, rinse, repeat with
samba.
    The last I remember the transfer speeds were pitiful with samba36
(somewhere around 45MBps to my 'store' zpool). I've been conservative
with the socket settings, but it might be time to bump that up along
with the mbuf cluster count (for some odd reason I haven't changed it
from the system default), reboot, and retest.
Thanks,
-Garrett