Re: samba+zfs

From: Kurt Touet <ktouet_at_gmail.com>
Date: Tue, 8 Nov 2011 16:28:56 -0600
On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man <dan_at_sunsaturn.com> wrote:
>
>
> Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1:
> asterisk:~# uname -a
> FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31
> 19:46:53 CDT 2011 droot_at_asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKERNEL
>  amd64
> asterisk:~#
>
>
>
> Dan.
>
>
> --
> Dan The Man
> CTO/ Senior System Administrator
> Websites, Domains and Everything else
> http://www.SunSaturn.com
> Email: Dan_at_SunSaturn.com
>
> On Tue, 8 Nov 2011, Dan The Man wrote:
>
>>
>> Ok here is some specs: this been running fine on 8.2 stable and i was sure
>> it was running fine on RC1 as well. I did some testing against samba 34 35
>> and 36 in the ports collection all with the same slow write problems.
>>
>> I did further testing mounting drive in question with NFS and it did not
>> suffer the same problem, so it seems just samba related here, where samba
>> would actually outperform my NFS mount before, now its taking 10x as long
>> to write anything.
>>
>> This is really most simplistic setup I have, all I want to do is map a
>> network drive at the house read/write so my laptop, desktop etc all have
>> access to it. I have played with all the smb.conf options, and can't seem
>> to find where the issue is, further research suggests others are
>> experiencing same problems with beta3 from following forum post:
>>
>> http://forums.freebsd.org/showthread.php?t=27300
>>
>> Hardware this is running on: I beleive a 4 year old amd chip and board,
>> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD as
>> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap
>> hitachi drives for my storage space,which is mirrored nightly with rsync
>> with another duplicate machine(cause I know someone is going to say why not
>> use raid5-raidz)
>>
>> Network specs: machine currently has dedicated IPV4 and gif0 tunneled IPV6
>> addresses to he.net.
>>
>> I've played with nearly every option in smb.conf disabling, enabling etc
>> and can't seem to find the issue: here are my current config file settings
>> on machine that could apply to samba:
>>
>> asterisk:~# cat /boot/loader.conf
>> autoboot_delay="5"
>> accf_data_load="YES"
>> aio_load="YES"
>> zfs_load="YES"
>> kern.maxbcache=64M
>> kern.ipc.maxpipekva=4M
>>
>> vfs.zfs.prefetch_disable=1
>> vm.kmem_size="1844M"
>> vfs.zfs.arc_min="1024M"
>> vfs.zfs.arc_max="1536M"
>> vfs.zfs.vdev.min_pending=2
>> vfs.zfs.vdev.max_pending=8
>> vfs.zfs.txg.timeout=5
>> vfs.zfs.zil_disable="1"
>> ahci_load="YES"
>> asterisk:~#
>>
>> asterisk:~# cat /usr/local/etc/smb.conf
>> # Global parameters
>> [global]
>>       workgroup = HOME
>>       netbios name = ASTERISK
>>       server string = "Primary backups"
>>       interfaces = sk0
>>       #smb ports = 139
>>       #security = USER
>>       security = SHARE
>>       encrypt passwords = Yes
>>       #socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536
>>       domain master = no
>>       wins support = yes
>>       guest account = root
>>       socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
>>       use sendfile = no
>>       level2 oplocks = True
>>       read raw = no
>>       write cache size = 262144
>>       min receivefile size = 16384
>>       aio read size = 16384
>>       aio write size = 16384
>>       aio write behind = yes
>>       dns proxy = no
>>       max log size = 50
>>       #log file = /dev/null
>>       log file = /var/log/samba.log
>>       debug level = 1
>>       syslog = 0
>>
>> [data]
>>       comment = "Primary backups"
>>       path = /data/public
>>       read only = No
>>       guest ok = Yes
>>       valid users = root
>> asterisk:~# asterisk:~# cat /etc/sysctl.conf
>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith
>> Exp $
>> #
>> #  This file is read when going to multi-user and its contents piped thru
>> #  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
>> #
>>
>> # Uncomment this to prevent users from seeing information about processes
>> that
>> # are being run under another UID.
>> #security.bsd.see_other_uids=0
>>
>> #raise file descriptors on the system
>> kern.maxfiles=204916
>> kern.maxfilesperproc=204916
>>
>> #raise sockets we can accept
>> kern.ipc.somaxconn=32768
>>
>> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html
>> kern.ipc.maxsockbuf=16777216
>> net.inet.tcp.rfc1323=1
>> net.inet.tcp.sendbuf_max=16777216
>> net.inet.tcp.recvbuf_max=16777216
>> net.inet.tcp.sendspace=65536
>> net.inet.tcp.recvspace=131072
>>
>> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations
>> net.inet.icmp.icmplim=500
>> kern.ipc.nmbjumbop=192000
>> kern.ipc.nmbclusters=229376
>> kern.ipc.maxsockets=204800
>> net.inet.tcp.maxtcptw=163840
>> #also add following to /boot/loader.conf
>> #vm.kmem_size=1844M
>> #kern.maxbcache=64M
>> #kern.ipc.maxpipekva=4M
>>
>> #default setting of net.inet.ip.portrange.first is to low, causing us
>> problems with bind
>> net.inet.ip.portrange.last=65535
>> net.inet.ip.portrange.first=1024
>>
>> #DOS protection
>> net.inet.tcp.msl=7500
>> net.inet.tcp.blackhole=2
>> net.inet.udp.blackhole=1
>> net.inet.icmp.icmplim=50
>> net.inet.ip.accept_sourceroute=0
>> net.inet.ip.sourceroute=0
>>
>> #some stuff for samba
>> kern.ipc.nmbclusters=32768
>> kern.maxvnodes=800000
>> net.inet.tcp.delayed_ack=0
>> net.inet.tcp.inflight.enable=0
>> net.inet.tcp.path_mtu_discovery=0
>> net.inet.tcp.recvbuf_auto=1
>> net.inet.tcp.recvbuf_inc=524288
>> net.inet.tcp.sendbuf_auto=1
>> net.inet.tcp.sendbuf_inc=524288
>> net.inet.udp.maxdgram=57344
>> net.inet.udp.recvspace=65535
>> net.local.stream.recvspace=65535
>> net.local.stream.sendspace=65535
>> net.inet.tcp.mssdflt=1460
>>
>> #IPSEC
>> net.inet.ip.forwarding=1
>> net.inet6.ip6.forwarding=1
>> kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules
>>
>>
>> #NFS--not concerned with data integrity when playing mostly already stored
>> movies
>> vfs.nfsrv.async=1
>>
>> #JAIL
>> #i like to use ping etc inside jail!
>> security.jail.allow_raw_sockets=1
>> asterisk:~#
>>
>>
>> Here are logs of me trying to mux a DTS mkv file from samba.log on debug
>> level 10, I get the following over and over again:
>>
>> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data)
>> [2011/11/08 03:24:00.067981,  3] smbd/process.c:1466(switch_message)
>>  switch message SMBwriteX (pid 64308) conn 0x805008450
>> [2011/11/08 03:24:00.067990,  4] smbd/uid.c:345(change_to_user)
>>  Skipping user change - already user
>> [2011/11/08 03:24:00.068001, 10]
>> locking/locking.c:120(strict_lock_default)
>>  is_locked: optimisation - exclusive oplock on file
>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>> [2011/11/08 03:24:00.068010, 10]
>> locking/locking.c:162(strict_lock_default)
>>  strict_lock_default: flavour = WINDOWS_LOCK brl start=83431665 len=65536
>> unlocked for fnum 49966 file
>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile)
>>  default_sys_recvfile: from = 33, to = 39, offset=83431665, count = 65536
>> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file)
>>  real_write_file (torrent_downloads_finished/Point.Break.1991.720p
>> (1).mkv): pos = 83431665, size = 65536, returned 65536
>> [2011/11/08 03:24:00.069013,  3] smbd/reply.c:4639(reply_write_and_X)
>>  writeX fnum=49966 num=65536 wrote=65536
>> [2011/11/08 03:24:00.069038, 10]
>> lib/util_sock.c:516(read_smb_length_return_keepalive)
>>  got smb length of 65600
>> [2011/11/08 03:24:00.069052, 10] smbd/reply.c:4459(is_valid_writeX_buffer)
>>  is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
>> [2011/11/08 03:24:00.069062,  6] smbd/process.c:1659(process_smb)
>>  got message type 0x0 of len 0x3f
>> [2011/11/08 03:24:00.069072,  3] smbd/process.c:1661(process_smb)
>>  Transaction 15398 of length 67 (65536 toread)
>> [2011/11/08 03:24:00.069081,  5] lib/util.c:332(show_msg)
>> [2011/11/08 03:24:00.069087,  5] lib/util.c:342(show_msg)
>>  size=63
>>  smb_com=0x2f
>>  smb_rcls=0
>>  smb_reh=0
>>  smb_err=0
>>  smb_flg=24
>>  smb_flg2=51207
>>  smb_tid=1
>>  smb_pid=65279
>>  smb_uid=0
>>  smb_mid=36032
>>  smt_wct=14
>>  smb_vwv[ 0]=  255 (0xFF)
>>  smb_vwv[ 1]=57054 (0xDEDE)
>>  smb_vwv[ 2]=49966 (0xC32E)
>>  smb_vwv[ 3]= 4337 (0x10F1)
>>  smb_vwv[ 4]= 1274 (0x4FA)
>>  smb_vwv[ 5]=65535 (0xFFFF)
>>  smb_vwv[ 6]=65535 (0xFFFF)
>>  smb_vwv[ 7]=    0 (0x0)
>>  smb_vwv[ 8]=    0 (0x0)
>>  smb_vwv[ 9]=    1 (0x1)
>>  smb_vwv[10]=    0 (0x0)
>>  smb_vwv[11]=   64 (0x40)
>>  smb_vwv[12]=    0 (0x0)
>>  smb_vwv[13]=    0 (0x0)
>>  smb_bcc=0
>> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data)
>> [2011/11/08 03:24:00.069170,  3] smbd/process.c:1466(switch_message)
>>  switch message SMBwriteX (pid 64308) conn 0x805008450
>> [2011/11/08 03:24:00.069179,  4] smbd/uid.c:345(change_to_user)
>>  Skipping user change - already user
>> [2011/11/08 03:24:00.069188, 10]
>> locking/locking.c:120(strict_lock_default)
>>  is_locked: optimisation - exclusive oplock on file
>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>> [2011/11/08 03:24:00.069197, 10]
>> locking/locking.c:162(strict_lock_default)
>>  strict_lock_default: flavour = WINDOWS_LOCK brl start=83497201 len=65536
>> unlocked for fnum 49966 file
>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv
>> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile)
>>  default_sys_recvfile: from = 33, to = 39, offset=83497201, count = 65536
>> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file)
>>  real_write_file (torrent_downloads_finished/Point.Break.1991.720p
>> (1).mkv): pos = 83497201, size = 65536, returned 65536
>> [2011/11/08 03:24:00.070004,  3] smbd/reply.c:4639(reply_write_and_X)
>>  writeX fnum=49966 num=65536 wrote=65536
>> [2011/11/08 03:24:00.070030, 10]
>> lib/util_sock.c:516(read_smb_length_return_keepalive)
>>  got smb length of 65600
>> [2011/11/08 03:24:00.070044, 10] smbd/reply.c:4459(is_valid_writeX_buffer)
>>  is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
>> [2011/11/08 03:24:00.070053,  6] smbd/process.c:1659(process_smb)
>>  got message type 0x0 of len 0x3f
>> [2011/11/08 03:24:00.070063,  3] smbd/process.c:1661(process_smb)
>>  Transaction 15399 of length 67 (65536 toread)
>> [2011/11/08 03:24:00.070072,  5] lib/util.c:332(show_msg)
>> [2011/11/08 03:24:00.070077,  5] lib/util.c:342(show_msg)
>>  size=63
>>  smb_com=0x2f
>>  smb_rcls=0
>>  smb_reh=0
>>  smb_err=0
>>  smb_flg=24
>>  smb_flg2=51207
>>  smb_tid=1
>>  smb_pid=65279
>>  smb_uid=0
>>  smb_mid=36102
>>  smt_wct=14
>>  smb_vwv[ 0]=  255 (0xFF)
>>  smb_vwv[ 1]=57054 (0xDEDE)
>>  smb_vwv[ 2]=49966 (0xC32E)
>>  smb_vwv[ 3]= 4337 (0x10F1)
>>  smb_vwv[ 4]= 1275 (0x4FB)
>>  smb_vwv[ 5]=65535 (0xFFFF)
>>  smb_vwv[ 6]=65535 (0xFFFF)
>>  smb_vwv[ 7]=    0 (0x0)
>>  smb_vwv[ 8]=    0 (0x0)
>>  smb_vwv[ 9]=    1 (0x1)
>>  smb_vwv[10]=    0 (0x0)
>>  smb_vwv[11]=   64 (0x40)
>>  smb_vwv[12]=    0 (0x0)
>>  smb_vwv[13]=    0 (0x0)
>>  smb_bcc=0
>>
>>
>> Hopefully maybe someone can shine some light on this....
>>
>>
>> Dan.
>>
>>
>> --
>> Dan The Man
>> CTO/ Senior System Administrator
>> Websites, Domains and Everything else
>> http://www.SunSaturn.com
>> Email: Dan_at_SunSaturn.com
>>
>> On Fri, 28 Oct 2011, Garrett Cooper wrote:
>>
>>> On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan_at_sunsaturn.com> wrote:
>>>>
>>>>
>>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs
>>>> its taking over an hour to just mux in things like DTS english, where it
>>>> was
>>>> 15 minutes on beta3.
>>>
>>> Hi Dan,
>>> - Can you do more deterministic / scientific benchmarks?
>>> - Did you upgrade Samba?
>>> - What is your system's operating hardware profile?
>>> Thanks!
>>> -Garrett
>>>

I had noticed a similar problem with doing large writes from my
windows workstation to my freebsd ZFS array.  Previously I had been
able to write at around 30MB/s (limited by a slow SATA controller),
and those speeds had dropped to 3-5MB/s with long pauses between
writes to the array (monitoring with zpool iostat).   However, I just
updated to stable/9 r227357 and the issue seems to have gone away; I'm
back up to 30MB/s writes.

Hope you find the same sort of thing,
-kurt
Received on Tue Nov 08 2011 - 21:53:36 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:20 UTC