Re: samba+zfs

From: Dan The Man <dan_at_sunsaturn.com>
Date: Tue, 8 Nov 2011 03:26:44 -0600 (CST)
Ok here is some specs: this been running fine on 8.2 stable and i was sure 
it was running fine on RC1 as well. I did some testing against samba 34 35 
and 36 in the ports collection all with the same slow write problems.

I did further testing mounting drive in question with NFS and it did not 
suffer the same problem, so it seems just samba related here, where samba 
would actually outperform my NFS mount before, now its taking 10x as long
to write anything.

This is really most simplistic setup I have, all I want to do is map a 
network drive at the house read/write so my laptop, desktop etc all have 
access to it. I have played with all the smb.conf options, and can't seem
to find where the issue is, further research suggests others are 
experiencing same problems with beta3 from following forum post:

http://forums.freebsd.org/showthread.php?t=27300

Hardware this is running on: I beleive a 4 year old amd chip and board, 
with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD as UFS 
boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap hitachi 
drives for my storage space,which is mirrored nightly with rsync with 
another duplicate machine(cause I know someone is going to say why not use 
raid5-raidz)

Network specs: machine currently has dedicated IPV4 and gif0 tunneled IPV6 
addresses to he.net.

I've played with nearly every option in smb.conf disabling, enabling etc 
and can't seem to find the issue: here are my current config file settings 
on machine that could apply to samba:

asterisk:~# cat /boot/loader.conf
autoboot_delay="5"
accf_data_load="YES"
aio_load="YES"
zfs_load="YES"
kern.maxbcache=64M
kern.ipc.maxpipekva=4M

vfs.zfs.prefetch_disable=1
vm.kmem_size="1844M"
vfs.zfs.arc_min="1024M"
vfs.zfs.arc_max="1536M"
vfs.zfs.vdev.min_pending=2
vfs.zfs.vdev.max_pending=8
vfs.zfs.txg.timeout=5
vfs.zfs.zil_disable="1"
ahci_load="YES"
asterisk:~#

asterisk:~# cat /usr/local/etc/smb.conf
# Global parameters
[global]
         workgroup = HOME
         netbios name = ASTERISK
         server string = "Primary backups"
         interfaces = sk0
         #smb ports = 139
         #security = USER
         security = SHARE
         encrypt passwords = Yes
         #socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536
         domain master = no
         wins support = yes
         guest account = root
         socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY
         use sendfile = no
         level2 oplocks = True
         read raw = no
         write cache size = 262144
         min receivefile size = 16384
         aio read size = 16384
         aio write size = 16384
         aio write behind = yes
         dns proxy = no
         max log size = 50
         #log file = /dev/null
         log file = /var/log/samba.log
         debug level = 1
         syslog = 0

[data]
         comment = "Primary backups"
         path = /data/public
         read only = No
         guest ok = Yes
         valid users = root
asterisk:~# 
asterisk:~# cat /etc/sysctl.conf
# $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith 
Exp $
#
#  This file is read when going to multi-user and its contents piped thru
#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes 
that
# are being run under another UID.
#security.bsd.see_other_uids=0

#raise file descriptors on the system
kern.maxfiles=204916
kern.maxfilesperproc=204916

#raise sockets we can accept
kern.ipc.somaxconn=32768

#http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html
kern.ipc.maxsockbuf=16777216
net.inet.tcp.rfc1323=1
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=131072

#NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations
net.inet.icmp.icmplim=500
kern.ipc.nmbjumbop=192000
kern.ipc.nmbclusters=229376
kern.ipc.maxsockets=204800
net.inet.tcp.maxtcptw=163840
#also add following to /boot/loader.conf
#vm.kmem_size=1844M
#kern.maxbcache=64M
#kern.ipc.maxpipekva=4M

#default setting of net.inet.ip.portrange.first is to low, causing us 
problems with bind
net.inet.ip.portrange.last=65535
net.inet.ip.portrange.first=1024

#DOS protection
net.inet.tcp.msl=7500
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=50
net.inet.ip.accept_sourceroute=0
net.inet.ip.sourceroute=0

#some stuff for samba
kern.ipc.nmbclusters=32768
kern.maxvnodes=800000
net.inet.tcp.delayed_ack=0
net.inet.tcp.inflight.enable=0
net.inet.tcp.path_mtu_discovery=0
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.sendbuf_inc=524288
net.inet.udp.maxdgram=57344
net.inet.udp.recvspace=65535
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
net.inet.tcp.mssdflt=1460

#IPSEC
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules


#NFS--not concerned with data integrity when playing mostly already stored 
movies
vfs.nfsrv.async=1

#JAIL
#i like to use ping etc inside jail!
security.jail.allow_raw_sockets=1
asterisk:~#


Here are logs of me trying to mux a DTS mkv file from samba.log on debug 
level 10, I get the following over and over again:

[2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data)
[2011/11/08 03:24:00.067981,  3] smbd/process.c:1466(switch_message)
   switch message SMBwriteX (pid 64308) conn 0x805008450
[2011/11/08 03:24:00.067990,  4] smbd/uid.c:345(change_to_user)
   Skipping user change - already user
[2011/11/08 03:24:00.068001, 10] 
locking/locking.c:120(strict_lock_default)
   is_locked: optimisation - exclusive oplock on file 
torrent_downloads_finished/Point.Break.1991.720p (1).mkv
[2011/11/08 03:24:00.068010, 10] 
locking/locking.c:162(strict_lock_default)
   strict_lock_default: flavour = WINDOWS_LOCK brl start=83431665 len=65536 
unlocked for fnum 49966 file 
torrent_downloads_finished/Point.Break.1991.720p (1).mkv
[2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile)
   default_sys_recvfile: from = 33, to = 39, offset=83431665, count = 65536
[2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file)
   real_write_file (torrent_downloads_finished/Point.Break.1991.720p 
(1).mkv): pos = 83431665, size = 65536, returned 65536
[2011/11/08 03:24:00.069013,  3] smbd/reply.c:4639(reply_write_and_X)
   writeX fnum=49966 num=65536 wrote=65536
[2011/11/08 03:24:00.069038, 10] 
lib/util_sock.c:516(read_smb_length_return_keepalive)
   got smb length of 65600
[2011/11/08 03:24:00.069052, 10] smbd/reply.c:4459(is_valid_writeX_buffer)
   is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
[2011/11/08 03:24:00.069062,  6] smbd/process.c:1659(process_smb)
   got message type 0x0 of len 0x3f
[2011/11/08 03:24:00.069072,  3] smbd/process.c:1661(process_smb)
   Transaction 15398 of length 67 (65536 toread)
[2011/11/08 03:24:00.069081,  5] lib/util.c:332(show_msg)
[2011/11/08 03:24:00.069087,  5] lib/util.c:342(show_msg)
   size=63
   smb_com=0x2f
   smb_rcls=0
   smb_reh=0
   smb_err=0
   smb_flg=24
   smb_flg2=51207
   smb_tid=1
   smb_pid=65279
   smb_uid=0
   smb_mid=36032
   smt_wct=14
   smb_vwv[ 0]=  255 (0xFF)
   smb_vwv[ 1]=57054 (0xDEDE)
   smb_vwv[ 2]=49966 (0xC32E)
   smb_vwv[ 3]= 4337 (0x10F1)
   smb_vwv[ 4]= 1274 (0x4FA)
   smb_vwv[ 5]=65535 (0xFFFF)
   smb_vwv[ 6]=65535 (0xFFFF)
   smb_vwv[ 7]=    0 (0x0)
   smb_vwv[ 8]=    0 (0x0)
   smb_vwv[ 9]=    1 (0x1)
   smb_vwv[10]=    0 (0x0)
   smb_vwv[11]=   64 (0x40)
   smb_vwv[12]=    0 (0x0)
   smb_vwv[13]=    0 (0x0)
   smb_bcc=0
[2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data)
[2011/11/08 03:24:00.069170,  3] smbd/process.c:1466(switch_message)
   switch message SMBwriteX (pid 64308) conn 0x805008450
[2011/11/08 03:24:00.069179,  4] smbd/uid.c:345(change_to_user)
   Skipping user change - already user
[2011/11/08 03:24:00.069188, 10] 
locking/locking.c:120(strict_lock_default)
   is_locked: optimisation - exclusive oplock on file 
torrent_downloads_finished/Point.Break.1991.720p (1).mkv
[2011/11/08 03:24:00.069197, 10] 
locking/locking.c:162(strict_lock_default)
   strict_lock_default: flavour = WINDOWS_LOCK brl start=83497201 len=65536 
unlocked for fnum 49966 file 
torrent_downloads_finished/Point.Break.1991.720p (1).mkv
[2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile)
   default_sys_recvfile: from = 33, to = 39, offset=83497201, count = 65536
[2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file)
   real_write_file (torrent_downloads_finished/Point.Break.1991.720p 
(1).mkv): pos = 83497201, size = 65536, returned 65536
[2011/11/08 03:24:00.070004,  3] smbd/reply.c:4639(reply_write_and_X)
   writeX fnum=49966 num=65536 wrote=65536
[2011/11/08 03:24:00.070030, 10] 
lib/util_sock.c:516(read_smb_length_return_keepalive)
   got smb length of 65600
[2011/11/08 03:24:00.070044, 10] smbd/reply.c:4459(is_valid_writeX_buffer)
   is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536
[2011/11/08 03:24:00.070053,  6] smbd/process.c:1659(process_smb)
   got message type 0x0 of len 0x3f
[2011/11/08 03:24:00.070063,  3] smbd/process.c:1661(process_smb)
   Transaction 15399 of length 67 (65536 toread)
[2011/11/08 03:24:00.070072,  5] lib/util.c:332(show_msg)
[2011/11/08 03:24:00.070077,  5] lib/util.c:342(show_msg)
   size=63
   smb_com=0x2f
   smb_rcls=0
   smb_reh=0
   smb_err=0
   smb_flg=24
   smb_flg2=51207
   smb_tid=1
   smb_pid=65279
   smb_uid=0
   smb_mid=36102
   smt_wct=14
   smb_vwv[ 0]=  255 (0xFF)
   smb_vwv[ 1]=57054 (0xDEDE)
   smb_vwv[ 2]=49966 (0xC32E)
   smb_vwv[ 3]= 4337 (0x10F1)
   smb_vwv[ 4]= 1275 (0x4FB)
   smb_vwv[ 5]=65535 (0xFFFF)
   smb_vwv[ 6]=65535 (0xFFFF)
   smb_vwv[ 7]=    0 (0x0)
   smb_vwv[ 8]=    0 (0x0)
   smb_vwv[ 9]=    1 (0x1)
   smb_vwv[10]=    0 (0x0)
   smb_vwv[11]=   64 (0x40)
   smb_vwv[12]=    0 (0x0)
   smb_vwv[13]=    0 (0x0)
   smb_bcc=0


Hopefully maybe someone can shine some light on this....


Dan.


--
Dan The Man
CTO/ Senior System Administrator
Websites, Domains and Everything else
http://www.SunSaturn.com
Email: Dan_at_SunSaturn.com

On Fri, 28 Oct 2011, Garrett Cooper wrote:

> On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan_at_sunsaturn.com> wrote:
>>
>>
>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs
>> its taking over an hour to just mux in things like DTS english, where it was
>> 15 minutes on beta3.
>
> Hi Dan,
> - Can you do more deterministic / scientific benchmarks?
> - Did you upgrade Samba?
> - What is your system's operating hardware profile?
> Thanks!
> -Garrett
>
Received on Tue Nov 08 2011 - 08:26:45 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:20 UTC