On Wed, Nov 11, 2009 at 5:26 PM, Dan Nelson <dnelson_at_allantgroup.com> wrote: > In the last episode (Nov 11), Sam Fourman Jr. said: >> On Wed, Nov 11, 2009 at 2:49 PM, Dan Nelson <dnelson_at_allantgroup.com> wrote: >> > In the last episode (Nov 11), Ivan Voras said: >> >> Sam Fourman Jr. wrote: >> >> > I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is >> >> > acting weird on writes. I get ~150mbit writes idk if this is good >> >> > or not? but it paused for a few seconds every once and awhile. >> >> >> >> You didn't give any "iostat" statistics - I suspect that if you >> >> correlate ifstat and iostat output that you will see that network >> >> "pauses" happen during spikes in IO. You should check for this and >> >> post your results. >> > >> > Yes, iostat would be useful here. "iostat -zxC 2" will give you >> > per-disk stats plus CPU usage every 2 seconds (CPU may be a factor if >> > you have compression enabled). >> > >> > On a Solaris box I admin, setting zfs_write_limit_override helped >> > stuttering while doing heavy writes. It's not exported on FreeBSD, but >> > it should be easy to add it as a RW sysctl; it lives in dsl_pool.c and >> > can be tweaked at runtime. Start big and tune it down so each write >> > burst takes under a second; it looks like you're writing solid for >> > around 6-8 seconds now. The number will vary depending on your disk >> > speed and how much ARC you have. >> >> here are some iostats for you. I do not believe I have compression enabled >> am I mistaken? isn'y SATA2 300MB/s? and I am doing ~6MB/s per disk? I >> built this machine with 4GB of memory because I thought ZFS would like it. >> now maybe a re(4) interface isnt the best choice. if that is the problem >> here I can change it. We spent ~$800 on a hardware RAID card thinking >> that it would help performance >> >> Why is it that with sftp we do not see the pauses in Network transfer? > > Your service times below are worrying (both for the NFS and sftp cases); > anything above 50ms for a sustained period is usually a problem. You > mention hardware RAID, but I see 6 da# devices; are these just hooked up in > passthrough mode, or is each da device backed by multiple SATA disks? What > kind of write caching do you have enabled on the RAID? Can you disable it? > It sort of looks like zfs is bursting more data to disk than the RAID card > has RAM for, and it's spending multiple seconds just trying to recover. > It's also odd that you're seeing queue depths up to 70 on those disks when > zfs by default should only do 35 (sysctl vfs.zfs.vdev.max_pending). > > NFS has data consistency guarantees that require it to flush to disk more > often than your sftp connection, so that may explain why the disk behaviour > is different. > here is some sysctl output (i believe my max pending is 35) # sysctl vfs.zfs vfs.zfs.arc_meta_limit: 67108864 vfs.zfs.arc_meta_used: 26956528 vfs.zfs.mdcomp_disable: 0 vfs.zfs.arc_min: 33554432 vfs.zfs.arc_max: 268435456 vfs.zfs.zfetch.array_rd_sz: 1048576 vfs.zfs.zfetch.block_cap: 256 vfs.zfs.zfetch.min_sec_reap: 2 vfs.zfs.zfetch.max_streams: 8 vfs.zfs.prefetch_disable: 1 vfs.zfs.recover: 0 vfs.zfs.txg.synctime: 5 vfs.zfs.txg.timeout: 30 vfs.zfs.scrub_limit: 10 vfs.zfs.vdev.cache.bshift: 16 vfs.zfs.vdev.cache.size: 10485760 vfs.zfs.vdev.cache.max: 16384 vfs.zfs.vdev.aggregation_limit: 131072 vfs.zfs.vdev.ramp_rate: 2 vfs.zfs.vdev.time_shift: 6 vfs.zfs.vdev.min_pending: 4 vfs.zfs.vdev.max_pending: 35 vfs.zfs.cache_flush_disable: 0 vfs.zfs.zil_disable: 0 vfs.zfs.version.zpl: 3 vfs.zfs.version.vdev_boot: 1 vfs.zfs.version.spa: 13 vfs.zfs.version.dmu_backup_stream: 1 vfs.zfs.version.dmu_backup_header: 2 vfs.zfs.version.acl: 1 vfs.zfs.debug: 0 vfs.zfs.super_owner: 0 I am using this ARECA ARC-1130 in JBOD Mode We have write cache disabled on the controller. arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable) > mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 22 at device 14.0 on pci4 ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.47 2009-06-25 arcmsr0: [ITHREAD] The motherboard in this machine is Asus M4A785-M http://usa.asus.com/product.aspx?P_ID=ef0qgvMIwOUagAVl&templete=2 if my hardware setup is the trouble, do I just need to start from scratch and select and build a better machine? in the end I was looking to get the fastest writes I can. I am just confused on where I went wrong with this machine. Sam Fourman Jr.Received on Wed Nov 11 2009 - 22:44:52 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:57 UTC