ZFS L2ARC - incorrect size and abnormal system load on r255173

From: Dmitriy Makarov <supportme_at_ukr.net>
Date: Mon, 07 Oct 2013 21:09:06 +0300

('binary' encoding is not supported, stored as-is) Hi all, On our production system on r255173 we have problem with abnormal high system load caused (not sure) with L2ARC placed on a few SSD, 490 GB total size. After a fresh boot everything seems to be fine, Load Average less then 5.00. But after some time (nearly day-two) Load Average jump to 10.. 20+ and system goes really slow, IO opearations with zfs pool grows from ms to even seconds. And L2ARC sysctls are pretty disturbing: [frv:~]$ sysctl -a | grep l2 vfs.zfs.l2arc_write_max: 25000000 vfs.zfs.l2arc_write_boost: 50000000 vfs.zfs.l2arc_headroom: 8 vfs.zfs.l2arc_feed_secs: 1 vfs.zfs.l2arc_feed_min_ms: 30 vfs.zfs.l2arc_noprefetch: 0 vfs.zfs.l2arc_feed_again: 1 vfs.zfs.l2arc_norw: 1 vfs.zfs.l2c_only_size: 1525206040064 vfs.cache.numfullpathfail2: 4 kstat.zfs.misc.arcstats.evict_l2_cached: 6592742547456 kstat.zfs.misc.arcstats.evict_l2_eligible: 734016778752 kstat.zfs.misc.arcstats.evict_l2_ineligible: 29462561417216 kstat.zfs.misc.arcstats.l2_hits: 576550808 kstat.zfs.misc.arcstats.l2_misses: 128158998 kstat.zfs.misc.arcstats.l2_feeds: 1524059 kstat.zfs.misc.arcstats.l2_rw_clash: 1429740 kstat.zfs.misc.arcstats.l2_read_bytes: 2896069043200 kstat.zfs.misc.arcstats.l2_write_bytes: 2405022640128 kstat.zfs.misc.arcstats.l2_writes_sent: 826642 kstat.zfs.misc.arcstats.l2_writes_done: 826642 kstat.zfs.misc.arcstats.l2_writes_error: 0 kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 1059415 kstat.zfs.misc.arcstats.l2_evict_lock_retry: 1640 kstat.zfs.misc.arcstats.l2_evict_reading: 0 kstat.zfs.misc.arcstats.l2_free_on_write: 8580680 kstat.zfs.misc.arcstats.l2_abort_lowmem: 2096 kstat.zfs.misc.arcstats.l2_cksum_bad: 212832715 kstat.zfs.misc.arcstats.l2_io_error: 5501886 kstat.zfs.misc.arcstats.l2_size: 1587962307584 kstat.zfs.misc.arcstats.l2_asize: 1425666718720 kstat.zfs.misc.arcstats.l2_hdr_size: 82346948208 kstat.zfs.misc.arcstats.l2_compress_successes: 41707766 kstat.zfs.misc.arcstats.l2_compress_zeros: 0 kstat.zfs.misc.arcstats.l2_compress_failures: 0 kstat.zfs.misc.arcstats.l2_write_trylock_fail: 8847701930 kstat.zfs.misc.arcstats.l2_write_passed_headroom: 21220076 kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 27619372107 kstat.zfs.misc.arcstats.l2_write_in_l2: 418007172085 kstat.zfs.misc.arcstats.l2_write_io_in_progress: 29279 kstat.zfs.misc.arcstats.l2_write_not_cacheable: 131001473113 kstat.zfs.misc.arcstats.l2_write_full: 63699 kstat.zfs.misc.arcstats.l2_write_buffer_iter: 1524059 kstat.zfs.misc.arcstats.l2_write_pios: 826642 kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 8433038008130560 kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 96529899 kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 9228464 Here is output from zfs-stats about L2: [frv:~]$ zfs-stats -L ------------------------------------------------------------------------ ZFS Subsystem Report Mon Oct 7 20:50:19 2013 ------------------------------------------------------------------------ L2 ARC Summary: (DEGRADED) Passed Headroom: 21.22m Tried Lock Failures: 8.85b IO In Progress: 29.32k Low Memory Aborts: 2.10k Free on Write: 8.59m Writes While Full: 63.71k R/W Clashes: 1.43m Bad Checksums: 213.07m IO Errors: 5.51m SPA Mismatch: 27.62b L2 ARC Size: (Adaptive) 1.44 TiB Header Size: 5.19% 76.70 GiB L2 ARC Evicts: Lock Retries: 1.64k Upon Reading: 0 L2 ARC Breakdown: 705.25m Hit Ratio: 81.82% 577.01m Miss Ratio: 18.18% 128.24m Feeds: 1.52m L2 ARC Buffer: Bytes Scanned: 7.49 PiB Buffer Iterations: 1.52m List Iterations: 96.55m NULL List Iterations: 9.23m L2 ARC Writes: Writes Sent: 100.00% 826.96k ---------------------------------------------------------------------- In /boot/loader.conf (128GB RAM in system): vm.kmem_size="110G" vfs.zfs.arc_max="100G" vfs.zfs.arc_min="80G" vfs.zfs.vdev.cache.size=16M vfs.zfs.vdev.cache.max="16384" vfs.zfs.txg.timeout="10" vfs.zfs.write_limit_min="134217728" vfs.zfs.vdev.cache.bshift="14" vfs.zfs.arc_meta_limit=53687091200 vfs.zfs.l2arc_write_max=25165824 vfs.zfs.l2arc_write_boost=50331648 vfs.zfs.l2arc_noprefetch=0 In /etc/sysctl.conf: vfs.zfs.l2arc_write_max=25000000 vfs.zfs.l2arc_write_boost=50000000 vfs.zfs.l2arc_noprefetch=0 vfs.zfs.l2arc_headroom=8 vfs.zfs.l2arc_feed_min_ms=30 vfs.zfs.arc_meta_limit=53687091200 How can L2 ARC Size: (Adaptive) be 1.44 TiB (up) with total physical size of L2ARC devices 490GB? Why this values can grow and become so high? kstat.zfs.misc.arcstats.l2_cksum_bad: 212832715 kstat.zfs.misc.arcstats.l2_io_error: 5501886 Thanks for any help and ideas! Received on Mon Oct 07 2013 - 16:25:13 UTC