('binary' encoding is not supported, stored as-is) Good day community, We ran the system in production with r259544 on it. 128Gb RAM, 72Gb - arc_max and 42Gb - arc_min set in loader.conf. The system ran both application (which are FS ops hungry) and 18Tb storage on this setup. >From the start, ARC clearly shows 72Gb of memory consumed, but during next few days it pressured out to even lower than arc_min (after 14 days of uptime there's 28Gb only used by ARC. The problem is: lesser data in the ARC, lesser performance from the entire system. First two days (when ARC slowly decreases to 55-60Gb) the system run fine with excellent performance telemetry (application response time) up to 7-10 days, when it reach ~30Gb and then the performance falls off to unacceptable level. So the question is: how to understand, who "eats" memory from ARC, and how to control this memory eating ? top, vmstat's follow (vm.kmem_size limited to 100Gb, if not, wired will reach 107Gb approx with the same ARC pressured out) last pid: 13273; load averages: 2.99, 1.55, 1.20 up 14+19:00:35 14:24:55 227 processes: 1 running, 226 sleeping CPU: 4.6% user, 0.0% nice, 5.9% system, 1.0% interrupt, 88.5% idle Mem: 667M Active, 30G Inact, 91G Wired, 1783M Cache, 3309M Buf, 933M Free ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other Swap: $ vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 208, 2, 208, 0, 0 UMA Zones: 2176, 0, 208, 0, 208, 0, 0 UMA Slabs: 80, 0, 2311694, 204306,718386402, 0, 0 UMA RCntSlabs: 88, 0, 9229, 41, 18930, 0, 0 UMA Hash: 256, 0, 3, 12, 79, 0, 0 4 Bucket: 32, 0, 1785, 21090,1248973183, 0, 0 6 Bucket: 48, 0, 185, 12763,148487075, 0, 0 8 Bucket: 64, 0, 65, 20643,41333608, 0, 0 12 Bucket: 96, 0, 9665, 9482,11049020, 0, 0 16 Bucket: 128, 0, 649, 4807,11971939, 11, 0 32 Bucket: 256, 0, 12630, 50790,31362215, 33, 0 64 Bucket: 512, 0, 23203, 6781,17367241,1336666, 0 128 Bucket: 1024, 0, 95458, 48746,810293851,317422, 0 vmem btag: 56, 0, 1411085, 96316,27776938,21231, 0 VM OBJECT: 256, 0, 771497, 279598,2938160439, 0, 0 RADIX NODE: 144, 0, 2598081, 1326882,4515451399, 0, 0 MAP: 240, 0, 3, 61, 3, 0, 0 KMAP ENTRY: 128, 0, 25, 874, 25, 0, 0 MAP ENTRY: 128, 0, 25469, 45149,11668899620, 0, 0 VMSPACE: 448, 0, 235, 782,149851562, 0, 0 fakepg: 104, 0, 0, 0, 0, 0, 0 mt_zone: 4112, 0, 263, 0, 263, 0, 0 16: 16, 0, 2812047, 93027,53798617567, 0, 0 32: 32, 0, 9723566, 1887934,12748376945, 0, 0 64: 64, 0,10475477, 6543585,20857570861, 0, 0 128: 128, 0, 1848882, 490254,23475247741, 0, 0 256: 256, 0, 914734, 702251,12021626624, 0, 0 512: 512, 0, 1307, 581,10911947626, 0, 0 1024: 1024, 0, 28407, 233,234462675, 0, 0 2048: 2048, 0, 511, 2463,2135442383, 0, 0 4096: 4096, 0, 507210, 85487,572913690, 0, 0 uint64 pcpu: 8, 0, 298, 598, 298, 0, 0 SLEEPQUEUE: 80, 0, 1666, 721, 1669, 0, 0 Files: 80, 0, 4779, 10221,3345112793, 0, 0 TURNSTILE: 136, 0, 1666, 414, 1669, 0, 0 rl_entry: 40, 0, 696, 804, 696, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 MAC labels: 40, 0, 0, 0, 0, 0, 0 PROC: 1208, 0, 256, 473,149840920, 0, 0 THREAD: 1168, 0, 1514, 151, 783235, 0, 0 cpuset: 72, 0, 791, 1134, 11746, 0, 0 audit_record: 1248, 0, 0, 0, 0, 0, 0 sendfile_sync: 64, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 41943045, 8248, 4127,8662647947, 0, 0 mbuf: 256, 41943045, 314, 13696,64235586097, 0, 0 mbuf_cluster: 2048, 6553600, 12375, 41, 20500, 0, 0 mbuf_jumbo_page: 4096, 3276800, 311, 2710,1565741448, 0, 0 mbuf_jumbo_9k: 9216, 970903, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 546133, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 2259,166082214, 0, 0 g_bio: 248, 0, 9, 36135,5971956549, 0, 0 DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, 0 ttyinq: 160, 0, 240, 810, 2520, 0, 0 ttyoutq: 256, 0, 126, 729, 1311, 0, 0 FPU_save_area: 832, 0, 0, 0, 0, 0, 0 taskq_zone: 48, 0, 0, 2656,154582289, 0, 0 VNODE: 472, 0, 926451, 683325,224955145, 0, 0 VNODEPOLL: 112, 0, 0, 140, 1, 0, 0 BUF TRIE: 144, 0, 64, 105884,13961458, 0, 0 NAMEI: 1024, 0, 1, 471,9070182588, 0, 0 S VFS Cache: 108, 0, 884046, 757454,505000615, 0, 0 STS VFS Cache: 148, 0, 0, 0, 0, 0, 0 L VFS Cache: 328, 0, 5, 106747,78972025, 0, 0 LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0 pipe: 744, 0, 10, 240,119813676, 0, 0 procdesc: 128, 0, 0, 0, 0, 0, 0 Mountpoints: 816, 0, 13, 67, 13, 0, 0 range_seg_cache: 64, 0, 3308624, 602274,2773749361, 0, 0 zio_cache: 920, 0, 17, 46059,27531825813, 0, 0 zio_link_cache: 48, 0, 12, 46634,21375486079, 0, 0 zio_buf_512: 512, 0, 1838128, 579528,233163218, 0, 0 zio_data_buf_512: 512, 0, 284, 468,280535356, 0, 0 zio_buf_1024: 1024, 0, 18918, 2674,70036984, 0, 0 zio_data_buf_1024: 1024, 0, 72, 244,25068218, 0, 0 zio_buf_1536: 1536, 0, 685, 111,15021333, 0, 0 zio_data_buf_1536: 1536, 0, 231, 135, 5491921, 0, 0 zio_buf_2048: 2048, 0, 464, 74,27382096, 0, 0 zio_data_buf_2048: 2048, 0, 252, 162, 3022922, 0, 0 zio_buf_2560: 2560, 0, 322, 39,16246431, 0, 0 zio_data_buf_2560: 2560, 0, 648, 44, 4092684, 0, 0 zio_buf_3072: 3072, 0, 266, 39,22464410, 0, 0 zio_data_buf_3072: 3072, 0, 272, 42, 3713205, 0, 0 zio_buf_3584: 3584, 0, 255, 12,13329596, 0, 0 zio_data_buf_3584: 3584, 0, 344, 15, 4160159, 0, 0 zio_buf_4096: 4096, 0, 11326, 31,296458917, 0, 0 zio_data_buf_4096: 4096, 0, 758, 22,55468107, 0, 0 zio_buf_5120: 5120, 0, 348, 13,14469868, 0, 0 zio_data_buf_5120: 5120, 0, 1048, 16,28617233, 0, 0 zio_buf_6144: 6144, 0, 286, 14,28793904, 0, 0 zio_data_buf_6144: 6144, 0, 1228, 12, 9882405, 0, 0 zio_buf_7168: 7168, 0, 287, 15,19650053, 0, 0 zio_data_buf_7168: 7168, 0, 551, 14, 3698354, 0, 0 zio_buf_8192: 8192, 0, 267, 15,41728685, 0, 0 zio_data_buf_8192: 8192, 0, 447, 17,13520347, 0, 0 zio_buf_10240: 10240, 0, 549, 14,25924463, 0, 0 zio_data_buf_10240: 10240, 0, 1035, 16,29565424, 0, 0 zio_buf_12288: 12288, 0, 496, 13,31221546, 0, 0 zio_data_buf_12288: 12288, 0, 1727, 14,15662238, 0, 0 zio_buf_14336: 14336, 0, 475, 11,15266392, 0, 0 zio_data_buf_14336: 14336, 0, 1000, 19,26789503, 0, 0 zio_buf_16384: 16384, 0, 774969, 25,765531478, 0, 0 zio_data_buf_16384: 16384, 0, 35652, 13,52893314, 0, 0 zio_buf_20480: 20480, 0, 890, 19,31495207, 0, 0 zio_data_buf_20480: 20480, 0, 1545, 15,36009279, 0, 0 zio_buf_24576: 24576, 0, 825, 12,36150814, 0, 0 zio_data_buf_24576: 24576, 0, 4914, 14,41401016, 0, 0 zio_buf_28672: 28672, 0, 701, 14,23285223, 0, 0 zio_data_buf_28672: 28672, 0, 1409, 21,27435301, 0, 0 zio_buf_32768: 32768, 0, 675, 13,31432326, 0, 0 zio_data_buf_32768: 32768, 0, 2415, 13,25945712, 0, 0 zio_buf_36864: 36864, 0, 622, 15,22148993, 0, 0 zio_data_buf_36864: 36864, 0, 1779, 15,25706882, 0, 0 zio_buf_40960: 40960, 0, 536, 18,23476092, 0, 0 zio_data_buf_40960: 40960, 0, 1673, 14,24666224, 0, 0 zio_buf_45056: 45056, 0, 513, 14,19162070, 0, 0 zio_data_buf_45056: 45056, 0, 1592, 16,17046396, 0, 0 zio_buf_49152: 49152, 0, 478, 17,24871123, 0, 0 zio_data_buf_49152: 49152, 0, 1419, 16,16353824, 0, 0 zio_buf_53248: 53248, 0, 467, 11,20353004, 0, 0 zio_data_buf_53248: 53248, 0, 1233, 16,17298271, 0, 0 zio_buf_57344: 57344, 0, 460, 13,23454274, 0, 0 zio_data_buf_57344: 57344, 0, 1267, 17,18596329, 0, 0 zio_buf_61440: 61440, 0, 423, 13,20273525, 0, 0 zio_data_buf_61440: 61440, 0, 1033, 15,15431244, 0, 0 zio_buf_65536: 65536, 0, 365, 13,20928232, 0, 0 zio_data_buf_65536: 65536, 0, 1040, 17,11739711, 0, 0 zio_buf_69632: 69632, 0, 351, 13,16369173, 0, 0 zio_data_buf_69632: 69632, 0, 1144, 14, 9285339, 0, 0 zio_buf_73728: 73728, 0, 344, 17,15927115, 0, 0 zio_data_buf_73728: 73728, 0, 932, 16, 6682428, 0, 0 zio_buf_77824: 77824, 0, 302, 13,13557151, 0, 0 zio_data_buf_77824: 77824, 0, 901, 15, 6842383, 0, 0 zio_buf_81920: 81920, 0, 301, 13,14443323, 0, 0 zio_data_buf_81920: 81920, 0, 909, 17, 5007810, 0, 0 zio_buf_86016: 86016, 0, 282, 11,12344084, 0, 0 zio_data_buf_86016: 86016, 0, 844, 17, 4623199, 0, 0 zio_buf_90112: 90112, 0, 306, 9,13898488, 0, 0 zio_data_buf_90112: 90112, 0, 861, 15, 6016346, 0, 0 zio_buf_94208: 94208, 0, 382, 13,13793511, 0, 0 zio_data_buf_94208: 94208, 0, 760, 18, 5061270, 0, 0 zio_buf_98304: 98304, 0, 283, 12,16091732, 0, 0 zio_data_buf_98304: 98304, 0, 752, 16, 5314097, 0, 0 zio_buf_102400: 102400, 0, 246, 13,14646577, 0, 0 zio_data_buf_102400: 102400, 0, 707, 20, 5504194, 0, 0 zio_buf_106496: 106496, 0, 237, 11,15777390, 0, 0 zio_data_buf_106496: 106496, 0, 704, 15, 4822634, 0, 0 zio_buf_110592: 110592, 0, 184, 13,15817496, 0, 0 zio_data_buf_110592: 110592, 0, 602, 18, 4925430, 0, 0 zio_buf_114688: 114688, 0, 166, 16,17045866, 0, 0 zio_data_buf_114688: 114688, 0, 532, 16, 5191669, 0, 0 zio_buf_118784: 118784, 0, 170, 15,12659006, 0, 0 zio_data_buf_118784: 118784, 0, 481, 17, 2124105, 0, 0 zio_buf_122880: 122880, 0, 196, 15,12948958, 0, 0 zio_data_buf_122880: 122880, 0, 524, 20, 2727770, 0, 0 zio_buf_126976: 126976, 0, 177, 12,17364224, 0, 0 zio_data_buf_126976: 126976, 0, 516, 13, 1926732, 0, 0 zio_buf_131072: 131072, 0, 179, 14,340379531, 0, 0 zio_data_buf_131072: 131072, 0, 488217, 9,287155442, 0, 0 lz4_ctx: 16384, 0, 0, 64,459560153, 0, 0 sa_cache: 80, 0, 925296, 694854,221400673, 0, 0 dnode_t: 864, 0, 3286406, 492342,164927567, 0, 0 dmu_buf_impl_t: 224, 0, 2735768, 1283814,1130573685, 0, 0 arc_buf_hdr_t: 216, 0, 8241154, 594326,869545121, 0, 0 arc_buf_t: 72, 0, 930962, 479568,1220508333, 0, 0 zil_lwb_cache: 192, 0, 30, 5830,45306321, 0, 0 zfs_znode_cache: 368, 0, 925295, 690235,221400672, 0, 0 ksiginfo: 112, 0, 720, 2535, 2627344, 0, 0 itimer: 352, 0, 0, 473, 4261, 0, 0 pf states: 296, 500006, 112, 473, 2146938, 0, 0 pf state keys: 88, 0, 118, 2042, 2220665, 0, 0 pf source nodes: 136, 500018, 0, 0, 0, 0, 0 pf mtags: 40, 0, 0, 0, 0, 0, 0 pf table entries: 160, 200000, 16, 59, 16, 0, 0 pf table counters: 64, 0, 0, 0, 0, 0, 0 pf frags: 80, 0, 0, 0, 0, 0, 0 pf frag entries: 32, 40000, 0, 0, 0, 0, 0 pf state scrubs: 40, 0, 0, 0, 0, 0, 0 KNOTE: 128, 0, 3586, 7636,4249425530, 0, 0 socket: 696, 4192805, 10988, 14362,617302776, 0, 0 unpcb: 240, 4192816, 397, 1491, 3964687, 0, 0 ipq: 56, 204835, 0, 0, 0, 0, 0 udp_inpcb: 392, 4192810, 161, 659,33932308, 0, 0 udpcb: 16, 4192955, 161, 2098,33932308, 0, 0 tcp_inpcb: 392, 4192810, 20662, 18868,579405676, 0, 0 tcpcb: 1024, 4192808, 4067, 7453,579405676, 0, 0 tcptw: 88, 27810, 16595, 11215,259638269,257834306, 0 syncache: 160, 15375, 0, 800,369362350, 0, 0 hostcache: 136, 15370, 54, 932, 2020, 0, 0 tcpreass: 40, 409600, 0, 2000, 121958, 0, 0 sackhole: 32, 0, 0, 1375, 25994, 0, 0 sctp_ep: 1400, 4192806, 0, 100, 103, 0, 0 sctp_asoc: 2344, 40000, 0, 24, 103, 0, 0 sctp_laddr: 48, 80012, 0, 415, 3, 0, 0 sctp_raddr: 720, 80000, 0, 130, 103, 0, 0 sctp_chunk: 136, 400026, 0, 435, 11, 0, 0 sctp_readq: 104, 400026, 0, 0, 0, 0, 0 sctp_stream_msg_out: 104, 400026, 0, 1406, 103, 0, 0 sctp_asconf: 40, 400000, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, 0 ripcb: 392, 4192810, 0, 0, 0, 0, 0 rtentry: 200, 0, 8, 312, 8, 0, 0 selfd: 56, 0, 1288, 2120,9888371057, 0, 0 SWAPMETA: 288, 16337503, 0, 0, 0, 0, 0 FFS inode: 168, 0, 1028, 1180, 3550806, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 1028, 1177, 3550806, 0, 0 cyclic_id_cache: 64, 0, 0, 558, 8, 0, 0 dtrace_state_cache: 4096, 0, 0, 5, 8, 0, 0 taskq_zone: 48, 0, 0, 0, 0, 0, 0 NCLNODE: 528, 0, 0, 0, 0, 0, 0 $ vmstat -m Type InUse MemUse HighUse Requests Size(s) devbuf 17816 38148K - 18364 16,32,64,128,256,512,1024,2048,4096 temp 83 13K - 363465851 16,32,64,128,256,512,1024,2048,4096 acpiintr 1 1K - 1 64 module 196 25K - 196 128 mtx_pool 2 16K - 2 acpica 175642 18132K - 482200 16,32,64,128,256,512,1024,2048,4096 CAM periph 88 22K - 142 16,32,64,128,256 osd 28 1K - 2691953 16,32,64 CAM queue 61 632K - 239 16,32,64,128,256,512,1024 pmchooks 1 1K - 1 128 acpitask 1 8K - 1 acpisem 26 4K - 26 128 kbdmux 7 18K - 7 16,512,1024,2048 pgrp 36 5K - 496601 128 session 32 4K - 33701 128 proc 2 512K - 2 subproc 989 1401K - 149847035 512,4096 cred 124 20K - 4925850262 64,256 plimit 18 5K - 33674 256 uidinfo 8 65K - 140875 128 LED 28 2K - 28 16,128 md_disk 1 2K - 1 2048 raid_data 0 0K - 1584 32,128,256 mps 13 1603K - 585860 16,32,64,128,256,512,1024,2048,4096 USB 37 47K - 43 16,32,128,256,512,1024,2048,4096 USBdev 31 8K - 31 32,64,128,512,4096 sysctl 0 0K - 29680596 16,32,64 sysctloid 7556 374K - 7704 16,32,64,128 sysctltmp 0 0K - 31532343 16,32,64,128,256,2048 tidhash 1 512K - 1 callout 13 4232K - 13 umtx 3330 417K - 3336 128 p1003.1b 1 1K - 1 16 bus 950 123K - 20210 16,32,64,128,256,1024,2048 bus-sc 129 241K - 4112 16,32,64,128,256,512,1024,2048,4096 devstat 114 230K - 114 32,4096 eventhandler 98 8K - 98 64,128 CAM dev queue 9 1K - 9 64 acpidev 162 11K - 162 64 kobj 92 368K - 2386 4096 Per-cpu 1 1K - 1 32 CAM SIM 9 3K - 9 256 rman 248 28K - 662 32,128 sbuf 1 1K - 62594 16,32,64,128,256,512,1024,2048,4096 CAM XPT 388 35K - 6374 16,32,64,128,256,512,1024,2048 CAM DEV 52 104K - 97 2048 stack 0 0K - 40 256 taskqueue 259 41K - 347 16,32,64,128,256 Unitno 25 2K - 521437312 32,64 vmem 3 1184K - 16 256,512,1024,2048,4096 ioctlops 0 0K - 2301578270 16,32,64,128,256,512,1024,2048,4096 select 687 86K - 687 128 iov 0 0K - 3070408744 16,32,64,128,256,512,2048 msg 4 30K - 4 2048,4096 sem 4 106K - 4 2048,4096 shm 1 20K - 1 tty 23 23K - 93 1024,4096 pts 3 1K - 33 256 mbuf_tag 0 0K - 2 32 shmfd 1 8K - 1 soname 5 1K - 44996938125 16,32,128 pcb 1508 16579K - 216776778 16,32,128,256,1024,2048 md_nvidia_data 0 0K - 264 512 acl 0 0K - 2073 4096 vfscache 1 32768K - 1 cl_savebuf 0 0K - 2176560 64 vfs_hash 1 16384K - 1 vnodes 1 1K - 6 32,256 mount 150 6K - 540 16,32,64,128,256,1024 CAM CCB 5 10K - 1978405860 2048 vnodemarker 0 0K - 860570 512 BPF 4 1K - 4 128 md_sii_data 0 0K - 264 512 ifnet 5 9K - 5 128,2048 ifaddr 30 7K - 30 32,256,4096 ether_multi 11 1K - 12 16,64 clone 3 1K - 3 128 arpcom 3 1K - 3 16 lltable 76 20K - 14079 256,512 entropy 1026 65K - 9778336 32,64,4096 scsi_da 0 0K - 6483 32,64,512 routetbl 55 11K - 53761245 32,128,256,512 igmp 4 1K - 4 256 in_multi 2 1K - 2 256 UART 9 7K - 9 16,1024 CAM path 95 3K - 997 32 SCSI ENC 297 388K - 5602301 16,64,256,1024,2048,4096 sctp_map 0 0K - 206 16 sctp_stro 0 0K - 103 1024 sctp_a_it 0 0K - 3 16 sctp_atcl 0 0K - 206 512 sctp_atky 0 0K - 309 32,64 sctp_athm 0 0K - 206 16 sctp_vrf 1 1K - 1 64 sctp_ifa 4 1K - 4 128 sctp_ifn 2 1K - 2 128 sctp_timw 31 8K - 31 256 sctp_iter 0 0K - 3 256 hostcache 1 28K - 1 syncache 1 64K - 1 pf_hash 3 2880K - 3 pf_ifnet 8 4K - 38 256,2048 pf_rule 32 27K - 32 128,1024 pf_osfp 1184 122K - 1184 64,128 pf_table 5 10K - 10 2048 audit_evclass 187 6K - 228 32 vm_pgdata 1 16384K - 1 UMAHash 76 19947K - 497 512,1024,2048,4096 pci_link 16 2K - 16 64,128 acpi_perf 12 6K - 12 512 DEVFS2 302 5K - 332 16 DEVFS3 659 165K - 857 256 DEVFS1 302 151K - 463 512 isadev 7 1K - 7 128 memdesc 1 4K - 1 4096 DEVFS_RULE 1 1K - 1 64 DEVFS 47 1K - 50 16,128 DEVFSP 1 1K - 18 64 cdev 7 2K - 7 256 pfs_nodes 21 6K - 21 256 filedesc 701689 15308K - 151391230 16,32,64,128,256,512,1024,2048,4096 sigio 5 1K - 5 64 filecaps 0 0K - 376693052 16,64 kdtrace 1774 395K - 150625287 64,256 kenv 96 12K - 112 16,32,64,128 apmdev 1 1K - 1 128 madt_table 0 0K - 1 4096 kqueue 14 110K - 7001897 256,512,2048,4096 proc-args 237 23K - 1744195324 16,32,64,128,256 pfs_vncache 0 0K - 17 64 hhook 2 1K - 2 256 ithread 156 28K - 156 32,128,256 GEOM 1310 250K - 15089 16,32,64,128,256,512,1024,2048 io_apic 2 4K - 2 2048 prison 2 5K - 2 16,4096 MCA 31 6K - 31 128,256 KTRACE 100 13K - 100 128 msi 16 2K - 16 128 nexusdev 3 1K - 3 16 linker 404 885K - 587 16,32,64,128,256,512,1024,2048,4096 lockf 41 6K - 3660069993 64,128,256,512 loginclass 3 1K - 22891 64 rpc 1 1K - 1 256 kstat_data 6 1K - 6 64 solaris 25382156 3763478K - 72008554115 16,32,64,128,256,512,1024,2048,4096 ufs_mount 3 13K - 3 512,4096 nullfs_hash 1 131072K - 1 nullfs_node 1 1K - 1 64 nullfs_mount 1 1K - 1 32 cyclic 48 4K - 50 16,64,128 fbt 29000 3881K - 29000 128 SDT 26 2K - 26 16,64 $ zfs-stats -a ------------------------------------------------------------------------ ZFS Subsystem Report Sat Jan 4 14:27:19 2014 ------------------------------------------------------------------------ System Information: Kernel Version: 1100004 (osreldate) Hardware Platform: amd64 Processor Architecture: amd64 ZFS Storage pool Version: 5000 ZFS Filesystem Version: 5 FreeBSD 11.0-CURRENT #5 r259544: Fri Dec 20 11:13:21 EET 2013 root 14:27 up 14 days, 19:03, 2 users, load averages: 1,27 1,56 1,27 ------------------------------------------------------------------------ System Memory: 0.54% 693.36 MiB Active, 24.32% 30.31 GiB Inact 72.98% 90.96 GiB Wired, 1.42% 1.78 GiB Cache 0.70% 898.79 MiB Free, 0.04% 48.04 MiB Gap Real Installed: 128.00 GiB Real Available: 99.96% 127.95 GiB Real Managed: 97.41% 124.65 GiB Logical Total: 128.00 GiB Logical Used: 74.25% 95.04 GiB Logical Free: 25.75% 32.96 GiB Kernel Memory: 3.92 GiB Data: 99.65% 3.90 GiB Text: 0.35% 14.00 MiB Kernel Memory Map: 100.00 GiB Size: 75.00% 75.00 GiB Free: 25.00% 25.00 GiB ------------------------------------------------------------------------ ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 425.50m Recycle Misses: 234.05m Mutex Misses: 3.60m Evict Skips: 502.81m ARC Size: 38.91% 28.02 GiB Target Size: (Adaptive) 58.33% 42.00 GiB Min Size (Hard Limit): 58.33% 42.00 GiB Max Size (High Water): 1:1 72.00 GiB ARC Size Breakdown: Recently Used Cache Size: 48.81% 20.50 GiB Frequently Used Cache Size: 51.19% 21.50 GiB ARC Hash Breakdown: Elements Max: 8.83m Elements Current: 93.29% 8.24m Collisions: 1.27b Chain Max: 27 Chains: 1.76m ------------------------------------------------------------------------ ARC Efficiency: 146.52b Cache Hit Ratio: 99.62% 145.96b Cache Miss Ratio: 0.38% 556.49m Actual Hit Ratio: 99.52% 145.81b Data Demand Efficiency: 92.35% 4.49b Data Prefetch Efficiency: 8.56% 64.91m CACHE HITS BY CACHE LIST: Most Recently Used: 0.98% 1.43b Most Frequently Used: 98.92% 144.38b Most Recently Used Ghost: 0.01% 20.56m Most Frequently Used Ghost: 0.10% 152.70m CACHE HITS BY DATA TYPE: Demand Data: 2.84% 4.15b Prefetch Data: 0.00% 5.56m Demand Metadata: 97.06% 141.67b Prefetch Metadata: 0.10% 143.60m CACHE MISSES BY DATA TYPE: Demand Data: 61.74% 343.57m Prefetch Data: 10.67% 59.35m Demand Metadata: 25.82% 143.71m Prefetch Metadata: 1.77% 9.86m ------------------------------------------------------------------------ L2 ARC Summary: (HEALTHY) Passed Headroom: 27.09m Tried Lock Failures: 916.86m IO In Progress: 13.17k Low Memory Aborts: 301.48k Free on Write: 1.31m Writes While Full: 226.67k R/W Clashes: 99.81k Bad Checksums: 0 IO Errors: 0 SPA Mismatch: 2.32b L2 ARC Size: (Adaptive) 278.93 GiB Header Size: 0.26% 750.98 MiB L2 ARC Evicts: Lock Retries: 6.89k Upon Reading: 2 L2 ARC Breakdown: 556.49m Hit Ratio: 39.35% 218.97m Miss Ratio: 60.65% 337.52m Feeds: 969.70k L2 ARC Buffer: Bytes Scanned: 4.23 PiB Buffer Iterations: 969.70k List Iterations: 60.83m NULL List Iterations: 1.63m L2 ARC Writes: Writes Sent: 100.00% 965.91k ------------------------------------------------------------------------ File-Level Prefetch: (HEALTHY) DMU Efficiency: 171.97b Hit Ratio: 68.01% 116.95b Miss Ratio: 31.99% 55.02b Colinear: 55.02b Hit Ratio: 0.00% 111.39k Miss Ratio: 100.00% 55.02b Stride: 116.77b Hit Ratio: 99.95% 116.72b Miss Ratio: 0.05% 56.07m DMU Misc: Reclaim: 55.02b Successes: 0.11% 60.47m Failures: 99.89% 54.96b Streams: 199.20m +Resets: 0.87% 1.73m -Resets: 99.13% 197.47m Bogus: 0 ------------------------------------------------------------------------ VDEV cache is disabled ------------------------------------------------------------------------ ZFS Tunables (sysctl): kern.maxusers 8525 vm.kmem_size 107374182400 vm.kmem_size_scale 1 vm.kmem_size_min 0 vm.kmem_size_max 1319413950874 vfs.zfs.arc_max 77309411328 vfs.zfs.arc_min 45097156608 vfs.zfs.arc_meta_used 19327359728 vfs.zfs.arc_meta_limit 19327352832 vfs.zfs.l2arc_write_max 33554432 vfs.zfs.l2arc_write_boost 134217728 vfs.zfs.l2arc_headroom 2 vfs.zfs.l2arc_feed_secs 1 vfs.zfs.l2arc_feed_min_ms 200 vfs.zfs.l2arc_noprefetch 1 vfs.zfs.l2arc_feed_again 1 vfs.zfs.l2arc_norw 1 vfs.zfs.anon_size 47123968 vfs.zfs.anon_metadata_lsize 0 vfs.zfs.anon_data_lsize 0 vfs.zfs.mru_size 21964981248 vfs.zfs.mru_metadata_lsize 4259141120 vfs.zfs.mru_data_lsize 10711522816 vfs.zfs.mru_ghost_size 23117245952 vfs.zfs.mru_ghost_metadata_lsize 23086417920 vfs.zfs.mru_ghost_data_lsize 30828032 vfs.zfs.mfu_size 2190997504 vfs.zfs.mfu_metadata_lsize 409456128 vfs.zfs.mfu_data_lsize 949248 vfs.zfs.mfu_ghost_size 21976749568 vfs.zfs.mfu_ghost_metadata_lsize 21912697344 vfs.zfs.mfu_ghost_data_lsize 64052224 vfs.zfs.l2c_only_size 292763954176 vfs.zfs.dedup.prefetch 1 vfs.zfs.nopwrite_enabled 1 vfs.zfs.mdcomp_disable 0 vfs.zfs.prefetch_disable 0 vfs.zfs.zfetch.max_streams 8 vfs.zfs.zfetch.min_sec_reap 2 vfs.zfs.zfetch.block_cap 256 vfs.zfs.zfetch.array_rd_sz 1048576 vfs.zfs.top_maxinflight 32 vfs.zfs.resilver_delay 2 vfs.zfs.scrub_delay 4 vfs.zfs.scan_idle 50 vfs.zfs.scan_min_time_ms 1000 vfs.zfs.free_min_time_ms 1000 vfs.zfs.resilver_min_time_ms 3000 vfs.zfs.no_scrub_io 0 vfs.zfs.no_scrub_prefetch 0 vfs.zfs.metaslab.gang_bang 131073 vfs.zfs.metaslab.debug_load 0 vfs.zfs.metaslab.debug_unload 0 vfs.zfs.metaslab.df_alloc_threshold 131072 vfs.zfs.metaslab.df_free_pct 4 vfs.zfs.metaslab.min_alloc_size 10485760 vfs.zfs.metaslab.load_pct 50 vfs.zfs.metaslab.unload_delay 8 vfs.zfs.metaslab.preload_limit 3 vfs.zfs.metaslab.preload_enabled 1 vfs.zfs.metaslab.weight_factor_enable 0 vfs.zfs.condense_pct 200 vfs.zfs.mg_alloc_failures 18 vfs.zfs.mg_noalloc_threshold 0 vfs.zfs.write_to_degraded 0 vfs.zfs.check_hostid 1 vfs.zfs.recover 0 vfs.zfs.deadman_synctime_ms 1000000 vfs.zfs.deadman_checktime_ms 5000 vfs.zfs.deadman_enabled 1 vfs.zfs.spa_asize_inflation 24 vfs.zfs.txg.timeout 10 vfs.zfs.vdev.cache.max 16384 vfs.zfs.vdev.cache.size 0 vfs.zfs.vdev.cache.bshift 16 vfs.zfs.vdev.trim_on_init 1 vfs.zfs.vdev.mirror.rotating_inc 0 vfs.zfs.vdev.mirror.rotating_seek_inc 5 vfs.zfs.vdev.mirror.rotating_seek_offset1048576 vfs.zfs.vdev.mirror.non_rotating_inc 0 vfs.zfs.vdev.mirror.non_rotating_seek_inc1 vfs.zfs.vdev.max_active 1000 vfs.zfs.vdev.sync_read_min_active 10 vfs.zfs.vdev.sync_read_max_active 10 vfs.zfs.vdev.sync_write_min_active 10 vfs.zfs.vdev.sync_write_max_active 10 vfs.zfs.vdev.async_read_min_active 1 vfs.zfs.vdev.async_read_max_active 3 vfs.zfs.vdev.async_write_min_active 1 vfs.zfs.vdev.async_write_max_active 10 vfs.zfs.vdev.scrub_min_active 1 vfs.zfs.vdev.scrub_max_active 2 vfs.zfs.vdev.aggregation_limit 131072 vfs.zfs.vdev.read_gap_limit 32768 vfs.zfs.vdev.write_gap_limit 4096 vfs.zfs.vdev.bio_flush_disable 0 vfs.zfs.vdev.bio_delete_disable 0 vfs.zfs.vdev.trim_max_bytes 2147483648 vfs.zfs.vdev.trim_max_pending 64 vfs.zfs.max_auto_ashift 13 vfs.zfs.zil_replay_disable 0 vfs.zfs.cache_flush_disable 0 vfs.zfs.zio.use_uma 1 vfs.zfs.zio.exclude_metadata 0 vfs.zfs.sync_pass_deferred_free 2 vfs.zfs.sync_pass_dont_compress 5 vfs.zfs.sync_pass_rewrite 2 vfs.zfs.snapshot_list_prefetch 0 vfs.zfs.super_owner 0 vfs.zfs.debug 0 vfs.zfs.version.ioctl 3 vfs.zfs.version.acl 1 vfs.zfs.version.spa 5000 vfs.zfs.version.zpl 5 vfs.zfs.trim.enabled 1 vfs.zfs.trim.txg_delay 32 vfs.zfs.trim.timeout 30 vfs.zfs.trim.max_interval 1 ------------------------------------------------------------------------ Received on Sat Jan 04 2014 - 12:08:39 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:46 UTC