('binary' encoding is not supported, stored as-is) Dear Andriy and FreeBSD community, After applying this path one of the systems runs fine (disk subsystem load low to moderate - 10-20% busy sustained), Then I saw this patch was merged to the HEAD and we apply it to the one of the systems with moderate to high disk load: 30-60% busy (11.0-CURRENT #7 r261118: Fri Jan 24 17:25:08 EET 2014) Within 4 days we experiencing the same leak(?) as without patch: last pid: 53841; load averages: 4.47, 4.18, 3.78 up 3+16:37:09 11:24:39 543 processes: 6 running, 537 sleeping CPU: 8.7% user, 0.0% nice, 14.6% system, 1.4% interrupt, 75.3% idle Mem: 22G Active, 1045M Inact, 98G Wired, 1288M Cache, 3284M Buf, 2246M Free ARC: 73G Total, 3763M MFU, 62G MRU, 56M Anon, 1887M Header, 4969M Other Swap: The ARC is populated within 30mins under load to the max (90Gb) then start decreasing. The delta between Wiread and ARC total start growing from typical 10-12Gb without L2 enabled to the 25Gb with L2 enabled and counting (4 hours ago was 22Gb delta). L2ARC statistics: L2 ARC Size: (Adaptive) 291.63 GiB Header Size: 0.25% 734.14 MiB L2 ARC Evicts: Lock Retries: 682 Upon Reading: 0 L2 ARC Breakdown: 106.56m Hit Ratio: 29.04% 30.95m Miss Ratio: 70.96% 75.62m Feeds: 317.18k So, again, what shall we do to better understand/mitigate the problem further ? Thank you. > on 15/01/2014 12:28 Vitalij Satanivskij said the following: > > Dear Andriy and FreeBSD community, > > > > Andriy Gapon wrote: > > AG> on 14/01/2014 07:27 Vladimir Sharun said the following: > > AG> > Dear Andriy and FreeBSD community, > > AG> > > > AG> >> I am not sure if the buffers are leaked somehow or if they are actually in use. > > AG> >> It's one of the very few places where data buffers are allocated without > > AG> >> charging ARC. In all other places it's quite easy to match allocations and > > AG> >> deallocations. But in L2ARC it is not obvious that all buffers get freed or > > AG> >> when that happens. > > AG> > > > AG> > After one week under load I think we figure out the cause: it's L2ARC. > > AG> > Here's the top's header for 7d17h of the runtime: > > AG> > > > AG> > last pid: 46409; load averages: 0.37, 0.62, 0.70 up 7+17:14:01 07:24:10 > > AG> > 173 processes: 1 running, 171 sleeping, 1 zombie > > AG> > CPU: 2.0% user, 0.0% nice, 3.5% system, 0.4% interrupt, 94.2% idle > > AG> > Mem: 8714M Active, 14G Inact, 96G Wired, 1929M Cache, 3309M Buf, 3542M Free > > AG> > ARC: 85G Total, 2558M MFU, 77G MRU, 28M Anon, 1446M Header, 4802M Other > > AG> > > > AG> > ARC related tunables: > > AG> > > > AG> > vm.kmem_size="110G" > > AG> > vfs.zfs.arc_max="90G" > > AG> > vfs.zfs.arc_min="42G" > > AG> > > > AG> > For more than 7 days of hard runtime the picture clearly shows: > > AG> > Wired minus ARC = 11..12Gb, ARC grow and shrinks in 80-87Gb range and the > > AG> > system runs just fine. > > AG> > > > AG> > So what shall we do with L2ARC leakage ? > > AG> > > AG> > > AG> Could you please try this patch > > AG> http://cr.illumos.org/~webrev/skiselkov/3995/illumos-gate.patch ? > > AG> > > > > While applying path to curent version of arc.c (r260622) I'm found next truble with compilation > > > > olaris/uts/common/fs/zfs/arc.c -o arc.o > > /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4628:18: error: use of > > undeclared identifier 'abl2' > > trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr, > > ^ > > 1 error generated. > > *** Error code 1 > > > > > > the code is - > > > > if (zio->io_error != 0) { > > /* > > * Error - drop L2ARC entry. > > */ > > list_remove(buflist, ab); > > ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize); > > ab->b_l2hdr = NULL; > > trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr, > > ab->b_size, 0); > > kmem_free(l2hdr, sizeof (l2arc_buf_hdr_t)); > > ARCSTAT_INCR(arcstat_l2_size, -ab->b_size); > > } > > > > > > Looks like it's part is freebsd specific changes. > > Can somebody help with this part of code ? > > > > The first hunk of the patch is renaming of abl2 to l2hdr. > > -- > Andriy Gapon > _______________________________________________ > freebsd-current_at_freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org" > Received on Tue Jan 28 2014 - 08:29:08 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:46 UTC