Re: ZFS: unlimited arc cache growth?

On Fri, 17 Apr 2009 16:18:17 +0200 Bernd Walter
<ticso_at_cicely7.cicely.de> wrote:

> On Fri, Apr 17, 2009 at 02:50:24PM +0200, Alexander Leidinger wrote:
> > Hi,
> > 
> > to fs_at_, please CC me, as I'm not subscribed.
> > 
> > I monitored (by hand) a while the sysctls
> > kstat.zfs.misc.arcstats.size and kstat.zfs.misc.arcstats.hdr_size.
> > Both grow way higher (at some point I've seen more than 500M) than
> > what I have configured in vfs.zfs.arc_max (40M).
> 
> My understanding about this is the following:
> vfs.zfs.arc_min/max are not used as min max values.
> They are used as high/low watermarks.
> If arc is more than max the arc a thread is triggered to reduce the
> arc cache until min, but in the meantime other threads can still grow
> arc so there is a race between them.

500M (more than 10 times my max) after a night seems to be a big race...

> > After a while FS operations (e.g. pkgdb -F with about 900
> > packages... my specific workload is the fixup of gnome packages
> > after the removal of the obsolete libusb port) get very slow (in my
> > specific example I let the pkgdb run several times over night and
> > it still is not finished).
> 
> I've seen many workloads were prefetching can saturate disks without
> ever being used.
> You might want to try disabling prefetch.
> Of course prefetching also grows arc.

Prefetching is already disabled in this case.

> > The big problem with this is, that at some point in time the
> > machine reboots (panic, page fault, page not present, during a
> > fork1). I have the impression (beware, I have a watchdog
> > configured, as I don't know if a triggered WD would cause the same
> > panic, the following is just a guess) that I run out of memory of
> > some kind (I have 1G RAM, i386, max kmem size 700M). I restarted
> > pkgdb several times after a reboot, and it continues to process the
> > libusb removal, but hey, this is anoying.
> 
> With just 700M kmem you should set arc values extremly small and
> avoid anything which can quickly grow it.
> Unfortunately accessing many small files is a know arc filling
> workload. Activating vfs.zfs.cache_flush_disable can help speeding up
> arc decreasing, with the obvous risks of course...

I have this:
---snip---
vfs.zfs.prefetch_disable=1
vm.kmem_size="700M"
vm.kmem_size_max="700M"
vfs.zfs.arc_max="40M"
vfs.zfs.vdev.cache.size="5M"
vfs.zfs.vdev.cache.bshift="13"  # device read ahead: 8k
vfs.zfs.vdev.max_pending="6"    # congruent request to the device, + for NCQ
---snip---

Bye,
Alexander.