Re: ZFS: unlimited arc cache growth?

On Fri, Apr 17, 2009 at 02:50:24PM +0200, Alexander Leidinger wrote:
> Hi,
> 
> to fs_at_, please CC me, as I'm not subscribed.
> 
> I monitored (by hand) a while the sysctls kstat.zfs.misc.arcstats.size  
> and kstat.zfs.misc.arcstats.hdr_size. Both grow way higher (at some  
> point I've seen more than 500M) than what I have configured in  
> vfs.zfs.arc_max (40M).

My understanding about this is the following:
vfs.zfs.arc_min/max are not used as min max values.
They are used as high/low watermarks.
If arc is more than max the arc a thread is triggered to reduce the
arc cache until min, but in the meantime other threads can still grow
arc so there is a race between them.

> After a while FS operations (e.g. pkgdb -F with about 900 packages...  
> my specific workload is the fixup of gnome packages after the removal  
> of the obsolete libusb port) get very slow (in my specific example I  
> let the pkgdb run several times over night and it still is not  
> finished).

I've seen many workloads were prefetching can saturate disks without
ever being used.
You might want to try disabling prefetch.
Of course prefetching also grows arc.

> The big problem with this is, that at some point in time the machine  
> reboots (panic, page fault, page not present, during a fork1). I have  
> the impression (beware, I have a watchdog configured, as I don't know  
> if a triggered WD would cause the same panic, the following is just a  
> guess) that I run out of memory of some kind (I have 1G RAM, i386, max  
> kmem size 700M). I restarted  pkgdb several times after a reboot, and  
> it continues to process the libusb removal, but hey, this is anoying.

With just 700M kmem you should set arc values extremly small and
avoid anything which can quickly grow it.
Unfortunately accessing many small files is a know arc filling workload.
Activating vfs.zfs.cache_flush_disable can help speeding up arc decreasing,
with the obvous risks of course...

> Does someone see something similar to what I describe (mainly the  
> growth of the arc cache way beyond what is configured)? Anyone with  
> some ideas what to try?

In my opinion the watermark mechanism can work as it is, but there should
be a forced max - currently there is no garantied limit at all.
Nevertheless it is up for the people which know the code to decide.

-- 
B.Walter <bernd_at_bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.