Re: ZFS: unlimited arc cache growth?

On Apr 17, 2009, at 8:50 AM, Alexander Leidinger wrote:
> to fs_at_, please CC me, as I'm not subscribed.
>
> I monitored (by hand) a while the sysctls  
> kstat.zfs.misc.arcstats.size and kstat.zfs.misc.arcstats.hdr_size.  
> Both grow way higher (at some point I've seen more than 500M) than  
> what I have configured in vfs.zfs.arc_max (40M).
>
> After a while FS operations (e.g. pkgdb -F with about 900  
> packages... my specific workload is the fixup of gnome packages  
> after the removal of the obsolete libusb port) get very slow (in my  
> specific example I let the pkgdb run several times over night and it  
> still is not finished).
>
> The big problem with this is, that at some point in time the machine  
> reboots (panic, page fault, page not present, during a fork1). I  
> have the impression (beware, I have a watchdog configured, as I  
> don't know if a triggered WD would cause the same panic, the  
> following is just a guess) that I run out of memory of some kind (I  
> have 1G RAM, i386, max kmem size 700M). I restarted  pkgdb several  
> times after a reboot, and it continues to process the libusb  
> removal, but hey, this is anoying.
>
> Does someone see something similar to what I describe (mainly the  
> growth of the arc cache way beyond what is configured)? Anyone with  
> some ideas what to try?

Can you provide the rest of the arcstats from sysctl?  Also, does your  
arc_reclaim_thread process get any cycles when this problem occurs?   
What happens if you kill the pkgdb -F manually before it completes?   
Does the arc cache size come back down or is it stuck at the  
abnormally high level?

At first glance it looks like the tunable limits the value of the  
arc_c target value, but that appears to only be a soft limit.  There  
is code in there to shrink an ARC that has exceeded its arc_c value.   
It looks like that code is supposed to run from the arc_reclaim_thread.

- Ben