Re: ZFS: unlimited arc cache growth?

On Apr 18, 2009, at 5:17 PM, Ben Kelly wrote:
> After the rsync completed my machine slowly evicts buffers until its  
> back down to about twice arc_c.  There was one case, however, where  
> I saw it stop at about four times arc_c.  In that case it was  
> failing to evict buffers due to a missed lock.  Its not clear yet if  
> it was a buffer lock or hash lock.  When this happens you'll see the  
> arcstats.mutex_missed sysctl go up.  I'm going to see if I can track  
> down why this is occuring under idle conditions.  That seems  
> suspicious to me.

Sorry to reply to my own mail, but I found some more information I  
thought I would share.

First, the missed mutex problem was an error on my part.  I had  
accidentally deleted a rather important line when I was instrumenting  
the code earlier.  Once this was replaced that missed mutex count  
dropped back to a more reasonable level.

Next, the arcstats.size value is not strictly the amount of cached  
data.  It represents a combination of cached buffers, actively  
referenced buffers, and "other" data.  In this case "other" data is  
things like dnode structures that are directly allocated using  
kmem_cache_alloc() and simply tacked on to the ARC accounting variable  
using arc_space_consume().  At this point I don't think the ARC has a  
way of signaling these "other" data users of memory pressure.

The actual amount of memory the ARC has cached that can actually be  
freed is limited to buffers it internally allocated that have zero  
active references.  This consists of the data and metadata lists for  
the MRU and MFU caches.

On my server right now I have an arc_c_max of about 40MB.  After  
running a simple find(1) over /usr/src I ended up with the following  
memory usage:

   arcstats.size = 132MB
   anonymous inflight buffers = 212KB
   MRU referenced buffers = 80MB
   MFU referenced buffers = 1KB
   dbuf structure "other" data = 8MB
   dnode structure "other" data = 25MB
   unknown "other" data (probably dbuf related) ~= 18MB
   evictable buffer data = 3KB

So right now the ARC has done the best it can to free up data.  If you  
define the cache as storing only inactive data, then basically the ARC  
has emptied the cache completely.  This just isn't visible from the  
exported arcstats.size variable.

I guess there is some question as to whether data is being referenced  
longer than it needs to be by outside consumers.

Anyway, just thought I would share what I found.  At this point it  
doesn't look like tweaking limits will really help.  Also, my previous  
idea that the inactive buffers were being prevented from eviction for  
too long was incorrect.

If anyone is interested I can put together a patch that exports the  
amount of evictable data in the cache.

- Ben