Re: ZFS slowness (not using cache?)

On Sun, 04 Nov 2007, Kris Kennaway wrote:
> David Taylor wrote:
>>
>>Hmm.  That prompted me to have a look at the arcstats, and I'm now rather
>>confused.  It seems to have plenty of cache free, but not actually using
>>it properly.
>>
>>I'm running i386 for now (on an amd64 motherboard), with 4GB of RAM
>>(~3.5Gb usable) and the following settings in loader.conf:
[snip]
>>
>>kern.maxdsiz="900M"
>>vm.kmem_size=1450M
>>vfs.zfs.arc_max=500M
>>vfs.root.mountfrom="zfs:tank/i386_root"
>>
>>This results in (sysctl):
>>
>>vm.kmem_size_scale: 3
>>vm.kmem_size_max: 335544320  # I have just noticed this is about 300MB, 
>>                             # far lower than vm.kmem_size.  Is that a 
>>                             problem?
> 
> No but you can't set kmem to 1500MB without first increasing the kernel 
> address space (1GB on i386) to some suitably larger value via KVA_PAGES.

Sorry, I should have mentioned that I have KVA_PAGES=512.
 
>>vm.kmem_size_min: 0
>>vm.kmem_size: 1520435200
>>
>>vfs.zfs.arc_min: 47513600
>>vfs.zfs.arc_max: 524288000
>>kstat.zfs.misc.arcstats.c_min: 47513600
>>kstat.zfs.misc.arcstats.c_max: 524288000
>>kstat.zfs.misc.arcstats.size: 86967808
>>
>>Regardless of what I do, I can't seem to get arcstats.size to exceed
>>100MB.  It initially goes up with disk usage, but then starts to
>>drop again, causing it to hold steady around 70-90MB.  Something
>>seems to be agressively pushing data out of the cache, despite it
>>being <20% full, which seems rather fishy.
>>Does anyone know what's going on?
> 
> Try increasing kern.maxvnodes.

Increasing this from 100k to 1M seems to have made no difference.

vfs.numvnodes is now growing to 150k and beyond, but arcstats.size
is still sitting at ~70MB.

-- 
David Taylor