Re: ZFS slowness (not using cache?) (was: ZFS hangs)

On Sun, 04 Nov 2007, Peter Schuller wrote:

> > For example, pkg_delete seems to be _extremely_ slow and ^T reports that
> > it is stuck waiting on zfs:(&zio->io_cv) for an unreasonable (IMO) amount
> > of time.
> 
> FWIW, I have seen pkg_install (and possibly other pkg_* tools) being extremely 
> slow seemingly as a result of the active set of files it touches exceededing 
> the amount cached. In particular I had this problem after converting to ZFS, 
> but prior to switching to amd64 and more RAM.
> 
> It would sit and churn on disk I/O forever, entirely seek bound. Tracing the 
> processes showed it traversing the package database over and over (presumably 
> recursively following dependencies or some such). So the same files were 
> touched any number of times. As a result, with too little cached, runtime 
> exploded (it took hours and hours upgrading my desktop using *binary* 
> pre-built packages because the larger packages with a lot of dependencies 
> would take forever to install and delete).

Hmm.  That prompted me to have a look at the arcstats, and I'm now rather
confused.  It seems to have plenty of cache free, but not actually using
it properly.

I'm running i386 for now (on an amd64 motherboard), with 4GB of RAM
(~3.5Gb usable) and the following settings in loader.conf:

geom_stripe_load="YES"
geom_label_load="YES"
zfs_load="YES"
snd_driver_load="YES"
nvidia_load="YES"

hw.ata.atapi_dma="1"
kern.maxfiles="25000"
kern.ipc.shmmax=67108864
kern.ipc.shmall=32768

kern.maxdsiz="900M"
vm.kmem_size=1450M
vfs.zfs.arc_max=500M
vfs.root.mountfrom="zfs:tank/i386_root"

This results in (sysctl):

vm.kmem_size_scale: 3
vm.kmem_size_max: 335544320  # I have just noticed this is about 300MB, 
                             # far lower than vm.kmem_size.  Is that a problem?
vm.kmem_size_min: 0
vm.kmem_size: 1520435200

vfs.zfs.arc_min: 47513600
vfs.zfs.arc_max: 524288000
kstat.zfs.misc.arcstats.c_min: 47513600
kstat.zfs.misc.arcstats.c_max: 524288000
kstat.zfs.misc.arcstats.size: 86967808

Regardless of what I do, I can't seem to get arcstats.size to exceed 100MB.
It initially goes up with disk usage, but then starts to drop again, causing
it to hold steady around 70-90MB.  Something seems to be agressively pushing
data out of the cache, despite it being <20% full, which seems rather fishy.
Does anyone know what's going on?

-- 
David Taylor