Re: ZFS tuning tips?

Craig Boston schrieb:
> Possibly related to reducing maxvnodes, I'm running into a strange
> problem that appears to be cache-related.  I noticed that during
> port installs, the "Registering installation" phase started taking
> forever.  I tracked it down to pkg_info -qO [name] going very slow and
> causing a _lot_ of disk access.
>
> It seems like the entries in /var/db/pkg were not being cached (or
> rather some old stuff wasn't being purged to make room for it).  Doing
> an umount /usr/obj seemed to help quite a bit (and caused vfs.numvnodes
> to drop from ~45000 to 4000), but it eventually happened again.  What
> perplexes me is that I still had about 1700M Free memory that should
> have been available for caching.

I have found that ZFS prefetch is killing performance on my test
server (which is just an old P3/733 with 512MB RAM). The prefetch
code appears to *always* extend the region to be read, not only in
case of sequential reads (as our UFS code does under control of the
vfs.read_max sysctl tunable).

Therefor I have:

	vfs.zfs.prefetch_disable="1"

in my "/boot/loader.conf", and it helps a lot!


All drives perform zero-cost read-ahead today by placing sectors
in reverse order and starting to read from the target track as
soon as the head is settled, stopping when the first requested
sector has been read (remember: sectors are in reverse order).

Since the drives cache data from the last 10 to 20 requests (at
least), another read has a good chance to find sectors following
the initial request in the drives cache, if no ZFS read-ahead is
performed. ZFS prefetch seems to extend each read request to the
value of recordsize (e.g. 128KB), even for short direct accesses
(e.g. to a DBM file). This may require an extra revolution of the
disk (which the read ahead issued by the drive's firmware avoids
by just reading what happens to be found between the point where
the head is settled and able to read data and the final requested
sector).

As long as ZFS prefetch is not adaptive, I do not consider it a
good idea to have it enabled by default. But I have not performed
reproducible benchmark runs to provide hard numbers, just observed
the qualitative difference (i.e. much reduced load and better
response after ZFS prefetch was disabled).

Regards, STefan