ZFS file caching question

If I have, say, 512MB RAM and a 1GB file which is written or read
sequentially on an otherwise idle system, I'd expect the last 512MB (-
epsilon) of the file's pages to be cached in RAM.  This is true for
UFS, but it does not appear to be the case with ZFS (see example
below).

Can somebody explain how the arc cache in ZFS relates to the normal
page cache used by traditional filesystems?  Are ZFS filesystems
cached exclusively in the arc cache, and not in the page cache?  Is
the arc cache per-filesystem, per-pool, or global for ZFS as a whole?

Hmm.. Could this be the cause of the problems with ZFS and mmap'ed files?


Thanks,

Drew



### host has 406MB free before file is written

%vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad8 ad10   in   sy   cs us sy id
 0 0 4  74232K   406M   309   0   1  11  1747 2518   0   0  358  303 3267  0  3 97
%dd if=/dev/zero of=zot bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 21.076331 secs (50945386 bytes/sec)

### host has 26MB free after file is written


%vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad8 ad10   in   sy   cs us sy id
 0 0 6  74232K    26M   303   0   1  11  1707 2565   0   0  359  299 3252  0  3 97

### at least the last 300MB or so are cached, and the disk is not hit
### when reading them.

%dd if=zot of=/dev/null iseek=700 bs=1m 
324+0 records in
324+0 records out
339738624 bytes transferred in 0.372665 secs (911646356 bytes/sec)
%


But with ZFS, this is not true:


### host has 412MB free before file is written
%vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad8 ad10   in   sy   cs us sy id
 0 0 6  84552K   412M   263   0   1   9  1520 2200   0   0  309  263 2854  0  3 97

%dd if=/dev/zero of=zot bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 17.447763 secs (61540372 bytes/sec)
%

### host has 254MB free after file is written

%vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad8 ad10   in   sy   cs us sy id
 0 0 6  84552K   254M   249   0   1   9  1441 2069   0   0  299  251 2776  0  3 97
%dd if=zot of=/dev/null iseek=700 bs=1m
324+0 records in
324+0 records out
339738624 bytes transferred in 4.545527 secs (74741306 bytes/sec)

### the last 300MB of the file are not cached, and the disk is
### hit when reading them

%vmstat
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad8 ad10   in   sy   cs us sy id
 0 0 6  84552K   254M   249   0   1   9  1441 2069   0   0  299  251 2776  0  3 97
%dd if=zot of=/dev/null iseek=700 bs=1m
324+0 records in
324+0 records out
339738624 bytes transferred in 4.545527 secs (74741306 bytes/sec)