Re: posix_fadvise noreuse disables file caching

On Sunday, January 29, 2012 10:08:10 am Tijl Coosemans wrote:
> On Wednesday 25 January 2012 17:29:22 John Baldwin wrote:
> > On Friday, January 20, 2012 2:12:13 pm John Baldwin wrote:
> >> On Thursday, January 19, 2012 11:39:42 am Tijl Coosemans wrote:
> >>> I recently noticed that multimedia/vlc generates a lot of disk IO when
> >>> playing media files. For instance, when playing a 320kbps mp3 gstat
> >>> reports about 1250kBps (=10000kbps). That's quite a lot of overhead.
> >>> 
> >>> It turns out that vlc sets POSIX_FADV_NOREUSE on the entire file and
> >>> reads in chunks of 1028 bytes. FreeBSD implements NOREUSE as if
> >>> O_DIRECT was specified during open(2), i.e. it disables all caching.
> >>> That means every 1028 byte read turns into a 32KiB read (new default
> >>> block size in 9.0) which explains the above numbers.
> >>> 
> >>> I've copied the relevant vlc code below (modules/access/file.c:Open()).
> >>> It's interesting to see that on OSX it sets F_NOCACHE which disables
> >>> caching too, but combined with F_RDAHEAD there's still read-ahead
> >>> caching.
> >>> 
> >>> I don't think POSIX intended for NOREUSE to mean O_DIRECT. It should
> >>> still cache data (and even do read-ahead if F_RDAHEAD is specified),
> >>> and once data is fetched from the cache, it can be marked WONTNEED.
> >> 
> >> POSIX doesn't specify O_DIRECT, so it's not clear what it asks for.
> >> 
> >>> Is it possible to implement it this way, or if not to just ignore
> >>> the NOREUSE hint for now?
> >> 
> >> I think it would be good to improve NOREUSE, though I had sort of
> >> assumed that applications using NOREUSE would do their own buffering
> >> and read full blocks.  We could perhaps reimplement NOREUSE by doing
> >> the equivalent of POSIX_FADV_DONTNEED after each read to free buffers
> >> and pages after the data is copied out to userland.  I also have an
> >> XXX about whether or not NOREUSE should still allow read-ahead as it
> >> isn't very clear what the right thing to do there is.  HP-UX (IIRC)
> >> has an fadvise() that lets you specify multiple policies, so you
> >> could specify both NOREUSE and SEQUENTIAL for a single region to
> >> get read-ahead but still release memory once the data is read once.
> >
> > So I've came up with this untested patch.  It uses
> > VOP_ADVISE(FADV_DONTNEED) after read(2) calls to a NOREUSE region, and
> > leaves read-ahead caching enabled for NOREUSE.  FADV_DONTNEED doesn't
> > do any good really for writes (it only flushes clean buffers), so I've
> > left write(2) operations as using IO_DIRECT still.  Does this sound
> > reasonable?  I've not yet tested this at all:
> 
> The patch drastically improves vlc, but there's still a tiny overhead.
> Without NOREUSE the disk is read in chunks of 128KiB (F_RDAHEAD buffer
> size). With NOREUSE there's an extra transfer of 32KiB (block size).

This is probably because vlc is not reading on block boundaries, so the 
noreuse is throwing away partial blocks at the end of a read that then have to 
be re-read.  We could maybe fix this by making FADV_DONTNEED only throw
away completely-contained blocks rather than completely-contained pages.
However, this will probably result in NOREUSE not actually throwing away
anything at all if an app always reads sub-blocksize chunks.

We could maybe make the case of vlc work ok in this case though by allowing
an extension where you can do 'posix_fadvise(SEQUENTIAL | NOREUSE)', and
in this case we could make the VOP_ADVISE(DONTNEED) in read() use an offset
of 0 rather than the start of the read request.

However, posix_fadvise() really is going to work best if the userland 
application reads aligned FS blocks.

-- 
John Baldwin