Re: Interesting speed benchmarks

Peter Jeremy <peterjeremy_at_optushome.com.au> wrote:
> On Sat, 2007-Jan-27 11:54:37 -0500, Brian Reichert wrote:
> >On Sat, Jan 27, 2007 at 09:43:52AM +1100, Peter Jeremy wrote:
> >> Note that dump/restore is the only tool that can correctly reproduce
> >> sparse files.  tar, cpio and pax also have filename and file size
> >> restrictions.  I don't think that cpio or pax support ACLs or file
> >> flags.
> >
> >I thought 'star' handled sparse files and all the extra magic?  Not
> >that this is germaine to the topic at hand...
> 
> You can detect a sparse file by comparing the length of the file
> with the number of allocated blocks.  You can't determine whether a
> specific block is all NULLs or not allocated without either bypassing
> the filesystem (which dump does) or using a syscall to retrieve a
> bitmap of allocated blocks (which FreeBSD doesn't have). =20

Both rsync and gtar create a sparse file by not writing
blocks of zeroes given -S option.  You are correct that there
is no special help for reading sparse files.  I recall
pjd_at_freebsd.org saying he plans to implement solaris specific
extensions to lseek() in zfs (lseek's whence arg can take
SEEK_HOLE/SEEK_DATA values, to seek to next null/nonnull
block).  It would be nice if this was done for all
filesystems.  Returning a bitmap is far more work than adding
SEEK_{HOLE,DATA}.

For backups I use vbackup & venti from plan9ports.  Like dump
vbackup knows ufs1 & ufs2 structure to walk the filesytem and
write only live data to venti (a SHA1-addressed block storage
server) so it correctly handles sparse files.  A full backup
is faster than dump.  All subseqeuent backups are incremental
and generally run very fast (for example I can backup a UFS2
system at about 10MB/s to a USB2 ATA disk.  dd shows write
speed of 23MB/s).

You can get at backed up data in a number of different ways:
1) vcat (makes an identical disk image given a snapshot's
   `score')
2) vftp (you can browse/copy files individually)
3) vnfs + vmount (make one or more snapshots accessible in
   filesystem namespace)

AFAIK flags, attributes & ACLs are maintained but are only
available if you use vcat.  See http://swtch.com/plan9ports