Re: FreeBSD 8.0 Performance (at Phoronix)

On Mon, 30 Nov 2009, David Naylor wrote:

> Phoronix recently published a comparative benchmark[1] for FreeBSD 7.2/8.0 
> against Linux and OpenSolaris.  I would like to bring some of the good and 
> bad to light (in the hopes that the developers with the correct expertise 
> will be intrigued).
>
> The tests were performed with a 'standard' installation of FreeBSD on a 
> Lenovo ThinkPad T61.
>
> I've tried to eliminate tests who's performance is a result of compiler 
> differences and/or 3rd party applications and tests who's statistical 
> significance are not so strong (subjective guess).

It would be nice to see the same tests rerun with more attention to 
well-understood benchmarking methodology (i.e., multiple runs, 
characterization of variance, statistical tests to decide if two results are 
likely to be from the same set, and what statistically significant difference 
exists).  At the very least, three or five runs of each test so we can 
consider spread would be very useful.  However, I agree with your 
characterizaitons that these are the likely areas of issue :-).

For tests that depend on file system namespace operations, seeing a 
measurement comparing ZFS directly would be interesting; likewise, perhaps 
using async file system mounts on FreeBSD without soft updates.  Jeff's 
forthcoming journaling changes would also be interesting to consider, but I'm 
not sure to what extent he'll see performance improvements, or whether it's 
just about the consistency improvements for his SU+J work.

Looking at compiler differences would also make a lot of sense for things like 
compression/ray tracing/etc.  Another area of comparison would simply be the 
version and contents of gzip, particularly it might be interesting to try the 
Linux gzip under FreeBSD's Linux emulation (and for a few other of the 
CPU-centric tests: use a Linux chroot or Debian/kFreeBSD to do side-by-side 
comparisons on the same kernel).

The threaded I/O is interesting and possiby the greatest concern.  We should 
try to reproduce these in "lab conditions" and attempt to understand the 
workloads more.  We do have known "strong" synchronization in the vnode write 
path, such that writes to the same vnode may be serialized in a strong way -- 
it could be that these tests show that up.

Robert

>
> Improvements for FreeBSD 8.0 vs 7.2:
> - 7-Zip Compression (page 3)
> - Timed MAFFT Alignment (page 5)
> - GraphicsMagick (page 5)
> - Threaded IO (64MB Random Write - 32 threads) (page 7)
> - Threaded IO (64MB Read - 32 threads) (page 7)
>
> Regressions for FreeBSD 8.0 vs 7.2:
> - Gzip compressions of a 2GB file (page 3)
> - C-Ray (page 4)
> - Threaded IO (64MB Write - 4 threads) (page 7)
> - Threaded IO (64MB Write - 32 threads) (page 7)
>
> Poor performance relative to Linux and OpenSolaris
> - Threaded IO (especially random writes) (page 7)
> - OpenSSL (RSA 4096bit) (page 8)
> - PostMark (disk transaction) (page 8)
>
> It appears that threaded activity on UFS does not fair well against Linux/ext4
> and OpenSolaris/ZFS.  Phoronix intends to do a comparative test against
> FreeBSD and OpenSolaris on ZFS.
>
> Regards,
>
> David
>
> 1)
> http://www.phoronix.com/scan.php?page=article&item=freebsd8_benchmarks&num=1
>