Re: freebsd perf testing

Den 10/11/2013 kl. 09.45 skrev Julian Elischer <julian_at_freebsd.org>:
> 
> it would be interesting to know what you did. and what conclusions you came to..

I created a system with a build server, a set of slaves, and a website to collect and publish the benchmark data in graphs and raw data and highlight significant changes.

The build server built FreeBSD and any required ports and benchmark software, and bundled it in installation images complete with a script to run the benchmarks on the slave and push data to the website.

The slaves were dedicated machines that booted via PXE, wiped the hard drive, fetched and installed an image and proceeded to run the specified benchmarks.

The website published graphs, and the graphs were linked to useful related data like raw measurements, slave hardware, commit messages, changed source files and binaries etc.

It actually worked pretty well. I only ran benchmarks over a couple of months of commits, but I did identify some significant regressions in MySQL and some of the unixbench microbenchmarks. I did not spend much time designing the benchmarking strategy, and “symbolics” has many good points there. I did find out that all the current continuous integration / performance benchmark frameworks (Jenkins etc) did not play well with a situation where the entire operating system is reinstalled.

A wide range of useful benchmarks can be collected in just 10-20 minutes, but building FreeBSD and all the ports takes much longer than that (1-2 hours using the default procedure from the handbook). Most of my work went into optimizing build times, installation image sizes and slave installation times, so I could build an image in just 10-15 minutes and use ca. 10MB disk space amortized, and install an image on a slave in just 2 minutes (cheap 2008-era hardware).

But having a complete archive of ready-to-install images for each commit is a real advantage, not just for performance benchmarking. Imagine being able to fetch a VirtualBox disk image for a random SVN commit, booting it and start debugging right away. Or you could write a script to test if a bug is present in a FreeBSD installation, and then let an automated process do a binary search to find the offending commit in maybe less than an hour.

My work was completed in 2008 and would (at least) require updating the scripts to handle the upgrade from GCC to Clang, and from CVS to SVN.

Erik