Re: freebsd perf testing

On Fri, Nov 08, 2013 at 12:28:02PM -0800, Julian Elischer wrote:
> On 11/8/13, 1:54 AM, Olivier Cochard-Labbé wrote:
> > On Fri, Nov 8, 2013 at 3:02 AM, Julian Elischer <julian_at_elischer.org 
> > <mailto:julian_at_elischer.org>>wrote:
> >
> >     Some time ago someone showed some freebsd performance graphs
> >     graphed against time.
> >     He had them up on a website that was updated each day or so.
> >
> >     I think they were network perf tests but I'm not sure.
> >     He indicated that he was going to continue the daily testing
> >     but I've not seen any mention of them since.
> >
> >     If you know who that was or how to find him let me (or gnn) know...
> >
> >
> > Hi Julian,
> >
> > Perhaps you are referring to my network performance graphs on this 
> > thread:
> > http://lists.freebsd.org/pipermail/freebsd-current/2013-April/041323.html
> >
> 
> yes, you are the person we are looking for.
> In yesterday's 'vendor summit' we were discussing performance 
> monitoring and your methodology was cited as one worth looking at.
> 
> The idea of graphing the output of various performance tests against 
> svn commit number is a very good one.
> I thonk it migh teven be worth doing these tests daily, and putting 
> the output onto a web site, showing the last month, the last year and 
> the whole range.
> it would even be interesting to put out 'xplot' files so that people 
> can zoom in and out using xplot to see exactly which revision was 
> responsinble for reversions or problems.
> 
> George..  this is what we mentioned at the meeting yesterday.
> 
> Julian
> 

As it happens I've been thinking over a design for something along these
lines recently. It's just some ideas at the moment but it might be of
interest to others. Forgive me; it's a long E-mail and it gets a bit
pie-in-the-sky too.

I was prompted to think about the problem in the first place because I
read commit mail and I see performance related changes going into the
tree from time to time. These changes often do not come with any
specific data and when they do its normally quite narrow in focus. For
instance, an organisation contributes performance improvements specific
to their workloads and without interest in anyone elses (fair enough).

Initially, what I wanted was a way of viewing how performance changed
for a number of workloads on a commit by commit basis. This sounds very
much like what you are after.

Anyway, after thinking about this for sometime it occurred to me that
much of the infrastructure required to do performance testing could be
generalised to all sorts of software experiments. E.g. software builds,
regression tests, and so on. So, my first conclusion was: build an
experimentation framework within which performance is one aspect.

Having decided this, I thought about the scope of experiments I wanted
to make. For instance, it would be good to test at least every supported
platform. On top of that I would like to be able to vary the relevant
configuration options too. Taking the product of commit, platform,
n-configuration options (not to mention compilers, etc...) you start to
get some pretty big numbers. The numbers grow far too fast and no person
or even organisation could feasibly cover the hardware resources
required to test every permutation. This led me to my next conclusion:
build a distributed system that allows for anyone to contribute their
hardware to the cause. Collectively the project, vendors, and users
could tackle a big chunk of this.

My rough sketch for how this would work is as follows. A bootable USB
image would be made for all platforms. This would boot up, connect to
the network and checkout a repository. The first phase of the process
would be to profile what the host can offer. For example, we might have
experiments that require four identical hard drives, or a particular CPU
type, and so on. Shell scripts or short programmes would be written,
e.g. "has-atom-cpu", with these returning either 1 or 0.

The results of this profiling would be submitted to a service. The
service matches the host with available experiments based on its
particular capabilities and current experimental priorities laid down by
the developers. A priority system would allow for the system to be
controlled precisely. If, for instance, major work is done to the VM
subsystem, relevant experiments could be prioritised over others for a
period.

Once a decision on the experiment to conduct has been made, the relevant
image must be deployed to the system. Free space on the USB device would
be used a staging area, with a scripted installation occurring after
reboot. The images would need to be built somewhere, since it doesn't
make sense to rebuild the system endlessly, especially if we're
including low-powered embedded devices (which we should be). One
possible solution to this would be to use more powerful contributed
hosts to cross-build images and make them available for download.

Finally, the experiments would be conducted. Data produced would be
submitted back to the project using another service where it could be
processed and analysed. To keep things flexible this would just consist
of a bunch of flat files, rather than trying to find some standardised,
one-size-fits all format. Statistics and graphics could be performed on
the data with R/Julia/etc. In particular I imagined DTrace scripts being
attached to experiments so that specific data can be collected. If
something warranting further investigation is found the experiment could
be amended with additional scripts, allowing developers to drill down
into issues.

After some time the process repeats with new image deployed and new
experiments conducted. I envisage some means of identifying individual
hosts so that a developer could repeat the same experiment on the same
host if desired.

Among the many potential problems with this plan, a big one is how would
we protect contributors privacy and security whilst still having a
realistic test environment? I guess the only way to do this would be to
(1) tell users that they should treat the system as if its hacked and
put it in its own network, (2) prevent the experiment images from
accessing anything besides FreeBSD.org.

In relation to network performance, this might not be much good, since
multiple hosts might be necessary. It might be possible to build that
into the design too but it's already more than complicated enough.

Anyhow, I think such a facility could be an asset if could be built. I
may try and put this together, but I've committed myself to enough
things already recently to take this any further at the moment. I'd be
interested to hear what people think, naturally.

--sym