Re: Feedback for performance tracker

Erik Cederstrand wrote:
> Hi!
> 
> This autumn, we have decided to grab the Performance Tracker entry[1]
> from the project ideas page and give it a spin as a subject for our
> thesis at the IT University of Copenhagen. The tracker intends to fill a
> hole in the range of tinderboxes and automatic stress/regression tests
> that FreeBSD already has.
> 
> The initial idea is to have a small collection of servers constantly
> performing benchmarks and publishing the results to a server with a web
> interface.
> 
> Before we start coding, we'd like to ask a couple of questions:
> 
> 1) Which benchmarks would you like to see being run?

- A database benchmark (multithreaded)
- File system benchmarks (IOzone, postmark, bonnie++)
- Network performance benchmark (someone else can mention which ones)
- Some kind of compiling test (buildworld for a specific release maybe?)
- NFS server/client for a specific workset
- USB throughput
- Firewire throughput
- 802.11 throughput (eek! Hey, it's a wishlist, right? :D )


> 2) Which tests do you perform regularly, which the tracker could automate?

- database benchmarks
- File system benchmarks (iozone and postmark)
- NFS (see file system benchmarks)


> 3) Which features in the web interface would you find most helpful?


It would be great to be able to see the various recorded numbers based 
on release, and also based on time.  So you can see the difference (a 
graph would be the best) between say March 1st and August 1st, and 
compare.  It would also be nice if this setup was portable enough for 
developers to run part or all of this suite in their labs, so they can 
watch the performance numbers change (or not) depending on what they do, 
*before* they commit it.


> Also, we'd greatly appreciate pointers to previous work in the area.
> 
> We welcome all comments and suggestions, but please bear in mind that we
> only have around 3 months full-time to develop the tracker.


Sounds like a great project - good luck!

Eric