Re: Benchmarking

From: Greg 'groggy' Lehey <grog_at_FreeBSD.org>
Date: Thu, 8 Apr 2004 10:48:57 +0930
On Thursday,  8 April 2004 at  1:20:48 +0200, Ivan Voras wrote:
> Kirk Strauser wrote:
>
>> Prepare to be flamed (not by me, but in general).  The tests are littered
>> with comments like:
>>
>>    Missing data in the above table signifies operations that were too fast
>>    to measure correctly by the bonnie++ program.
>
>> People will wonder why you chose not to increase the number of files being
>> tested so as to get meaningful results.  Viewers are left looking at a
>> mostly-empty table where the BSDs dominate the results by a large margin,
>> but are told:
>
> Id really didn't occurr to me until it was already to late (too many
> systems were benchmarked already).

This sounds like another way of saying "I published before I had
finished my strategy".  You should consider retesting.

>
>>    Linux clearly wins the IO throughput test, having a score upto about
>>    130% better than nearest BSD, either by having a better SCSI driver, or
>>    because the system itself is just faster.
>>
>> Since you didn't *publish* any results were Linux won, I'll have to assume
>> that your private data demonstrated a different conclusion.
>
> And what do you call the results of bonnie++ sequential IO output test
> (in the table)? :) This is what I was reffering to in the cited
> text.

Agreed, it looks as if Linux is much better there.  But the per
character tests should have made you wonder what's going on here.
Linux can't possibly be 100 times as fast as FreeBSD, and NetBSD can't
be 4 times as fast again.  I'd assume that you've got some effect
related to how the character I/O is compiled.  I saw similar (but
opposite :-) effects with bonnie a couple of years ago (see
http://www.lemis.com/grog/diary-nov2000.html#30).  As I commented at
the time, something else must be causing the problem.  This is one of
the reasons why I don't like bonnie(++) as a benchmark.

>> Basically, you ran some tests on divergent systems and got some
>> results, but that's about the only conclusion I was able to get
>> from it.
>
> You're perfectly entitled to :)

You have the beginnings of something interesting here, but I have the
feeling you're not done yet.  It would be interesting to examine the
obvious differences and investigate what's really causing them.

Greg
--
Note: I discard all HTML mail unseen.
Finger grog_at_FreeBSD.org for PGP public key.
See complete headers for address and phone numbers.

Received on Wed Apr 07 2004 - 16:19:02 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:50 UTC