Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

On 22.12.11 11:56, Igor Mozolevsky wrote:
> On 22 December 2011 05:54, Daniel Kalchev<daniel_at_digsys.bg>  wrote:
>>   Of course, it is meaningless, the Ferrari will lose big time in the fuel
>> consumption comparison! I believe it will also lose the price comparison as
>> well. Not to speak the availability comparison.
> That's an oxymoron, right? The comparison cannot be meaningless---the
> reality is F430 will indeed use up more fuel than Prius. If a
> benchmark demonstrates a true reality, how can that benchmark be
> possibly meaningless??? Same benchmark might be irrelevant to someone
> who wants to know how fast they can get from A to B, but irrelevant is
> not a synonym for meaningless!

That benchmark is especially meaningless and a waste of time, because by 
design the Prius is constructed to consume 'less' fuel at the cost of 
lower engine power and the Ferrari is designed to waste fuel for the 
sake of high engine power.
Of course, you can compare them, but this is not exactly benchmark.

As for how fast to get from point A to point B. If you observe speed 
limits, that will depend only on the pilot, no? :)
Both cars are sufficiently faster than the imposed speed limits.

The same can be said for the FreeBSD and the Linux platforms. Today. 
Years ago, Linux was much worse, but they.. hm.. learned. :)
On commodity hardware, you can expect about the same results from both 
OS. There will be differences due to drivers, different optimizations etc.
On very specific hardware, such as systems with many CPUs and lots of 
memory, you may see one better than the other -- this in most cases will 
be relevant to tuning, but also to overall system architecture.

>
>> Any 'benchmark' has a goal. You first define the goal and then measure how
>> different contenders achieve it. Reaching the goal may have several
>> measurable metrics, that you will use to later declare the winner in each.
>> Besides, you need to define a baseline and be aware of what theoretical
>> max/min values are possible.
> Treating a benchmark as a binary win/lose is rather naive, it's not a
> competition, and (I hope) no serious person ever does that. A proper
> benchmark shows true strength and weaknesses so than a well-informed
> intelligent decision can be taken by an individual according to that
> individual's needs. The caveat, of course, is making your methodology
> clear and methods repeatable!

Err... a benchmark produces metrics. It does not produce conclusions. Or 
at least, should not :)
It takes context and understanding of both the subject and methodology 
used to draw any conclusion out of particular benchmark.

No benchmark shows strengths and weaknesses -- these are subject of 
interpretation and any 'score' you have in a benchmark might be the 
result of poor benchmark design and/or implementation.

You may make an very "scientific", well documented and repeatable 
benchmark, such as this one:

time dd if=/dev/zero of=/dev/null

.. then optimize your particular OS to run it at the highest possible 
rate... and so what? Do you know what this benchmark measures? :)

Daniel