Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

On Wed, Dec 21, 2011 at 1:49 PM, Johan Hendriks <joh.hendriks_at_gmail.com> wrote:
> Nice page, but one thing i do not get is the following.
>
> [quote]
> If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC 4.7
> then the results are unlikely to tell you anything meaningful about FreeBSD
> vs Ubuntu.
> [/quote]
>
> That is a little strange in my opinion.
> It tells me that FreeBSD falls more and more behind on Linux.
> The reason is or could be that FreeBSD cannot or will not include GCC 4.7
> and that FreeBSD will not be on par with Linux anymore.

When benchmarking two systems, you need to make sure that everything
possible is the same (constants) and that the only differences between
the two systems are what you want to benchmark (variables).

For example, if you want to compare the performance of GCC-compiled
binaries, then you would use the same hardware host, the same OS
install, the same source code, and only change the compiler versions
used to compile the "benchmark" binaries.  That way, the only variable
is "version of GCC", everything else is constant, and thus the
benchmark is actually testing the performance of GCC.

Likewise, if you want to benchmark the performance of two OSes, you
need to eliminate as many variables as possible:
  - same hardware
  - running the same benchmark binaries
  - using the same versions of GCC
  - using the same filesystems
  - etc
That gives you the starting point.

Then, you modify one of the constants above, and re-run the benchmarks.

Then you modify one more of the constants above, and re-run the benchmarks.

Etc.  Each time, you vary only 1 thing, so that you can measure the
impact of that *ONE* thing.

Comparing "random binary compile with GCC X on FreeBSD Y on filesystem
Z on hardware config A" against "random binary built with GCC Q on
Linux R on fileystem S on hardware config B" doesn't show anything.
Was the performance difference due to hardware?  Filesystem?  OS?  GCC
version? Something else?

You can't use a shotgun the thread a needle.  :)

> And what if in the future LLVM gets on par with Linux, is it stil fair to
> compare FreeBSD with Linux?

Then you add "compiler suite" to the list of variables, and you make
it a constant in the first run, and then vary it one piece at a time
in later runs, to isolate whether or not it affects performance.

In order to do a proper comparison of any two "things", you have to
first make them as equal as possible, and then vary things one bit at
a time until you are at the "default" configuration for each.  Only
then can you really, truly, empirically say why A is
better/faster/more-uber than B.

Unfortunately, doing it "right" requires a lot of time, effort, time,
and more time.

-- 
Freddie Cash
fjwcash_at_gmail.com