Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

Hi,

I extended the gcc part a little bit to make it a little bit more clear when it matters.

Bye,
Alexander.

-- 
Send via an Android device, please forgive brevity and typographic and spelling errors. 

Stefan Esser <se_at_freebsd.org> hat geschrieben:Am 21.12.2011 22:49, schrieb Johan Hendriks:
> Nice page, but one thing i do not get is the following.
> 
> [quote]
> If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC
> 4.7 then the results are unlikely to tell you anything meaningful about
> FreeBSD vs Ubuntu.
> [/quote]
> 
> That is a little strange in my opinion.
> It tells me that FreeBSD falls more and more behind on Linux.
> The reason is or could be that FreeBSD cannot or will not include GCC
> 4.7 and that FreeBSD will not be on par with Linux anymore.
> To compare it with Formula1 cars.
> If Mercedes decide to use the engine from 2 seasons back (the engine
> version 4.2.1) in there 2012 car, and Ferrari uses there new Engine
> (version 4.7).
> Can we not compare them anymore because of the decission from Mercedes
> to use the old engine?
> No we just say, if you want to win a race, get the Ferrari.
> 
> It is the reallity, FreeBSD uses 4.2.1 as there compiler!!!

As has been pointed out by others, FreeBSD ships with gcc-4.2.1 (with
some local modifications and fixes) as the system compiler.

> If you tune up FreeBSD to use the GCC 4.7 compiler, or downgrade linux
> to 4.2.1, then that will tell me nothing about FreeBSD vs Linux.

The gcc version distributed with FreeBSD was chosen for license reasons,
not for technical reasons. If you are OK with installing a GPLv3
licensed compiler on your systems, then just do it and take advantage of
the improved code generated by it.

> I my opinion, you benchmark the latest release of Linux, FreeBSD,
> Solaris, Windows and whatever OS you want to compare!

As you probably know, Linux is just the kernel and the distributions add
user space programs, including a compiler. You can easily create a
"FreeBSD distribution" with more advanced compiler and use or even sell
it. But the FreeBSD project was cautious to not heavily depend on a
GPLv3 compiler (for reasons openly discussed at the time this decision
was made).

> You want to benchmark the release and not a tuned version against a
> standard version.
> And that in general are the versions most of us users will use.

If you compare operating systems from a technical point of view, then
you'll be interested in relative performance of algorithms and methods
chosen. This is best achieved by using the same compiler for each of the
candidates.

If you compare performance from a user point of view, you are correct
that performance delivered out of the box (without complicated tuning)
may be, what counts for most users. But those users that depend on best
performance e.g. for a FreeBSD based embedded product or a data center,
may tune the system, including compilation with a newer compiler than
the system default.

> And what if in the future LLVM gets on par with Linux, is it stil fair
> to compare FreeBSD with Linux?

You can always compare anything with whatever you like (even apples with
oranges), but you need to be aware of what you compare and what your
goals are, to be able to draw reasonable conclusions.

If you want to test out of the box performance, then test with system
compilers (or just those binaries delivered with the system).

If you want to test for code efficiency or scalability, then use the
same compilers for each system under test to remove differences
introduced by the compilers (which are an external component not
developed by the FreeBSD people).

> Or do we say, well we are on par, but it is not fair, yes we used the
> latest releases, but you can not blame Linux because they are still
> using GCC.

Depends on what you want or need to measure ...

> No what we will see then are haleluja blogs that FreeBSD is on par with
> Linux.

Such blog messages are not common in the FreeBSD community. FreeBSD used
to have big technical and performance advantages when Linux was young,
but even then, there was technical discussion between camps (and many
concepts were implemented in Linux based on BSD examples; I have taken
part in such discussions myself, some 15 to 20 years back).

> For me peformance is not a show stopper, and for the most of us i think
> it is not.
> FreeBSD for me is a clean system that does the job perfect and has a
> very helpful community.

Well, this are valid aspects, too, and very hard to with benchmarks ;-)

Regards, STefan