Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

Stefan Esser schreef:
> Am 21.12.2011 22:49, schrieb Johan Hendriks:
>> Nice page, but one thing i do not get is the following.
>>
>> [quote]
>> If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC
>> 4.7 then the results are unlikely to tell you anything meaningful about
>> FreeBSD vs Ubuntu.
>> [/quote]
>>
>> That is a little strange in my opinion.
>> It tells me that FreeBSD falls more and more behind on Linux.
>> The reason is or could be that FreeBSD cannot or will not include GCC
>> 4.7 and that FreeBSD will not be on par with Linux anymore.
>> To compare it with Formula1 cars.
>> If Mercedes decide to use the engine from 2 seasons back (the engine
>> version 4.2.1) in there 2012 car, and Ferrari uses there new Engine
>> (version 4.7).
>> Can we not compare them anymore because of the decission from Mercedes
>> to use the old engine?
>> No we just say, if you want to win a race, get the Ferrari.
>>
>> It is the reallity, FreeBSD uses 4.2.1 as there compiler!!!
> As has been pointed out by others, FreeBSD ships with gcc-4.2.1 (with
> some local modifications and fixes) as the system compiler.

>> If you tune up FreeBSD to use the GCC 4.7 compiler, or downgrade linux
>> to 4.2.1, then that will tell me nothing about FreeBSD vs Linux.
> The gcc version distributed with FreeBSD was chosen for license reasons,
> not for technical reasons. If you are OK with installing a GPLv3
> licensed compiler on your systems, then just do it and take advantage of
> the improved code generated by it.
It does not matter what the decission is to use the old compiler, it is 
a fact that the base comes with 4.2.x
Does that mean we can not compare/benchmark against other distributions 
because they use GPLv3 stuff.
No, i want to know where standard released FreeBSD stands against 
standard released Linux distributions.
If you compare benchmark userland applictions, then it is fair to use 
the latest compiler for the userland software also on FreeBSD.
But what if the ports tree defaults to LLVM, then again we want to know 
what FreeBSD does against a Linux distribution.
Why because that is what most of us will be using...!!

If we start to compile all the ports with gcc 4.7 to be on par in 
comparising and benchmarking, why spend all the time getting LLVM as the 
default compiler for ports also?
Why not take that effort into making the WHOLE ports tree to compile 
with GCC4.7?

Reason, because FreeBSD goes the LLVM route. That is a decission FreeBSD 
is making!
And that choise will be the FreeBSD that is used in comparising and 
benchmarks on the net , not the utterly overcopiled and tuned FreeBSD 
against stock Ubuntu or whatever Linux distribution.

If it is a good or bad choice! That we will see in the 
comparising/benchmarks we will be seeing when that time comes.

Same goes for the scheduler! and all the other subsystems FreeBSD has 
choosen, that makes FreeBSD.

>
>> I my opinion, you benchmark the latest release of Linux, FreeBSD,
>> Solaris, Windows and whatever OS you want to compare!
> As you probably know, Linux is just the kernel and the distributions add
> user space programs, including a compiler. You can easily create a
> "FreeBSD distribution" with more advanced compiler and use or even sell
> it. But the FreeBSD project was cautious to not heavily depend on a
> GPLv3 compiler (for reasons openly discussed at the time this decision
> was made).
I know Linux is a kernel, re read Linux as Linux Distribution!
Yes you can use a more advanced compiler on FreeBSD, BUT you can do that 
on Linux also ,so where do you stop?
Are you going to spend a month to compare a fullly tuned up FreeBSD 
system against a Linux distribution?
No because the users will not spend months tuning and recompile there 
servers.
They use the FreeBSD version that comes with the CD!
And that we want to compare/benchmark against a Linux distribution.

>> You want to benchmark the release and not a tuned version against a
>> standard version.
>> And that in general are the versions most of us users will use.
> If you compare operating systems from a technical point of view, then
> you'll be interested in relative performance of algorithms and methods
> chosen. This is best achieved by using the same compiler for each of the
> candidates.
>
> If you compare performance from a user point of view, you are correct
> that performance delivered out of the box (without complicated tuning)
> may be, what counts for most users. But those users that depend on best
> performance e.g. for a FreeBSD based embedded product or a data center,
> may tune the system, including compilation with a newer compiler than
> the system default.
>
>> And what if in the future LLVM gets on par with Linux, is it stil fair
>> to compare FreeBSD with Linux?
> You can always compare anything with whatever you like (even apples with
> oranges), but you need to be aware of what you compare and what your
> goals are, to be able to draw reasonable conclusions.
>
> If you want to test out of the box performance, then test with system
> compilers (or just those binaries delivered with the system).
>
> If you want to test for code efficiency or scalability, then use the
> same compilers for each system under test to remove differences
> introduced by the compilers (which are an external component not
> developed by the FreeBSD people).
>
>> Or do we say, well we are on par, but it is not fair, yes we used the
>> latest releases, but you can not blame Linux because they are still
>> using GCC.
> Depends on what you want or need to measure ...
I am mainly talking about the out of the box comparison, because that is 
what most sites will do when doing a benchmark.
they want to know if the the all new version of FreeBSD can compete with 
the latest release of Ubuntu.
That are the benchmarks you will see on the net.


>> No what we will see then are haleluja blogs that FreeBSD is on par with
>> Linux.
> Such blog messages are not common in the FreeBSD community. FreeBSD used
> to have big technical and performance advantages when Linux was young,
> but even then, there was technical discussion between camps (and many
> concepts were implemented in Linux based on BSD examples; I have taken
> part in such discussions myself, some 15 to 20 years back).
Well not  quite,  i remember the mysql on SCHED_ULE reports, and they 
where quit haleluja about the whole thing.
Not wrong at all, we are all human and like to compete and even more 
like to be on top in what we do.


>> For me peformance is not a show stopper, and for the most of us i think
>> it is not.
>> FreeBSD for me is a clean system that does the job perfect and has a
>> very helpful community.
> Well, this are valid aspects, too, and very hard to with benchmarks ;-)
>
> Regards, STefan
I am not saying FreeBSD is bad and is not performing, i just do not 
understand why FreeBSD is doing all these things to look good.
Again, most comparisings/benchmarks will be stock *BSD vs a stock Linux 
distribution.
If we want to look good, FreeBSD must make sure there released version 
gets on par!
We all know FreeBSD is quite conservative with its default settings.
Maybe we need to set some default settings less conservative.
But i do not know if that is good just for the numbers!

Regards,
Johan