Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

From: O. Hartmann <ohartman_at_zedat.fu-berlin.de>
Date: Fri, 23 Dec 2011 02:05:38 +0100
On 12/22/11 10:02, Johan Hendriks wrote:
> Stefan Esser schreef:
>> Am 21.12.2011 22:49, schrieb Johan Hendriks:
>>> Nice page, but one thing i do not get is the following.
>>>
>>> [quote]
>>> If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC
>>> 4.7 then the results are unlikely to tell you anything meaningful about
>>> FreeBSD vs Ubuntu.
>>> [/quote]
>>>
>>> That is a little strange in my opinion.
>>> It tells me that FreeBSD falls more and more behind on Linux.
>>> The reason is or could be that FreeBSD cannot or will not include GCC
>>> 4.7 and that FreeBSD will not be on par with Linux anymore.
>>> To compare it with Formula1 cars.
>>> If Mercedes decide to use the engine from 2 seasons back (the engine
>>> version 4.2.1) in there 2012 car, and Ferrari uses there new Engine
>>> (version 4.7).
>>> Can we not compare them anymore because of the decission from Mercedes
>>> to use the old engine?
>>> No we just say, if you want to win a race, get the Ferrari.
>>>
>>> It is the reallity, FreeBSD uses 4.2.1 as there compiler!!!
>> As has been pointed out by others, FreeBSD ships with gcc-4.2.1 (with
>> some local modifications and fixes) as the system compiler.
> 
>>> If you tune up FreeBSD to use the GCC 4.7 compiler, or downgrade linux
>>> to 4.2.1, then that will tell me nothing about FreeBSD vs Linux.
>> The gcc version distributed with FreeBSD was chosen for license reasons,
>> not for technical reasons. If you are OK with installing a GPLv3
>> licensed compiler on your systems, then just do it and take advantage of
>> the improved code generated by it.
> It does not matter what the decission is to use the old compiler, it is
> a fact that the base comes with 4.2.x
> Does that mean we can not compare/benchmark against other distributions
> because they use GPLv3 stuff.
> No, i want to know where standard released FreeBSD stands against
> standard released Linux distributions.
> If you compare benchmark userland applictions, then it is fair to use
> the latest compiler for the userland software also on FreeBSD.
> But what if the ports tree defaults to LLVM, then again we want to know
> what FreeBSD does against a Linux distribution.
> Why because that is what most of us will be using...!!

Who ever tried to use gcc 4.6 to compile the base system knows that it
is no eays task and it isn't so easy to simply change the compiler! This
is also true for a lot of ports.

If it is so easy to use a more modern compiler as some of the statements
made here would suggest, then I would expect a dedicated chapter in the
handbook! In such a case, every systems administrator trying to make a
long-term decission what operating system might be the base for the
future, does not need to be an enthusiastic of either BSD or Linux to
understand how to tweak - fanboys, developers or enthusiasts have
already choosen, apart from any rational or reason.

What matters are advantages which can be approved. Downlad the DVD,
install the OS, do some adjustments regarding to some pages of the
manual, choose a propper set of applicable software to benchmark,
compile, benchmark the system. Phoronix did so.

Well, it's hard for me to find the chapter in the handbook which
describes the performance tuning of SCHED_ULE and its sysctl tweaks,
someone may call me stupid and point me to the page, please ...


> 
> If we start to compile all the ports with gcc 4.7 to be on par in
> comparising and benchmarking, why spend all the time getting LLVM as the
> default compiler for ports also?
> Why not take that effort into making the WHOLE ports tree to compile
> with GCC4.7?
> 
> Reason, because FreeBSD goes the LLVM route. That is a decission FreeBSD
> is making!

Yes, and it is legitime to question that and bring pro and contra for
that decission. But since "FreeBSD" is obviously a small club of people
sitting like a duck on eggs (and, by the way, not their own genuine
invented eggs, more or less reingeneered eggs), those decissions get
more obscure than they seem to be anyway.


> And that choise will be the FreeBSD that is used in comparising and
> benchmarks on the net , not the utterly overcopiled and tuned FreeBSD
> against stock Ubuntu or whatever Linux distribution.
> 
> If it is a good or bad choice! That we will see in the
> comparising/benchmarks we will be seeing when that time comes.
> 
> Same goes for the scheduler! and all the other subsystems FreeBSD has
> choosen, that makes FreeBSD.
> 

... sometimes the underdog has to pick up what's left ...

>>
>>> I my opinion, you benchmark the latest release of Linux, FreeBSD,
>>> Solaris, Windows and whatever OS you want to compare!
>> As you probably know, Linux is just the kernel and the distributions add
>> user space programs, including a compiler. You can easily create a
>> "FreeBSD distribution" with more advanced compiler and use or even sell
>> it. But the FreeBSD project was cautious to not heavily depend on a
>> GPLv3 compiler (for reasons openly discussed at the time this decision
>> was made).
> I know Linux is a kernel, re read Linux as Linux Distribution!
> Yes you can use a more advanced compiler on FreeBSD, BUT you can do that
> on Linux also ,so where do you stop?
> Are you going to spend a month to compare a fullly tuned up FreeBSD
> system against a Linux distribution?
> No because the users will not spend months tuning and recompile there
> servers.
> They use the FreeBSD version that comes with the CD!
> And that we want to compare/benchmark against a Linux distribution.


When it comes to the question what to benchmark, what would you suggest?
Only the Linux kernel, so called LINUX and a couple of self selected,
self compiled, probably seld optimized GNU userland tools?

At the end I pick up a distribution as it comes from the "vendor", pick
up some informations about tweaks for several target workloads and start
benchmarking.

> 
>>> You want to benchmark the release and not a tuned version against a
>>> standard version.
>>> And that in general are the versions most of us users will use.
>> If you compare operating systems from a technical point of view, then
>> you'll be interested in relative performance of algorithms and methods
>> chosen. This is best achieved by using the same compiler for each of the
>> candidates.
>>
>> If you compare performance from a user point of view, you are correct
>> that performance delivered out of the box (without complicated tuning)
>> may be, what counts for most users. But those users that depend on best
>> performance e.g. for a FreeBSD based embedded product or a data center,
>> may tune the system, including compilation with a newer compiler than
>> the system default.
>>
>>> And what if in the future LLVM gets on par with Linux, is it stil fair
>>> to compare FreeBSD with Linux?
>> You can always compare anything with whatever you like (even apples with
>> oranges), but you need to be aware of what you compare and what your
>> goals are, to be able to draw reasonable conclusions.
>>
>> If you want to test out of the box performance, then test with system
>> compilers (or just those binaries delivered with the system).
>>
>> If you want to test for code efficiency or scalability, then use the
>> same compilers for each system under test to remove differences
>> introduced by the compilers (which are an external component not
>> developed by the FreeBSD people).
>>
>>> Or do we say, well we are on par, but it is not fair, yes we used the
>>> latest releases, but you can not blame Linux because they are still
>>> using GCC.
>> Depends on what you want or need to measure ...
> I am mainly talking about the out of the box comparison, because that is
> what most sites will do when doing a benchmark.
> they want to know if the the all new version of FreeBSD can compete with
> the latest release of Ubuntu.
> That are the benchmarks you will see on the net.

And this is the right way to do. The only way.

I start my project now. I got funding now. I have a time contraint to
keep to fullfill my scientific work, say, a PhD or a project. I need to
look into the future, I need to make decissions for a long haul
development. So, I pick up the OSs of my choice and benchmark them as
they are. All right, if there is a chapter in the handbook explaining
how to tune towards a specific expected workload, much better, so do so
... If this is not the way it is, please correct me!

> 
> 
>>> No what we will see then are haleluja blogs that FreeBSD is on par with
>>> Linux.
>> Such blog messages are not common in the FreeBSD community. FreeBSD used
>> to have big technical and performance advantages when Linux was young,
>> but even then, there was technical discussion between camps (and many
>> concepts were implemented in Linux based on BSD examples; I have taken
>> part in such discussions myself, some 15 to 20 years back).
> Well not  quite,  i remember the mysql on SCHED_ULE reports, and they
> where quit haleluja about the whole thing.
> Not wrong at all, we are all human and like to compete and even more
> like to be on top in what we do.
> 
> 
>>> For me peformance is not a show stopper, and for the most of us i think
>>> it is not.
>>> FreeBSD for me is a clean system that does the job perfect and has a
>>> very helpful community.
>> Well, this are valid aspects, too, and very hard to with benchmarks ;-)
>>
>> Regards, STefan
> I am not saying FreeBSD is bad and is not performing, i just do not
> understand why FreeBSD is doing all these things to look good.
> Again, most comparisings/benchmarks will be stock *BSD vs a stock Linux
> distribution.
> If we want to look good, FreeBSD must make sure there released version
> gets on par!
> We all know FreeBSD is quite conservative with its default settings.
> Maybe we need to set some default settings less conservative.
> But i do not know if that is good just for the numbers!
> 
> Regards,
> Johan

oh


Received on Fri Dec 23 2011 - 00:05:41 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:22 UTC