Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

On 22.12.11 11:02, Johan Hendriks wrote:
> Stefan Esser schreef:
>> Am 21.12.2011 22:49, schrieb Johan Hendriks:
>>> If you tune up FreeBSD to use the GCC 4.7 compiler, or downgrade linux
>>> to 4.2.1, then that will tell me nothing about FreeBSD vs Linux.
>> The gcc version distributed with FreeBSD was chosen for license reasons,
>> not for technical reasons. If you are OK with installing a GPLv3
>> licensed compiler on your systems, then just do it and take advantage of
>> the improved code generated by it.
> It does not matter what the decission is to use the old compiler, it 
> is a fact that the base comes with 4.2.x
> Does that mean we can not compare/benchmark against other 
> distributions because they use GPLv3 stuff.

If you intend to compare operating systems 'as shipped', then forget 
about comparing third party programs on top of these operating systems. 
Compare only any software that is already available with the operating 
system, as shipped.

There is plenty of software in the base system and this software is 
specifically different (say) in FreeBSD and Linux.

Anyone made such comparison?


> No, i want to know where standard released FreeBSD stands against 
> standard released Linux distributions.
> If you compare benchmark userland applictions, then it is fair to use 
> the latest compiler for the userland software also on FreeBSD.
> But what if the ports tree defaults to LLVM, then again we want to 
> know what FreeBSD does against a Linux distribution.
> Why because that is what most of us will be using...!!

It is pretty easy to tell the ports system to use latest gcc, as 
described here 
http://www.freebsd.org/doc/en/articles/custom-gcc/article.html

>
> If we start to compile all the ports with gcc 4.7 to be on par in 
> comparising and benchmarking, why spend all the time getting LLVM as 
> the default compiler for ports also?

Because, FreeBSD attempts to be as GPL free as possible. GPL is 
incompatible with many of the applications of FreeBSD.

> Why not take that effort into making the WHOLE ports tree to compile 
> with GCC4.7?

This has already been done. See above.

>
> Reason, because FreeBSD goes the LLVM route. That is a decission 
> FreeBSD is making!

LLVM, as well as GCC are just choices in FreeBSD.

What FreeBSD is making is, to make it safe to use LLVM to compile 
everything on FreeBSD, including the kernel.

As you may have already noticed, some ports require to be compiled with 
a specific version of gcc, or a very specific compiler -- there is 
nothing wrong with this -- this is external software after all.

> And that choise will be the FreeBSD that is used in comparising and 
> benchmarks on the net , not the utterly overcopiled and tuned FreeBSD 
> against stock Ubuntu or whatever Linux distribution.

Earlier on this thread I mentioned, that FreeBSD and Linux philosophies 
differ. While Linux (well, some distributions, to be correct) will try 
to optimize certain parts of the OS/applications in order to do well in 
benchmarks -- FreeBSD takes a different approach. The FreeBSD (and BSD 
UNIX, in general) approach is "do the right thing". This may produce the 
results slower, but the environment is more stable in general and in the 
long run, the results noticeable better.

This argument, by the way reminds me of the AT&T vs BSDI lawsuit, where 
at the time UCB was involved there was lengthly discussion (at court), 
about "sloppy programming, but we had to have something for the 
deadline" (AT&T) vs "well, we have designed the architecture we think is 
appropriate, there might be few unimplemented things, but we are working 
on it".

>
> Same goes for the scheduler! and all the other subsystems FreeBSD has 
> choosen, that makes FreeBSD.

What about the scheduler?

>
>
> Yes you can use a more advanced compiler on FreeBSD, BUT you can do 
> that on Linux also ,so where do you stop?

Can you compile the entire Linux system, kernel and userland and 
external packages with LLVM?

> Are you going to spend a month to compare a fullly tuned up FreeBSD 
> system against a Linux distribution?

I would do that, if:

- I have a task for which I need tuned system (that is, hardware would 
be at limits);
- The application is available on both;
- There is evidence or suggestion that the application under Linux will 
perform much better;

> No because the users will not spend months tuning and recompile there 
> servers.
> They use the FreeBSD version that comes with the CD!

On servers? :-)

> And that we want to compare/benchmark against a Linux distribution.

No, we don't. We run our servers, we don't want to compare/benchmark 
them with Linux for no reason.

We want however to identify design or implementation weaknesses in 
FreeBSD and fix these. This rarely happens by comparing to Linux 
distributions. There are better things to be observed in say, Solaris.

If we were selling FreeBSD, we would be interested in publishing 
benchmarks that demonstrate how superior to Linux it is. We would tune 
FreeBSD to beat Linux in most benchmarks available and we will ignore 
the fact that in real-worls scenarios it will be worse. For.. we would 
have already made the sale.

Now, you will say "but Linux is free". :)
Thing is, the goal most Linux distributions have is to dominate with 
'market share'. The primary purpose is to convince the software 
developers to port their software for Linux. This has had some positive 
results but in general -- not enough.
There are better 'open source' approaches and the FreeBSD ports 
collection proves that.

>>
>>> Or do we say, well we are on par, but it is not fair, yes we used the
>>> latest releases, but you can not blame Linux because they are still
>>> using GCC.
>> Depends on what you want or need to measure ...
>
> I am mainly talking about the out of the box comparison, because that 
> is what most sites will do when doing a benchmark.
> they want to know if the the all new version of FreeBSD can compete 
> with the latest release of Ubuntu.
> That are the benchmarks you will see on the net.

Compete on what?

> If we want to look good, FreeBSD must make sure there released version 
> gets on par!

Everything has its cost.

> We all know FreeBSD is quite conservative with its default settings.

It is always better to have reliable OS, that can boot on almost any 
hardware, not crash for no reason and once it is up and running, you can 
always tune it to your taste.

Maybe you are looking for an "tune FreeBSD for your usage" type of 
application?

> Maybe we need to set some default settings less conservative.

If such settings could make FreeBSD less reliable or less stable -- 
definitely NOT.

> But i do not know if that is good just for the numbers!
>

Another idea, you may make your own FreeBSD derivative, such as PC-BSD 
or FreeNAS (both, special-purpose setups, optimized for what they do) 
and compare it with someone else's Linux distribution.
By the way, there is FreeBSD based "Linux" distribution: Debian kFreeBSD 
-- however weird this sounds.

Has anyone compared recent PC-BSD with Ubuntu?

Daniel