On 21 December 2011 22:03, Freddie Cash <fjwcash_at_gmail.com> wrote: > On Wed, Dec 21, 2011 at 1:49 PM, Johan Hendriks <joh.hendriks_at_gmail.com> wrote: >> Nice page, but one thing i do not get is the following. >> >> [quote] >> If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC 4.7 >> then the results are unlikely to tell you anything meaningful about FreeBSD >> vs Ubuntu. >> [/quote] >> >> That is a little strange in my opinion. >> It tells me that FreeBSD falls more and more behind on Linux. >> The reason is or could be that FreeBSD cannot or will not include GCC 4.7 >> and that FreeBSD will not be on par with Linux anymore. > > When benchmarking two systems, you need to make sure that everything > possible is the same (constants) and that the only differences between > the two systems are what you want to benchmark (variables). Yes and no, but to be perfectly frank, the statement, as it stands, is a bit of a nonsense. Let me illustrate in a different way. This is macro~ vs micro~comparison of systems and depends on what you are trying to get out of the benchmark. Using the same argument one can say that Ferrari F430 vs Toyota Prius is a meaningless comparison because the under-the-hood equipment is different. Now, it is absolutely correct to say that in A vs B comparisons, only one thing should be changed and the rest should remain constant. The important thing is, however, to determine the scope of your benchmark: you are not benchmarking a component of A vs a component of B, but you are benchmarking A as a whole system and B as a whole system. Thus, the thing that changes is the system itself. Going back to F430 vs Prius, you first decide what you want to benchmark (acceleration, top speed, fuel consumption, ride comfort, &c) then you measure that aspect in each of the system---you are not looking at the wiring, engine, wheels, &c individually but *at a whole system*. You use the same route, time of day, driver, drive pattern, weather conditions, &c, the only thing that changes is the car! Similarly, FreeBSD vs Linux, you want to a) determine what metric you want to benchmark (NFS throughput, HTTP client handling, SMPT throughput, prime number computation) and b) *scientifically* measure the system against that metric... This would essentially amount to having identical set up and tests, and only changing the hard disks (one containing Linux and another one containing FreeBSD). I don't see why this is such a difficult concept to grasp. Cheers, -- Igor M. :-)Received on Wed Dec 21 2011 - 21:34:07 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:22 UTC