Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang

On Sun, Mar 13, 2011 at 3:19 PM, Jakub Lach <jakub_lach_at_mailplus.pl> wrote:

>
>
> Vinícius Zavam wrote:
> >
> >
> > i'm still curious about things like CPUTYPE= and -march= configured as
> > native, gentlemen.
> > is it the "golden egg" to use with our system or not? why "natives"
> > aren't in the benchs?
> >
> > /me feels confused.
> >
> >
> > --
> > Vinícius Zavam
> > profiles.google.com/egypcio
> >
>
> Apparently -march=native would equal -march=core2
> with 65nm generation Core2s, this is not the case with
> Penryns.. But there are none in the test?
>
> However, I agree that testing with -march=native
> would be simpler and more straightforward.
>
> regards,
> - Jakub Lach
> --
>




The compilers Clang and GCC may also be compared with the following design
because ( on the same computer , multiple parameters are measured ) :

      Clang Version x
 ------------------------------------
 Repeated Measures --->
                              p(1)   p(2)   p(3)   p(4) and other parameters
up to p(m)
                               -----  -----  -----  -----
 Computer 1              value  value  value  value   .....
 Computer 2              value  value  value  value   .....
.
.
.
 Computer n              value  value  value  value   .....



      GCC Version x
 ------------------------------------
 Repeated Measures --->
                                p(1)   p(2)   p(3)   p(4) and other
parameters up to p(m)
                                 -----  -----  -----  -----
 Computer n+1            value  value  value  value   .....
 Computer n+2            value  value  value  value   .....
.
.
.
 Computer n+n            value  value  value  value   .....




For each compiler the same number of computers are used ( This is called
balanced design ) .
Evaluation of unbalanced designs may not be available in used statistical
packages , and
theoretically , they are NOT very good .


Here factors are :
                   (1) Compilers ( CLang , GCC )
                   (2) Measured parameters as Compilation Parameters
                       such as
                       ( No optimization )
                       ( Optimization Level 1 )
                       ( Optimization Level 2 )
                       ( Processor Kind 1 )
                       ( Processor Kind 2 ) and
                       ( Code Generation Kind 1 )
                       ( Code Generation Kind 2 ) and/or
                         any number of other parameters

Number of computers as n should be greater than Number of parameters as m .

Subjects are the computers which no one of them is equal to the other .
Measured parameters are also called treatments .

In statistical analysis packages and
   books on experimental designs
this design is called

two-factor experiment with repeated measures on one factor .

Also the other names may be used :

Repeated measures design , or
within-subjects factorial design , or
multifactor experiments having repeated measures on the same elements



Inclusion of two GCC versions into the above table as another compiler may
NOT be very well ,
because GCC compilers are likely VERY CORRELATED with each other ( because
they are using
the  same code generator perhaps some patches applied to distinguish the
versions ).


To obtain an idea about correlation strength of the GCC compilers ,
CANONICAL correlation analysis may be used when there are multiple
parameters
( do NOT use two-by-two correlation coefficients when there are more than
two parameters ) , or ,
simple correlation when there are only two parameters ( one for each
compiler ) .


Design is as follows :

                GCC Version x           GCC version y
                ---------------------   ---------------------
                 p(1)  P(2) ... p(k)    p(1) (p2) ... p(k)

Computer 1   v    v        v       v     v       v
computer 2   v    v        v       v     v       v
.
.
.
Computer n  v    v        v       v     v       v


where
     p(1) , p(2) , ... , p(k) are the measured parameters ,
      k : Number of parameters for each block individually
          ( There may be different parameter sets , but
            for our purposes equivalent parameter sets are required )
      n : Number of observations , where each computer should be different
from
          the others .

      v : Value measured for a parameter

When there is a significant CANONICAL correlation between two compiler
related
values blocks :
 (i) it is NOT possible to include the two compilers in the above
     repeated measures design because of high collinearity .
 (2) Selection of BEST compiler is possible because two compilers are very
similar
     ( there are no difference between them other than performance level ) .


When the CANONICAL correlation is NOT significant ,
the other GCC compiler may be included
as a third compiler into the repeated measures design .




http://en.wikipedia.org/wiki/Repeated_measures_design
http://en.wikipedia.org/wiki/Category:Experimental_design

http://en.wikipedia.org/wiki/Canonical_correlation
http://en.wikipedia.org/wiki/Category:Multivariate_statistics


Thank you very much .


Mehmet Erol Sanliturk