On Sunday 07 May 2006 22:16, you wrote: > On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > The results in my mail were mean values over 2 runs, > > only once did I see really huge (more than 10%) differences between > > several subsequent runs with same settings, this case was clearly > > mentioned in the results. > > FYI, 2 is not really enough, you should do at least 10 repetitions of > each test to reduce variance (which can be a lot, despite what you > saw!) and so that differences between them can be accurately > estimated. Ministat is really helpful for this. > I'm well aware that 2 is not enough for quality measurements and I certainly would have liked to do more repetitions, but I was running against a clock - this machine might be shipped out to client any time and I wanted to test several combinations of OS [fbsd 6, fbsd current, current + watsons patch, linux] with different threading library and scheduler combinations at different thread counts and nice values. This creates nice combinatorial explosion. Even a single run on one OS ver + one of the schedulers with 2 repetitions of each test takes about 2 hours, so I had to make some compromises. But i believe the trends are clear enough from these results and while I certainly can't say anything about <~5% performance changes what we see are consistent trends and 20%+ performance differences.Received on Sun May 07 2006 - 18:27:42 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:55 UTC