Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

On 12/15/2011 07:25 AM, Stefan Esser wrote:
> Am 15.12.2011 11:10, schrieb Michael Larabel:
>> No, the same hardware was used for each OS.
>>
>> In terms of the software, the stock software stack for each OS was used.
> Just curious: Why did you choose ZFS on FreeBSD, while UFS2 (with
> journaling enabled) should be an obvious choice since it is more similar
> in concept to ext4 and since that is what most FreeBSD users will use
> with FreeBSD?

I was running some ZFS vs. UFS tests as well and this happened to have 
ZFS on when I was running some other tests.

>
> Did you tune the ZFS ARC (e.g. vfs.zfs.arc_max="6G") for the tests?

The OS was left in its stock configuration.

>
> And BTW: Did your measured run times account for the effect, that Linux
> keeps much more dirty data in the buffer cache (FreeBSD has a low limit
> on dirty buffers since under realistic load the already cached data is
> much more likely to be reused and thus more valuable than freshly
> written data; aggressively caching dirty data would significantly reduce
> throughput and responsiveness under high load). Given the hardware specs
> of the test system, I guess that Linux accepts at least 100 times the
> dirty data in the buffer cache, compared to FreeBSD (where this number
> is at most in the tens of megabyte range).
>
> If you did not, then your results do not represent a server load (which
> I'd expect relevant, if you are testing against Oracle Linux 6.1
> server), where continuous performance is required. Tests that run on an
> idle system starting in a clean state and ignoring background flushing
> of the buffer cache after the timed program has stopped are perhaps
> useful for a very lowly loaded PC, but not for a system with high load
> average as the default.
>
> I bet that if you compared the systems under higher load (which
> admittedly makes it much harder to get sensible numbers for the program
> under test) or with reduced buffer cache size (or raise the dirty buffer
> limit in FreeBSD accordingly, which ought to be possible with sysctl
> and/or boot time tuneables, e.g. "vfs.hidirtybuffers").
>
> And a last remark: Single benchmark runs do not provide reliable data.
> FreeBSD comes with "ministat" to check the significance of benchmark
> results. Each test should be repeated at least 5 times for meaningful
> averages with acceptable confidence level.

The Phoronix Test Suite runs most tests a minimum of three times and if 
the standard deviation exceeds 3.5% the run count is dynamically 
increased, among other safeguards.

-- Michael

>
> Regards, STefan
>