Re: The case for FreeBSD

On Sun, 6 Feb 2005, Scott Long wrote:

> There has been a lot of recent talk and advocacy for NetBSD 2.0 from the
> NetBSD team.  Most recently there were a series of articles posted my
> Chritos Zoulas describing why NetBSD is relevant and why it's a better
> choice than either FreeBSD or OpenBSD.  While I strongly applaud the
> accomplishments of the NetBSD team and happily agree that NetBSD 2.0 is
> a strong step forward for them, I take a bit of exception to many of
> their claims and much of their criticisms of FreeBSD. 

I think the technical arguments for FreeBSD in your e-mail are excellent
-- we a number of really incredible features that make FreeBSD both an
excellent server platform, and an excellent platform to build into other
products.  These features set us clearly ahead in some very important
areas.  However, I'm sure Christos's report was not centered on bashing
us.  I noticed a lot of "We got X from FreeBSD" and "We should get X from
FreeBSD".  As you might expect from an annual report, it was very
introspective -- and appropriately so -- and it spoke to strategies for
improving NetBSD.  And the observation that code moves in both directions
is important: NetBSD has picked up some neat stuff from us in the past few
years, including KQueue, UFS2, OpenPAM, etc.  Because we work hard and
develop good features, they get to reap the benefit (which is part of the
BSD model).  We've also picked up some neat stuff, including support in
developing new hardwre platforms, some of the micro-benchmark
optimizations, etc.  Hopefully, we've both been good about crediting where
these things come from.  And you can see that strategy in the plans for
the future in both camps as well -- NetBSD's plans to pick up SACK from us
are just one example.  If I were them, I'd also start grabbing some of our
SMP infrastructure -- if not the overall locking strategy, certainly the
abstractions.

I agree that the "NetBSD is more scalable than FreeBSD"  conclusion
stemming from a previously posted set of benchmarks was bogus: scalability
should be measured as a property of real-world applications, not the
constant in the file descriptor allocation routine.  I.e., if you really
want to measure scalability, run a web benchmark or a database benchmark.
If NetBSD still comes out faster in a properly run macro-benchmark, then
that's an important thing to know, and an important target --
microbenchmarks are nice to optimize for, but the big picture is more
important, and not mentioned in the benchmark paper.  Everything was fine
in the paper, with the exception of the abstract/conclusion, which had one
of those "then a miracle occurs"  moments in reasoning. :-)  So I wouldn't
dismiss the technical results, but the "headline" was pretty out of line
with regard to the technical results -- this follows in an increasingly
long history of weird open source "scalability" benchmarks, in which a set
of microbenchmarks is used to justify an overall argument that isn't
supported by the technical content. By reaching less far, the argument
would become more accurate.  As it turns out, it's a lot easier to run
micro-benchmarks properly than macro-benchmarks properly, which is
probably why micro-benchmarks predominate.

I think it is the case that lately the NetBSD folk have been doing a more
effective PR job than they have been previously, and have managed to grab
some headlines as a result.  Our response, in as much is one is needed,
should be three-fold: where NetBSD can demonstrate a quantitative
performance of feature benefit over FreeBSD, we should figure out why, and
fix it where it makes sense and is important.  We should do some
comprehensive benchmarking of our own, and use them to improve our
system as well as document its benefits.  The NetBSD benchmark was a
classic example of micro-benchmarks feeding performance optimization, and
vice-versa -- not without benefit, but true none-the-less.  However, our
primary objections to it haven't been that we may not sometimes not win
every micro-benchmark, but that it missed what's really important in
scalability -- scalable application performance on the platform.

There are some important ways that out SMPng/KSE architecture should (and
do)  perform really well: for example, our threading library runs threads
on more than one CPU at a time by default (which the NetBSD SA
implementation currently doesn't), and we can excute parallel system calls
in kernel, which should be quite measurable (and I've seen results
suggesting that it really is).  However, because we took such a cut in
performance early in 5.x development, we've not been doing continuous or
effective long-term benchmarking.  I have a nice history of MySQL results
improving over the last year -- basically doubling or more in performance
as we improved, especially on SMP -- but that's hardly comprehensive.
We've built excellent project infrastructure for build management,
packaging, etc, but we need to work more effectively as a project, not
just as individuals, to manage performance the same way.

We should do a better PR job of identifying areas of particular strength
-- your list of features would make the great beginnings of a white paper
on FreeBSD network stack technology, for example.  Many of the features
there have set the standard for implementation in other systems -- kqueue
is now widely used across many platforms, and those that don't have kqueue
try to immitate with a bit of NIH on the side :-).

Finally, we should continue our heavy investment of development (and
other) resources into the FreeBSD project -- we're a very large project
with a very large number of extremely competent developers.  The BSD
platforms have defined operating system technology for many years, and in
many ways continue to do so.  We should make sure we continue to do that. 
Projects like SMPng, KSE, etc, have come an incredibly long way in the
past few years, and reflect both the huge challenges they involve, and the
level of investment we've made.  Commercial UNIX vendors such as Sun
easily spent far more in the way of resources to bring these features to
Solaris, and probably took even longer than we did to bring them to
fruition.  We have the advantages of a very open development process, and
we take advantage of them.  While there's more work to do, we can actually
see the finish line on these projects.  With recent work by Jeff on
VFS/UFS, we're reaching the point where a significant percentage of our
kernel is fully threaded, parallelized, and able to run preemptively.  Now
that the benefits are starting to pay off is the time we can reap the best
benefits from careful work -- i.e., we should now be able to perform
optimization and illustrate some of the potential gains from the more
mature kernel architecture.

So I guess my message would be this: we should focus our energy in
demonstrating (and making sure) FreeBSD is the best platform out there.
We've done an incredible job, but we need to keep doing it.  This includes
doing a better job with PR.  We need to do exactly what you've laid the
groundwork for: making an argument for FreeBSD as a platform.

Robert N M Watson