Re: Turning TESTS on by default

On Sat, Jun 7, 2014 at 3:53 PM, Konstantin Belousov <kostikbel_at_gmail.com> wrote:
> Could somebody point out a popular software system that spills the
> tests or other developer-only[*] stuff into the production install ? I
> immediately remember the perl and its modules which have very extensive
> test suite, but the test suite is not installed.
>
> [*] As is, developers of the system, not developers utilizing the product
> as the base.

That's the key you are missing: the FreeBSD test suite, just as the
NetBSD test suite, have been designed from the very beginning as a
tool for *both* developers and end users. Most test suites out there,
if not all, do not have such a goal and thus it is uninteresting, and
likely harmful, to install them (if only because it is non-trivial to
execute them in a controlled manner).

The FreeBSD test suite is intended to let users (especially sysadmins)
routinely assess the functionality of their system, especially right
after deployment and during system upgrades. This approach is partly
intended to cover the fact that the FreeBSD project cannot possibly
have[*] a large-enough testing farm to validate most hardware/software
combinations run by our users. It is also partly intended to inspire
confidence in the system by our users by being very transparent about
what works and what may possibly not work. NetBSD encourages you to do
test runs (even from cron) in its afterboot(8) manual page.

* Mind you: companies such as Microsoft have extremely large test
environments and, yet, they continuously fail to catch specific
failures in some weird system combination. It is just a fact of the
complexity of OSes. The difference between "them and us" is that they
cannot afford to ship their test suite because it's a very important
asset of theirs and because of the potential PR implications of users
uncovering failures here and there (my guesses though). I recommend
reading "The Old New Thing" blog for insightful details on all the
testing that Windows goes thorough, for example.