Re: Turning TESTS on by default

On Sat, Jun 07, 2014 at 12:31:01PM -0600, Alan Somers wrote:
> On Fri, Jun 6, 2014 at 9:33 PM, Konstantin Belousov <kostikbel_at_gmail.com> wrote:
> > On Fri, Jun 06, 2014 at 03:14:52PM -0400, Julio Merino wrote:
> >> Hello all,
> >>
> >>
> >> TL;DR
> >> -----
> >>
> >> I plan to turn the TESTS src.conf knob ON by default on Tuesday once I
> >> have been able to perform enough sanity-checks of the build and all of
> >> them pass.
> >>
> >> The impact of this is that the FreeBSD Test Suite (see tests(7)) will
> >> be built and installed by default under /usr/tests/ along with the
> >> private atf libraries and binaries. There should be no other changes
> >> and this should be transparent to everyone.
> >>
> >> If this happens to break the world in any way, we can trivially roll
> >> the change back to fix the fallout.
> >>
> >>
> >> Some details
> >> ------------
> >>
> >> TESTS was never intended to be disabled by default. However, the
> >> original patches that were committed months ago related to this
> >> feature broke the build and the easiest way out (instead of reverting
> >> the commits) was to set the knob to disabled. Unfortunately, it stayed
> >> that way even after the discovered problems were fixed.
> >>
> >> I am confident enough now that we have ironed out all major issues
> >> that this might introduce, so it is about time to enable TESTS by
> >> default again in HEAD.
> >>
> >> The benefits of this are that 1) we allow end users (especially
> >> consumers of binary releases!) to run the tests out of the box, as it
> >> has always been intended; and 2) we will be able to run the official
> >> release builds through testing via Jenkins, instead of having to issue
> >> custom builds.
> > This is very weird and unprobable.  Users cannot care less about running
> > the test suite, they use OS to run applications.  IMO enabling installation
> > of the stuff that bloats the system but have no practical use for the
> > system consumer should not be allowed by default.
> 
> I disagree.  Sure, some users won't care.  Probably even most users
> won't care.  But some of our users are active supporters of FreeBSD.
> They evangelize, they file PRs, and they help other users on the
> forums.  Those users will run the tests.  Some of them will find bugs
> that we didn't, because they'll be using different hardware and
> different configurations.  Plus, shipping a test suite exudes an aura
> of quality (if the tests pass, that is).  So I think that we should
> install the tests, but in a separate installation set, just like
> games.

I would agree with your arguments, and in fact not bother with the
proposal at all, if most systems were installed using installer.  I am
very much confident that significant part of the population is installed
or updated using make build/installworld.

If somebody cares to run tests, she certainly cares enough to be able to
turn the knob on.  Otherwise, the tests take sometimes precious space on /
or /usr, for nothing.

Could somebody point out a popular software system that spills the
tests or other developer-only[*] stuff into the production install ? I
immediately remember the perl and its modules which have very extensive
test suite, but the test suite is not installed.

[*] As is, developers of the system, not developers utilizing the product
as the base.