Re: FreeBSD Testing Facility

On Sun, Feb 24, 2013 at 9:25 AM, Giorgos Keramidas <keramida_at_freebsd.org>wrote:

> On 2013-02-21 07:04, Matthew Jacob <mjacob_at_freebsd.org> wrote:
> > On 2/21/2013 5:04 AM, Mehmet Erol Sanliturk wrote:
> > >Dear All ,
> > >
> > >During development of FreeBSD , testing is very vital .
> > >...
> >
> > This in general is a good suggestion. Most companies do such
> > automated testing as a matter of course.
> >
> > Note however that this is a volunteer effort. Were you volunteering
> > to set up such an automated, possibly testzilla driven, facility? It
> > would certainly help the quality, although as others have noted
> > snapshots are often likely to be broken.
>
> To the OP:
>
> Like Matthew has said, this is a volunteer effort.  So if you have
> experience with setting up testing automation, and you are willing to
> help us set up something like this, please go ahead :-)
>
> I've worked in places where the following types of tests are used:
>
>   - Presubmit tests that check specific parts of functionality.
>
>   - Commit-related tests that run asynchronously in the background,
>     and report back later (e.g. through email).
>
>   - Test systems that cache previous results and report a simple 'green'
>     vs. 'red' status for _every_ single commit.
>
>   - System tests that check for particular functionality, health
>     criteria, etc. - some times fully automated, some other times
>     requiring a token amount of manual support.
>
> So here are two important questions, regarding the tests you mentioned:
>
> When you speak about 'testing FreeBSD', which type of tests are you
> interested in seeing?
>
> Are you willing to help us set up something that runs the type of tests
> that you want to see?
>
>


To another message I replied as follows :

http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039984.html


I want say that again to manage such a system is not possible for me very
much , additionally for lack of facilities .

My wish is that let's take this subject to be worked in detail and by a
group effort , design , implement and apply such a testing facility .

In this work , if anything I can supply , I will never keep myself back .

Without reflective responses , if we consider my first message :

http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039976.html

a very rough outline is supplied .

Reason is such a suggestion is to see many failures of *BSD systems that
these can be eliminated if such a system is designed , implemented , and
applied .


In many messages by FreeBSD developers , it is said that it is not possible
to establish a testing farm by the FreeBSD project and maintain it , which
is quite correct .


Assume the following :

If any one is downloading and trying to install and run Current development
iso files , itis very likely that the main intention is to test and help to
its development .

By counting number of such testing attempts , it is possible to estimate
the number of people which can participate to a more coordinated testing
system .

My opinion is that such a system ( a distributed testers population ) is
already present , and over time , even it may be enlarged .


Actually , problem is large enough to establish a working group to attack
it .

There is a necessity to have committers to generate the necessary  ftp
sites and manage them .

At present , there is a port system . The testing facility will be similar
to that system , but different programs will be employed .


The first action will be to design what will be tested and how .

Since FreeBSD is composed from components ( boot , kernel , each system
programs , etc. , )
, every component will be a testing subject .

In the svn , there are already  testing parts .

By combining these , and supplying , developing missing parts will generate
a testing framework .


At the beginning , it will not be possible to generate a state of the art
testing facility .
By starting from the existing parts , over time , it will be possible to ad
tests one by one .
This will be similar to development of ports system : Over time it has been
populated one by one .


As a an applicable first step , a FreeBSD developer , may establish a wiki
page or use an existing one to develop a "Testing Requirements and Design ,
Implementation and Applications" page .

A mailing list may be established to discuss testing problems .

An ftp site may be established to apply tests as suggested in my first
message .

As an example :

Assume that a part related to video display cards is under development ,
such as KMS .
The developer(s) will have a limited number of cards available in their
computers .
A potential people population exists which they use such cards , for
example RADEON , or INTEL cards .

In the mailing lists , it may be announced that testers are needed for
specific video card branch , such as RADEON . People , fills a form to
participate in the tests .

The developer prepares a file just like a port/package .

Sends a message to the subscribers wishing to test this problem .

Possible testers , upon receiving that message , may enter a command , such
as

# test_apply  name_of_testing_package

just similar to pkg_add command .


The test_apply program , downloads the testing package and applies it .
At the end , the best action is to generate a structured e-mail , the
results may be sent to the testing data base just a like bug tracker system
.

I emphasize structured messages for their automatic processing possibility .
If this is not very feasible , at least a formal message may be requested
to evaluate results
easily . Otherwise results like "Tests failed" will not be very helpful for
the developers .

A script in there processes the returned e-mails and generates a report to
developer(s) ,.


A cycle of such tests continues up to a satisfactory result .

Some parts , such as the following , are dependent very much to hardware to
be used :

-  Booting process
- Network communications
- Video display units
- USB attached components
- Parts requiring BIOS cooperation ,
- Parts detecting , managing main board parts ( circuits , ports , etc. )
- Parts detecting , managing attached peripheral devices .

All of these may be subject of such distributed testing actions .

After testing many of the components and ensuing that they are working ,
there comes to
testing of their combinations , for example  :

 - File systems in local disks,
 - NFS like systems

Then , a whole test : Testing complete install iso files . Here , there
will not be a booting or hardware detection  failure , but failures related
to cooperative working of systems under user applications .
These failures may occur between mismatched components during their
interaction .

Therefore , such a testing facility requires cooperation of many people to
function properly .

A group of leaders may organize the efforts .


Thank you very much .