Re: Need help fixing tests in CURRENT

On Sep 9, 2014, at 10:17, Alan Somers <asomers_at_freebsd.org> wrote:

> On Tue, Sep 9, 2014 at 11:13 AM,  <yaneurabeya_at_gmail.com> wrote:
>> On Sep 9, 2014, at 10:11, Allan Jude <allanjude_at_freebsd.org> wrote:
>> 
>>> On 2014-09-09 13:05, Craig Rodrigues wrote:
>>>> Hi,
>>>> 
>>>> I did a buildworld/installworld of current in a bhyve VM, and then did the
>>>> following.
>>>> 
>>>> pkg install devel/kyua
>>>> cd /usr/tests
>>>> kyua test
>>>> kyua report
>>>> kyua report-html
>>>> kyua report-junit
>>>> 
>>>> I then published the results in Jenkins:
>>>> 
>>>> The results are here:
>>>> 
>>>> https://jenkins.freebsd.org/jenkins/job/FreeBSD_HEAD-tests2/lastCompletedBuild/testReport/
>>>> 
>>>> There seem to be 31 test failures.  Can folks look into those
>>>> and see if the tests need to be fixed
>>>> 
>>>> Right now I ran these steps manually, but hopefully over time
>>>> we can automate more of this stuff.
>>> 
>>> Most of the errors (jot, m4, sed, etc) seems to be:
>>> 
>>> kyua-tap-tester: Configuration variables not supported; ignoring
>>> 'has.cleanup=false'
>>> kyua-tap-tester: Configuration variables not supported; ignoring
>>> 'unprivileged-user=tests’
>> 
>> Not quite…
>> 
>>> while everything else about the test was successful.
>> 
>> Hi Craig,
>>        Taking a stab at the issues, most of the problems are because of https://github.com/jmmv/kyua/issues/103 . The other issues with atf ( https://github.com/jmmv/atf/issues/8 ) and kyua ( https://github.com/jmmv/kyua/issues/109 ) are known (I’m sure about ATF, not 100% sure about kyua, but my gut instinct says yes).
>> Thanks!
>> -Garrett
> 
> There are still a few genuine failures.  usr.bin.yacc.yacc_tests.main
> is failing for reasons that have nothing to do with ATF or Kyua.  So
> are bin.pkill.pkill-j_test.main and bin.pkill.pgrep-j_test.main.

	Ugh… the pgrep/pkill -j issue is back ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191019 )… it was fixed for a few weeks, so I suspect it’s related to the changes that kib_at_ and others have been making to how the proc bits in the kernel are handled.
	The yacc issue is new ( https://jenkins.freebsd.org/jenkins/job/FreeBSD_HEAD-tests2/lastCompletedBuild/testReport/usr.bin.yacc/yacc_tests/main/ ). It passes on my CURRENT/i386 VM, but it’s a bit out of date (2 weeks old). I’ve filed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193499 to track the issue.
Thanks!
-Garrett

$ (cd /usr/tests/usr.bin/yacc/; sudo kyua debug yacc_tests:main)
...
...ok ./yacc/ok_syntax1.tab.c
...ok ./yacc/ok_syntax1.tab.h
** testing ./pure_calc.y
...ok ./yacc/pure_calc.error
...ok ./yacc/pure_calc.output
...ok ./yacc/pure_calc.tab.c
...ok ./yacc/pure_calc.tab.h
** testing ./pure_error.y
...ok ./yacc/pure_error.error
...ok ./yacc/pure_error.output
...ok ./yacc/pure_error.tab.c
...ok ./yacc/pure_error.tab.h
** testing ./quote_calc.y
...ok ./yacc/quote_calc.error
...ok ./yacc/quote_calc.output
...ok ./yacc/quote_calc.tab.c
...ok ./yacc/quote_calc.tab.h
...ok ./yacc/quote_calc-s.error
...ok ./yacc/quote_calc-s.output
...ok ./yacc/quote_calc-s.tab.c
...ok ./yacc/quote_calc-s.tab.h
** testing ./quote_calc2.y
...ok ./yacc/quote_calc2.error
...ok ./yacc/quote_calc2.output
...ok ./yacc/quote_calc2.tab.c
...ok ./yacc/quote_calc2.tab.h
...ok ./yacc/quote_calc2-s.error
...ok ./yacc/quote_calc2-s.output
...ok ./yacc/quote_calc2-s.tab.c
...ok ./yacc/quote_calc2-s.tab.h
** testing ./quote_calc3.y
...ok ./yacc/quote_calc3.error
...ok ./yacc/quote_calc3.output
...ok ./yacc/quote_calc3.tab.c
...ok ./yacc/quote_calc3.tab.h
...ok ./yacc/quote_calc3-s.error
...ok ./yacc/quote_calc3-s.output
...ok ./yacc/quote_calc3-s.tab.c
...ok ./yacc/quote_calc3-s.tab.h
** testing ./quote_calc4.y
...ok ./yacc/quote_calc4.error
...ok ./yacc/quote_calc4.output
...ok ./yacc/quote_calc4.tab.c
...ok ./yacc/quote_calc4.tab.h
...ok ./yacc/quote_calc4-s.error
...ok ./yacc/quote_calc4-s.output
...ok ./yacc/quote_calc4-s.tab.c
...ok ./yacc/quote_calc4-s.tab.h
** testing ./varsyntax_calc1.y
...ok ./yacc/varsyntax_calc1.error
...ok ./yacc/varsyntax_calc1.output
...ok ./yacc/varsyntax_calc1.tab.c
...ok ./yacc/varsyntax_calc1.tab.h
yacc_tests:main  ->  passed
$ uname -a
FreeBSD isilon-fuji-current.local 11.0-CURRENT FreeBSD 11.0-CURRENT #7 r270227+e6e86c2(isilon-atf)-dirty: Sat Aug 23 02:26:59 PDT 2014     ngie_at_isilon-fuji-current.local:/usr/obj/usr/src/sys/FUJI  i386