[ansi-test-devel] [imp-hackers] Procedure for adding tests to the ansi test suite

[Tobias C. Rittweiler]

Sam Steingold writes:

> On Mon, Jul 27, 2009 at 4:14 PM, Tobias C. Rittweiler wrote:
>
> >  * the new tests should be visibly separate from what's currently
> >    there. Perhaps this should be reflected on the file system layout of
> >    the source tree.
>
> what for?
> do we need to separate the scripture of St. Paul from our later
> inferior commentaries?
>
> >  * it should be possible to exclude the new tests when running the test
> >    suite.
>
> one can already exclude (or, rather, ignore the results of) any number
> of tests using rt::*expected-failures*.

People use the ansi test suite as a regression test for changes on their
implementations. By having run the suite in the past, they're used to
that currently N tests are failing.

If new tests are added, it may be that suddenly N+M tests are failing,
indicating regressions their changes introduced---which they actually
did not do.


> > Should a disapproval by as few as one person who can not be convinced
> > differently in reasonable time be enough to deny a proposal? Perhaps
> > this should be tightened up to a person representing an implementation.
>
> Here we are stumbling into the question of what is our mandate.
>  [1] Are we creating a comprehensive test of the existing ANSI CL standard?
>  [2] Are we creating a de-facto extension to the ANSI CL standard by
> finding what things are done uniformly by implementations but not
> specified by ANSI?
>
> if [1], then I don't think such veto power is a good idea.
> I am not even sure that the fact that someone "represents an
> implementation" should give that person extra weight.
>
> I think that if someone (not necessarily representing an
> implementation) thinks that the test is wrong, that person should
> register the different opinion in the comment to the test in the
> sources. If the opinion is valid, I think multiple results should be
> accepted.
> This, of course, should happen after a discussion on ansi-test-devel,
> when it becomes clear that both opinions are not going away.


Yes multiple results should be accepted where it makes sense. But if you
test for behaviour, it's often the case that either an implementation
exhibits the behaviour, or does not. Creating a test which can be both T
and NIL does not make much sense.

Tests where people's opinions differ about could be tagged as being
controversial which could be excluded from a run of the suite.

Thanking for your thoughts,

  -T.