[ansi-test-devel] Procedure for adding tests to the ansi test suite

[Sam Steingold]

On Mon, Jul 27, 2009 at 4:14 PM, Tobias C. Rittweiler<tcr at freebits.de> wrote:
>
> Can anyone think of a good procedure for getting new tests into the test
> suite?

I am not sure we need more procedure than Paul had when he wrote the tests.

>  * the new tests should be visibly separate from what's currently
>    there. Perhaps this should be reflected on the file system layout of
>    the source tree.

what for?
do we need to separate the scripture of St. Paul from our later
inferior commentaries?

>  * it should be possible to exclude the new tests when running the test
>    suite.

one can already exclude (or, rather, ignore the results of) any number
of tests using rt::*expected-failures*.

>  * a proposal explaining the justification for the test.

indeed, both the commit message and the test itself should bear a
brief explanation of why these results are expected.

>  * there should be at least N people who agree that the test checks for
>    something that matches with their understanding of the
>    specification.
>
> More to the last point:
>
> Preferably the N people represent different implementations.
>
> What's a good N? >= 2 ?
>
> Should a disapproval by as few as one person who can not be convinced
> differently in reasonable time be enough to deny a proposal? Perhaps
> this should be tightened up to a person representing an implementation.

Here we are stumbling into the question of what is our mandate.
 [1] Are we creating a comprehensive test of the existing ANSI CL standard?
 [2] Are we creating a de-facto extension to the ANSI CL standard by
finding what things are done uniformly by implementations but not
specified by ANSI?

if [1], then I don't think such veto power is a good idea.
I am not even sure that the fact that someone "represents an
implementation" should give that person extra weight.

I think that if someone (not necessarily representing an
implementation) thinks that the test is wrong, that person should
register the different opinion in the comment to the test in the
sources. If the opinion is valid, I think multiple results should be
accepted.
This, of course, should happen after a discussion on ansi-test-devel,
when it becomes clear that both opinions are not going away.

if [2], then, of course, you are right, and the implementers should
have the final say about what goes in.

I think we should stick with [1] on ansi-tests and leave [2] to ad-hoc
efforts like cl-ext:exit &c.

> Where should discussion take place? I'm sending this mail both to the
> implementation-hackers and ansi-test mailing list mostly to reach as
> many people who may be concerned.

IMO, ansi-tests is the right forum.
I wish I knew how to set Reply-to: in gmail.

-- 
Sam Steingold <http://sds.podval.org>