[imp-hackers] [ansi-test-devel] Procedure for adding tests to the ansi test suite

[Sam Steingold]

On Tue, Jul 28, 2009 at 12:31 PM, Tobias C. Rittweiler<tcr at freebits.de> wrote:
> Sam Steingold writes:
>>
>> one can already exclude (or, rather, ignore the results of) any number
>> of tests using rt::*expected-failures*.
>
> People use the ansi test suite as a regression test for changes on their
> implementations. By having run the suite in the past, they're used to
> that currently N tests are failing.
>
> If new tests are added, it may be that suddenly N+M tests are failing,
> indicating regressions their changes introduced---which they actually
> did not do.

people who use the ansi-tests like this should be very careful before
they do "svn up" and should read the logs very carefully after they do
that.
they should also use rt::*expected-failures*.

Note that we cannot cripple the ansi test suite to placate those who
misuse it as a regression suite.

> Yes multiple results should be accepted where it makes sense. But if you
> test for behaviour, it's often the case that either an implementation
> exhibits the behaviour, or does not.

if the behavior is optional, there is no reason to test for it in the
ansi suite.
if it is mandatory, only one result should be accepted.
implementations are encouraged to grow their own regression test suites.

> Creating a test which can be both T
> and NIL does not make much sense.

no one is proposing this.

> Tests where people's opinions differ about could be tagged as being
> controversial which could be excluded from a run of the suite.

I am opposed to the notion that the "new" tests are somehow "inferior"
to the "old" tests.
if the users want to excluded certain tests, they can easily do that
from their own scaffolding.
see, e.g., clisp/utils/clispload.lsp.

-- 
Sam Steingold <http://sds.podval.org>