[imp-hackers] [ansi-test-devel] Procedure for adding tests to the ansi test suite

[Raymond Toy]

Sam Steingold wrote:
> On Tue, Jul 28, 2009 at 12:31 PM, Tobias C. Rittweiler<tcr at freebits.de> wrote:
>   
>> Sam Steingold writes:
>>     
>>> one can already exclude (or, rather, ignore the results of) any number
>>> of tests using rt::*expected-failures*.
>>>       
>> People use the ansi test suite as a regression test for changes on their
>> implementations. By having run the suite in the past, they're used to
>> that currently N tests are failing.
>>
>> If new tests are added, it may be that suddenly N+M tests are failing,
>> indicating regressions their changes introduced---which they actually
>> did not do.
>>     
>
> people who use the ansi-tests like this should be very careful before
> they do "svn up" and should read the logs very carefully after they do
> that.
> they should also use rt::*expected-failures*.
>
> Note that we cannot cripple the ansi test suite to placate those who
> misuse it as a regression suite.
>   
I think I agree with Sam on all points. 

The one nice thing to have would be able to turn on or off (defaulting
to on) tests.  For example, it is unlikely that cmucl will support the
nil vectors are strings (or whatever that hairy case is), or that cmucl
will support specialized arrays for unsigned-byte 7, 15, and 31, which
seem to be required.

Ray (who does use the ansi-tests as a regression suite)