[asdf-devel] About regression tests and libraries

[Faré]

Sorry to parasite this mailing-list, but I'm trying to design the
testing aspect of XCVB right now. The constraint is that we want to be
able to use make as a backend, and that test results should thus be
reified as files if we want to avoid re-running already-computed
tests.

My dim ideas about it include the following:
* test reports are targets depending on properly compiled fasls or images,
 produce test reports.
* a test report is a file the first form of which is a simple SEXP to
be read, following some standardized structure to specify overall
success, status of individual tests (including error message, maybe
information as to last success, etc.).
* from the test reports, a success witness (empty file) may be created
on success.
* the success witness fails to be created when the test wasn't a
success, triggering an error for make to catch.
* preparing the makefile erases the success witnesses.

If ASDF's test-op is to be standardized, may I suggest that we agree
on the format for a SEXP to be returned, that will report the test
results?

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
A child of five would understand this. Send someone to fetch a child of five.
        -- Groucho Marx




2009/8/5 Robert Goldman <rpgoldman at sift.info>:
> Nikodemus Siivola wrote:
>> 2009/8/5 Robert Goldman <rpgoldman at sift.info>:
>>
>>> Part of the problem with test-op is that the desired behavior has not
>>> been specified by the ASDF community.  Because of the nature of ASDF, it
>>> is impossible for
>>>
>>> (asdf:test-system <system>)
>>>
>>> to return a value indicating whether or not <system> has passed its tests.
>>
>> I would propose that as a matter of good style implementations of
>> TEST-OP should signal an error is tests do not pass. It is the
>> simplest way to make sure the result is not misinterpreted...
>>
>> Cheers,
>>
>>  -- Nikodemus
>
> A second point is that test libraries are often very chatty, and it's
> unpleasant to have their output mixed in the same stream with the output
> of the compiler and whatever all else ASDF might happen to trigger
> splatting onto the output stream.  So I really think that some stream
> argument is essential, even if we were to accept your proposal.
>
> best,
> r