[asdf-devel] About regression tests and libraries
fahree at gmail.com
Thu Aug 6 01:23:29 UTC 2009
Sorry to parasite this mailing-list, but I'm trying to design the
testing aspect of XCVB right now. The constraint is that we want to be
able to use make as a backend, and that test results should thus be
reified as files if we want to avoid re-running already-computed
My dim ideas about it include the following:
* test reports are targets depending on properly compiled fasls or images,
produce test reports.
* a test report is a file the first form of which is a simple SEXP to
be read, following some standardized structure to specify overall
success, status of individual tests (including error message, maybe
information as to last success, etc.).
* from the test reports, a success witness (empty file) may be created
* the success witness fails to be created when the test wasn't a
success, triggering an error for make to catch.
* preparing the makefile erases the success witnesses.
If ASDF's test-op is to be standardized, may I suggest that we agree
on the format for a SEXP to be returned, that will report the test
[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
A child of five would understand this. Send someone to fetch a child of five.
-- Groucho Marx
2009/8/5 Robert Goldman <rpgoldman at sift.info>:
> Nikodemus Siivola wrote:
>> 2009/8/5 Robert Goldman <rpgoldman at sift.info>:
>>> Part of the problem with test-op is that the desired behavior has not
>>> been specified by the ASDF community. Because of the nature of ASDF, it
>>> is impossible for
>>> (asdf:test-system <system>)
>>> to return a value indicating whether or not <system> has passed its tests.
>> I would propose that as a matter of good style implementations of
>> TEST-OP should signal an error is tests do not pass. It is the
>> simplest way to make sure the result is not misinterpreted...
>> -- Nikodemus
> A second point is that test libraries are often very chatty, and it's
> unpleasant to have their output mixed in the same stream with the output
> of the compiler and whatever all else ASDF might happen to trigger
> splatting onto the output stream. So I really think that some stream
> argument is essential, even if we were to accept your proposal.
More information about the asdf-devel