[asdf-devel] About regression tests and libraries

[Robert Goldman]

Faré wrote:
> Sorry to parasite this mailing-list, but I'm trying to design the
> testing aspect of XCVB right now. The constraint is that we want to be
> able to use make as a backend, and that test results should thus be
> reified as files if we want to avoid re-running already-computed
> tests.
> 
> My dim ideas about it include the following:
> * test reports are targets depending on properly compiled fasls or images,
>  produce test reports.
> * a test report is a file the first form of which is a simple SEXP to
> be read, following some standardized structure to specify overall
> success, status of individual tests (including error message, maybe
> information as to last success, etc.).
> * from the test reports, a success witness (empty file) may be created
> on success.
> * the success witness fails to be created when the test wasn't a
> success, triggering an error for make to catch.
> * preparing the makefile erases the success witnesses.
> 
> If ASDF's test-op is to be standardized, may I suggest that we agree
> on the format for a SEXP to be returned, that will report the test
> results?

Based on our experience in a large, multi-language framework, I'd
suggest a related enhancement:

* specify a translation from the s-expression test result format to XML
that complies with what junit expects.

In collaboration with folks from BBN, my colleague John Maraist has
built this capability into his NST test framework.  This means that a
large system with java components can all be tested together.

best,
r