[asdf-devel] About regression tests and libraries

[Robert Goldman]

Tobias C. Rittweiler wrote:
> Nikodemus Siivola writes:
> 
>> 2009/8/5 Robert Goldman <rpgoldman at sift.info>:
>>
>>> Part of the problem with test-op is that the desired behavior has not
>>> been specified by the ASDF community.  Because of the nature of ASDF, it
>>> is impossible for
>>>
>>> (asdf:test-system <system>)
>>>
>>> to return a value indicating whether or not <system> has passed its tests.
>> I would propose that as a matter of good style implementations of
>> TEST-OP should signal an error is tests do not pass. It is the
>> simplest way to make sure the result is not misinterpreted...
> 
> You wrote "tests do not pass"; just to make sure, the plural was
> intentional, right? You did /not/ mean it should signal an error for
> each failing test, did you?
> 
> If I'm running the TEST-OP on some system (with the user, not developer,
> hat on my head), I'd like to get a nice summary about all failing tests
> to give a sense about the state of the software. I'm not interested in
> the minutae of the tests themselves.
> 
> Signalling an ASDF:TEST-OP-FAILED condition, perhaps even making that
> hidden in some function, so people can just write (ASDF:TEST-OP-FAIL)
> within their PERFORM method, is sure an idea if that better fits the
> architecture of ASDF.

This would actually accommodate itself nicely to the non-value-returning
structure of ASDF.

A remaining trouble:  because of the flat plan-then-execute nature of
ASDF, I don't believe the stack contents while performing a test-op
lends itself to accumulating together results of component tests.

Consider a case where we have a system A and in order to do a TEST-OP on
that system, one needs to do a TEST-OP on B and C.  I don't believe that
(PERFORM TEST-OP A) will be on the stack when doing (PERFORM 'TEST-OP
B), so that there's no obvious way to grab up the test results from B
and accumulate them inside the TEST-OP-FAILED condition object for A.

If I understand correctly, TRAVERSE does a postorder traversal of the
tree of systems below A, meaning that (PERFORM TEST-OP A) will be called
after (PERFORM TEST-OP B) and (PERFORM TEST-OP C) have exited.

Best,
r