[asdf-devel] About regression tests and libraries

[Robert Goldman]

Nikodemus Siivola wrote:
> 2009/8/5 Robert Goldman <rpgoldman at sift.info>:
> 
>> Part of the problem with test-op is that the desired behavior has not
>> been specified by the ASDF community.  Because of the nature of ASDF, it
>> is impossible for
>>
>> (asdf:test-system <system>)
>>
>> to return a value indicating whether or not <system> has passed its tests.
> 
> I would propose that as a matter of good style implementations of
> TEST-OP should signal an error is tests do not pass. It is the
> simplest way to make sure the result is not misinterpreted...
> 
>

I suppose, but typically a regression test suite should attempt to run
ALL of its tests, collecting a set of failing tests, and then returning
that information.  And demanding interactive attention ("I'm going to
throw you into the debugger if I fail") may also be undesirable in
practice.  I suppose handler-case would be enough for that, but I'm
still uncomfortable with the idea.

Best,
r