Introspecting test suite passing/failing

[Robert Goldman]

I have pushed the system `fiveam-asdf`, which supports integration 
between ASDF and the FIVEAM test library, to the contribs directory in 
the ASDF repo.  Please have a look.

Be warned! It is old, and not being broke, hasn't been fixed.  It 
inappropriately is housed in the `ASDF` package and inappropriately 
exports extensions from that package.

But I believe it's still useful as an example of how to raise conditions 
when the test operation goes wrong.  Catching those exceptions can be 
used to cause a build to fail in a CI system, typically by running lisp 
in batch mode and having it exit with a nonzero error code if the test 
operation fails.

Best,
r


On 5 Jun 2018, at 4:09, Mark Evenson wrote:

> We use ASDF to encapsulate the building and testing many systems under 
> an
> automated test runner.
>
> For a given system, ASDF:TEST-SYSTEM always returns boolean truth as 
> long as
> the invocation of the underlying test suite succeeds.
>
> This means there is no programatic way to provide a boolean as to 
> whether all
> the tests passed or not on the invocation of a test suite to the 
> invoker of
> ASDF:TEST-SYSTEM.
>
> It seems that this issue has been raised before, as the ASDF manual 
> documents
> TEST-OP 
> <https://www.common-lisp.net/project/asdf/asdf/Predefined-operations-of-ASDF.html#Predefined-operations-of-ASDF> 
> with:
>
>> The results of this operation are not defined by ASDF. It has proven 
>> difficult
>> to define how the test operation should signal its results to the 
>> user in a way
>> that is compatible with all of the various test libraries and test 
>> techniques
>> in use in the community, and given the fact that ASDF operations do 
>> not return
>> a value indicating success or failure. For those willing to go to the 
>> effort,
>> we suggest defining conditions to signal when a test-op fails, and 
>> storing in
>> those conditions information that describes which tests fail.
>
> Is this still the best current practice to introspect the situation of 
> failing
> tests?  Can someone point me to an example implementation of this 
> technique?
>
>
> -- 
> "A screaming comes across the sky.  It has happened before but there 
> is nothing
> to compare to it now."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20180605/4e94a118/attachment.html>