PATCH: SIGNAL a condition on test failures, for use with ASDF:TEST-SYSTEM

[Robert Goldman]

Vladimir -- maybe compare this suggestion with what is in the 
fiveam-asdf contrib in the ASDF repo?

Also (and we should probably move this to ASDF-DEVEL) it would be good 
to have *all* of the tests run to completion even if one or more fail by 
default, and have the option to exit after the first failure.

This could be done by having an `ASDF-SINGLE-TEST-FAILURE` and an 
`ASDF-TEST-FAILURE` and raising the first at each failure, but by 
default trapping them and bundling them into the `ASDF-TEST-FAILURE` 
condition.

It might make sense to bundle FiveAM's test report *into* the 
`ASDF-TEST-FAILURE` (or subclass thereof), so that it can be printed at 
an appropriate time.

On 14 Sep 2019, at 20:48, Vladimir Sedach wrote:

> Hello!
>
> I have a proposed patch to FiveAM on GitHub:
> https://github.com/sionescu/fiveam/pull/58
>
> I recently wrote a script to run FiveAM tests for one of my
> libraries on many different implementations on your own machine:
> https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp
>
> I do not want to copy-paste that script for all of my libraries. It
> would be nice to contribute a generalized version of the script to
> Roswell (on which the script is based), and just be able to say "ros
> test my-system in all installed implementations" for any system.
>
> The first step to doing that is to get ASDF:TEST-SYSTEM to report
> errors using a common interface.
>
> Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use
> conditions to signal test failures. I like this idea. Since you do
> not have to handle conditions raised by SIGNAL, the functionality can
> be added without impacting existing use or interfaces of FiveAM (the
> use of WARNING instead of ERROR as the parent condition of
> TEST-SPEC-FAILURE also ensures this, as many people tend to abuse
> HANDLER-CASE to indiscriminately catch ERRORs).
>
> With this patch, all projects that use FiveAM and define ASDF test-op
> should be testable from a generic version of the Roswell script
> without any changes.
>
> The basic mechanism for test automation with FiveAM is then very
> simple:
>
> (handler-case (asdf:test-system "system")
>   (fiveam:test-spec-failure (condition)
>     (princ condition uiop:*stderr*)
>     (uiop:quit 1)))
>
> Other test libraries can have supported added as well:
>
> (handler-case (asdf:test-system "system")
>   ((or fiveam:test-spec-failure prove:test-failure) (condition)
>     (princ condition uiop:*stderr*)
>     (uiop:quit 1)))
>
> The goal is to introduce a parent condition in ASDF that FiveAM, etc., 
> will then inherit from:
>
> (handler-case (asdf:test-system "system")
>   (asdf:test-failure (condition)
>     (princ condition uiop:*stderr*)
>     (uiop:quit 1)))
>
> Ultimately, any system using any test library should be testable this 
> way, ensuring that test automation becomes trivial to build.
>
> I am going to propose adding the parent condition to ASDF, but in the
> meantime (until the ASDF patch makes it to the repository, another
> release comes out, release becomes widely available…), I would like 
> to get this code into FiveAM.
>
> Vladimir

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20190914/5fcf5e59/attachment.html>