PATCH: SIGNAL a condition on test failures, for use with ASDF:TEST-SYSTEM

[Faré]

Note that my recommended solution is for a test library to create a
report file with a test-report-op (that always succeeds at creating a
report file, but the report file may indicate failure), and the
test-op just depends on the test-report-op, and signals and error if
it didn't indicate success.

For extra win, the test system may even use fine-grained dependencies,
wherein each file in the test system would depend on loading a
specific file (or set of files) in the tested system rather than on
loading the entire system; then it would not recompute a test if its
dependencies have not changed—but that probably wouldn't be safe
unless these dependencies are actually enforced like xcvb or bazelisp
would (i.e. uiop wouldn't do).

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
#.(let((q`(#0=lambda(q)(labels((p(q)(if(atom q)(intern(#1=reverse(string
q)))(#1#(mapcar #'p q))))(q(q)(subst q(eq q q)'(#2=defun p(&aux #2#nufed
,@''#3=etouq(xua& #3#)p tsil)((#0#(q)(setq q 't tsil q nufed(eval(list q
(list'quote q)))))#3#)))))(nconc(q q)(p(q q)))))))(eval`(,q',q)))

On Sat, Sep 14, 2019 at 9:58 PM Robert Goldman <rpgoldman at sift.info> wrote:
>
> Vladimir -- maybe compare this suggestion with what is in the fiveam-asdf contrib in the ASDF repo?
>
> Also (and we should probably move this to ASDF-DEVEL) it would be good to have all of the tests run to completion even if one or more fail by default, and have the option to exit after the first failure.
>
> This could be done by having an ASDF-SINGLE-TEST-FAILURE and an ASDF-TEST-FAILURE and raising the first at each failure, but by default trapping them and bundling them into the ASDF-TEST-FAILURE condition.
>
> It might make sense to bundle FiveAM's test report into the ASDF-TEST-FAILURE (or subclass thereof), so that it can be printed at an appropriate time.
>
> On 14 Sep 2019, at 20:48, Vladimir Sedach wrote:
>
> Hello!
>
> I have a proposed patch to FiveAM on GitHub:
> https://github.com/sionescu/fiveam/pull/58
>
> I recently wrote a script to run FiveAM tests for one of my
> libraries on many different implementations on your own machine:
> https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp
>
> I do not want to copy-paste that script for all of my libraries. It
> would be nice to contribute a generalized version of the script to
> Roswell (on which the script is based), and just be able to say "ros
> test my-system in all installed implementations" for any system.
>
> The first step to doing that is to get ASDF:TEST-SYSTEM to report
> errors using a common interface.
>
> Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use
> conditions to signal test failures. I like this idea. Since you do
> not have to handle conditions raised by SIGNAL, the functionality can
> be added without impacting existing use or interfaces of FiveAM (the
> use of WARNING instead of ERROR as the parent condition of
> TEST-SPEC-FAILURE also ensures this, as many people tend to abuse
> HANDLER-CASE to indiscriminately catch ERRORs).
>
> With this patch, all projects that use FiveAM and define ASDF test-op
> should be testable from a generic version of the Roswell script
> without any changes.
>
> The basic mechanism for test automation with FiveAM is then very
> simple:
>
> (handler-case (asdf:test-system "system")
> (fiveam:test-spec-failure (condition)
> (princ condition uiop:*stderr*)
> (uiop:quit 1)))
>
> Other test libraries can have supported added as well:
>
> (handler-case (asdf:test-system "system")
> ((or fiveam:test-spec-failure prove:test-failure) (condition)
> (princ condition uiop:*stderr*)
> (uiop:quit 1)))
>
> The goal is to introduce a parent condition in ASDF that FiveAM, etc., will then inherit from:
>
> (handler-case (asdf:test-system "system")
> (asdf:test-failure (condition)
> (princ condition uiop:*stderr*)
> (uiop:quit 1)))
>
> Ultimately, any system using any test library should be testable this way, ensuring that test automation becomes trivial to build.
>
> I am going to propose adding the parent condition to ASDF, but in the
> meantime (until the ASDF patch makes it to the repository, another
> release comes out, release becomes widely available…), I would like to get this code into FiveAM.
>
> Vladimir