<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
</head>
<body>
<div style="font-family:sans-serif"><div style="white-space:normal">
<p dir="auto">Vladimir -- maybe compare this suggestion with what is in the fiveam-asdf contrib in the ASDF repo?</p>
<p dir="auto">Also (and we should probably move this to ASDF-DEVEL) it would be good to have <em>all</em> of the tests run to completion even if one or more fail by default, and have the option to exit after the first failure.</p>
<p dir="auto">This could be done by having an <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">ASDF-SINGLE-TEST-FAILURE</code> and an <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">ASDF-TEST-FAILURE</code> and raising the first at each failure, but by default trapping them and bundling them into the <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">ASDF-TEST-FAILURE</code> condition.</p>
<p dir="auto">It might make sense to bundle FiveAM's test report <em>into</em> the <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">ASDF-TEST-FAILURE</code> (or subclass thereof), so that it can be printed at an appropriate time.</p>
<p dir="auto">On 14 Sep 2019, at 20:48, Vladimir Sedach wrote:</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">Hello!<br>
<br>
I have a proposed patch to FiveAM on GitHub:<br>
<a href="https://github.com/sionescu/fiveam/pull/58" style="color:#777">https://github.com/sionescu/fiveam/pull/58</a><br>
<br>
I recently wrote a script to run FiveAM tests for one of my<br>
libraries on many different implementations on your own machine:<br>
<a href="https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp" style="color:#777">https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp</a><br>
<br>
I do not want to copy-paste that script for all of my libraries. It<br>
would be nice to contribute a generalized version of the script to<br>
Roswell (on which the script is based), and just be able to say "ros<br>
test my-system in all installed implementations" for any system.<br>
<br>
The first step to doing that is to get ASDF:TEST-SYSTEM to report<br>
errors using a common interface.<br>
<br>
Because of the way ASDF is designed, ASDF:TEST-SYSTEM needs to use<br>
conditions to signal test failures. I like this idea. Since you do<br>
not have to handle conditions raised by SIGNAL, the functionality can<br>
be added without impacting existing use or interfaces of FiveAM (the<br>
use of WARNING instead of ERROR as the parent condition of<br>
TEST-SPEC-FAILURE also ensures this, as many people tend to abuse<br>
HANDLER-CASE to indiscriminately catch ERRORs).<br>
<br>
With this patch, all projects that use FiveAM and define ASDF test-op<br>
should be testable from a generic version of the Roswell script<br>
without any changes.<br>
<br>
The basic mechanism for test automation with FiveAM is then very<br>
simple:<br>
<br>
(handler-case (asdf:test-system "system")<br>
(fiveam:test-spec-failure (condition)<br>
(princ condition uiop:*stderr*)<br>
(uiop:quit 1)))<br>
<br>
Other test libraries can have supported added as well:<br>
<br>
(handler-case (asdf:test-system "system")<br>
((or fiveam:test-spec-failure prove:test-failure) (condition)<br>
(princ condition uiop:*stderr*)<br>
(uiop:quit 1)))<br>
<br>
The goal is to introduce a parent condition in ASDF that FiveAM, etc., will then inherit from:<br>
<br>
(handler-case (asdf:test-system "system")<br>
(asdf:test-failure (condition)<br>
(princ condition uiop:*stderr*)<br>
(uiop:quit 1)))<br>
<br>
Ultimately, any system using any test library should be testable this way, ensuring that test automation becomes trivial to build.<br>
<br>
I am going to propose adding the parent condition to ASDF, but in the<br>
meantime (until the ASDF patch makes it to the repository, another<br>
release comes out, release becomes widely available…), I would like to get this code into FiveAM.<br>
<br>
Vladimir</p>
</blockquote></div>
<div style="white-space:normal">
</div>
</div>
</body>
</html>