<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
</head>
<body>
<div style="font-family:sans-serif"><div style="white-space:normal">
<p dir="auto">On 7 Jun 2018, at 1:03, Mark Evenson wrote:</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><blockquote style="border-left:2px solid #777; color:#999; margin:0 0 5px; padding-left:5px; border-left-color:#999"><p dir="auto">On Jun 7, 2018, at 00:21, Robert Goldman <rpgoldman@sift.info> wrote:<br>
<br>
When I replaced fiveam-tester-system with :fiveam-tester-system or "FIVEAM-TESTER-SYSTEM", the load went fine for me with the latest ASDF on Allegro, SBCL, and ABCL 1.5.0.</p>
</blockquote><p dir="auto">I must have been somehow introducing errors by being non-rigorous in my testing yesterday because when I woke up fresh today to revisit this problem, I was indeed able to get both the keyword package designator and the string package designator forms to work under sbcl-1.4.8, ccl-1.11.5, and lispworks-pro-7.1.0.0.<br>
<br>
I worked over my code to the point that I am ready to start integrating with my test harness invocation code.<br>
<br>
Instead of using ERROR for raising the condition, I am [going to use SIGNAL][1]. This will mean that as long as no handler for ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked by a developer normally. But when the automated test runner invokes the tests with an [appropiate handler installed][2], I’ll be able to make the test invocation “go red."<br>
<br>
[1]: <a href="https://github.com/emotiq/asdf-test-harness/blob/master/asdf-test-harness.lisp#L42" style="color:#777">https://github.com/emotiq/asdf-test-harness/blob/master/asdf-test-harness.lisp#L42</a><br>
<br>
[2]: <a href="https://github.com/emotiq/asdf-test-harness/blob/master/eg/run-with-handler.lisp" style="color:#777">https://github.com/emotiq/asdf-test-harness/blob/master/eg/run-with-handler.lisp</a></p>
</blockquote></div>
<div style="white-space:normal">
<p dir="auto">This sounds like an improvement. I may import it into the main version of fiveam-asdf.</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><blockquote style="border-left:2px solid #777; color:#999; margin:0 0 5px; padding-left:5px; border-left-color:#999"><p dir="auto">Really, no one should be running with an ASDF that's older than 3.2 these days. I thought that all the OSS lisps were distributed with at least 3.2 now. Why are you running with 3.1.x?</p>
</blockquote><p dir="auto">I’m attempting to support a rather diverse set of Lisp hackers using a fair mix of development environments so it is organizationally easier for me to use the built-in ASDF. asdf-3.1.1 is what ccl-1.11.5 ships with, so it is one of versions I would like to support.<br>
<br>
My intention for ASDF-TEST-HARNESS will be to provide a single mechanism to support test invocation for variety of differing test frameworks. The necessary [ASDF definition][3] will be quite simple, merely naming the test framework, and the package to test:<br>
<br>
(defsystem "asdf-test-harness-example"<br>
:defsystem-depends-on (asdf-test-harness)<br>
:class :harness-test<br>
:test-framework :lisp-unit<br>
:test-package :lisp-unit-example<br>
:components ((:file "lisp-unit-example")))<br>
<br>
[3]: <a href="https://github.com/emotiq/asdf-test-harness/blob/master/eg/asdf-test-harness-example.asd" style="color:#777">https://github.com/emotiq/asdf-test-harness/blob/master/eg/asdf-test-harness-example.asd</a></p>
</blockquote></div>
<div style="white-space:normal">
<p dir="auto">That sounds interesting. Note for some bigger systems we have had test operations that work over multiple packages (hence the longer form for the specification in the fiveam-tester-system class.</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">Almost all test frameworks which I have reviewed support a method to introspect a package for running the tests contained therein. All one will have to do to support a new framework will be write a specialization on a generic method which given the package name, runs the tests, collects the results, and returns a boolean truth value as to whether that test invocation should be considered pass (“Green”) or fail (“Red”).</p>
</blockquote></div>
<div style="white-space:normal">
<p dir="auto">OK, but note that you need a means of handling cases where running one <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">test-system</code> involves testing multiple subsidiary systems. In which case the boolean return will have to be handled carefully, because presumably you want your overall test to have <code style="background-color:#F7F7F7; border-radius:3px; margin:0; padding:0 0.4em" bgcolor="#F7F7F7">AND</code> logical semantics. You'll have to signal component system test failures and then have a handler that will defer them until all the tests are finally run.</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">Initially, I plan to support LISP-UNIT, PROVE, and FIVEAM.</p>
</blockquote></div>
<div style="white-space:normal">
<p dir="auto">This sounds great. We could talk about whether to continue to use asdf/contribs/ for this. Maybe not, since you will presumably want to maintain this code in your own repo.</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">Now, if I can only get Quicklisp to work again with :DEFSYSTEM-DEPENDS-ON clauses, I will be a pig in slop heaven…</p>
</blockquote></div>
<div style="white-space:normal">
<p dir="auto">I'm not sure what this issue is, but it's complicated by the fact that QL operates with such an old version of ASDF. In effect this means that QL will operate with whatever is the current ASDF in your environment, because ASDF won't downgrade itself. This means that people are running with all kinds of different ASDF versions, making debugging difficult.</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">Thanks very much for the help.</p>
</blockquote></div>
<div style="white-space:normal">
<p dir="auto">No problem!</p>
<p dir="auto">Best,<br>
r</p>
</div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">-- <br>
"A screaming comes across the sky. It has happened before but there is nothing<br>
to compare to it now."</p>
</blockquote></div>
<div style="white-space:normal">
</div>
</div>
</body>
</html>