Introspecting test suite passing/failing

[Robert Goldman]

On 7 Jun 2018, at 1:03, Mark Evenson wrote:

>> On Jun 7, 2018, at 00:21, Robert Goldman <rpgoldman at sift.info> wrote:
>>
>> When I replaced fiveam-tester-system with :fiveam-tester-system or 
>> "FIVEAM-TESTER-SYSTEM", the load went fine for me with the latest 
>> ASDF on Allegro, SBCL, and ABCL 1.5.0.
>
> I must have been somehow introducing errors by being non-rigorous in 
> my testing yesterday because when I woke up fresh today to revisit 
> this problem, I was indeed able to get both the keyword package 
> designator and the string package designator forms to work under 
> sbcl-1.4.8, ccl-1.11.5, and lispworks-pro-7.1.0.0.
>
> I worked over my code to the point that I am ready to start 
> integrating with my test harness invocation code.
>
> Instead of using ERROR for raising the condition, I am [going to use 
> SIGNAL][1].  This will mean that as long as no handler for 
> ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked 
> by a developer normally.  But when the automated test runner invokes 
> the tests with an [appropiate handler installed][2], I’ll be able to 
> make the test invocation “go red."
>
> [1]: 
> https://github.com/emotiq/asdf-test-harness/blob/master/asdf-test-harness.lisp#L42
>
> [2]: 
> https://github.com/emotiq/asdf-test-harness/blob/master/eg/run-with-handler.lisp

This sounds like an improvement.  I may import it into the main version 
of fiveam-asdf.
>
>> Really, no one should be running with an ASDF that's older than 3.2 
>> these days. I thought that all the OSS lisps were distributed with at 
>> least 3.2 now. Why are you running with 3.1.x?
>
> I’m attempting to support a rather diverse set of Lisp hackers using 
> a fair mix of development environments so it is organizationally 
> easier for me to use the built-in ASDF.  asdf-3.1.1 is what ccl-1.11.5 
> ships with, so it is one of versions I would like to support.
>
> My intention for ASDF-TEST-HARNESS will be to provide a single 
> mechanism to support test invocation for variety of differing test 
> frameworks.  The necessary [ASDF definition][3] will be quite simple, 
> merely naming the test framework, and the package to test:
>
> (defsystem "asdf-test-harness-example"
>   :defsystem-depends-on (asdf-test-harness)
>   :class :harness-test
>   :test-framework :lisp-unit
>   :test-package :lisp-unit-example
>   :components ((:file "lisp-unit-example")))
>
> [3]: 
> https://github.com/emotiq/asdf-test-harness/blob/master/eg/asdf-test-harness-example.asd

That sounds interesting.  Note for some bigger systems we have had test 
operations that work over multiple packages (hence the longer form for 
the specification in the fiveam-tester-system class.
>
> Almost all test frameworks which I have reviewed support a method to 
> introspect a package for running the tests contained therein. All one 
> will have to do to support a new framework will be write a 
> specialization on a generic method which given the package name, runs 
> the tests, collects the results, and returns a boolean truth value as 
> to whether that test invocation should be considered pass 
> (“Green”) or fail (“Red”).

OK, but note that you need a means of handling cases where running one 
`test-system` involves testing multiple subsidiary systems.  In which 
case the boolean return will have to be handled carefully, because 
presumably you want your overall test to have `AND` logical semantics.  
You'll have to signal component system test failures and then have a 
handler that will defer them until all the tests are finally run.
>
> Initially, I plan to support LISP-UNIT, PROVE, and FIVEAM.

This sounds great.  We could talk about whether to continue to use 
asdf/contribs/ for this.  Maybe not, since you will presumably want to 
maintain this code in your own repo.
>
> Now, if I can only get Quicklisp to work again with 
> :DEFSYSTEM-DEPENDS-ON clauses, I will be a pig in slop heaven…

I'm not sure what this issue is, but it's complicated by the fact that 
QL operates with such an old version of ASDF.  In effect this means that 
QL will operate with whatever is the current ASDF in your environment, 
because ASDF won't downgrade itself.  This means that people are running 
with all kinds of different ASDF versions, making debugging difficult.
>
> Thanks very much for the help.

No problem!

Best,
r

>
>
> -- 
> "A screaming comes across the sky.  It has happened before but there 
> is nothing
> to compare to it now."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20180607/6e45a66e/attachment.html>