Introspecting test suite passing/failing

[Mark Evenson]

> On Jun 7, 2018, at 00:21, Robert Goldman <rpgoldman at sift.info> wrote:
> 
> When I replaced fiveam-tester-system with :fiveam-tester-system or "FIVEAM-TESTER-SYSTEM", the load went fine for me with the latest ASDF on Allegro, SBCL, and ABCL 1.5.0.

I must have been somehow introducing errors by being non-rigorous in my testing yesterday because when I woke up fresh today to revisit this problem, I was indeed able to get both the keyword package designator and the string package designator forms to work under sbcl-1.4.8, ccl-1.11.5, and lispworks-pro-7.1.0.0.

I worked over my code to the point that I am ready to start integrating with my test harness invocation code.  

Instead of using ERROR for raising the condition, I am [going to use SIGNAL][1].  This will mean that as long as no handler for ASDF:TEST-SUITE-FAILURES is installed, the test suite can be invoked by a developer normally.  But when the automated test runner invokes the tests with an [appropiate handler installed][2], I’ll be able to make the test invocation “go red."

[1]: https://github.com/emotiq/asdf-test-harness/blob/master/asdf-test-harness.lisp#L42

[2]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/run-with-handler.lisp

> Really, no one should be running with an ASDF that's older than 3.2 these days. I thought that all the OSS lisps were distributed with at least 3.2 now. Why are you running with 3.1.x?

I’m attempting to support a rather diverse set of Lisp hackers using a fair mix of development environments so it is organizationally easier for me to use the built-in ASDF.  asdf-3.1.1 is what ccl-1.11.5 ships with, so it is one of versions I would like to support.

My intention for ASDF-TEST-HARNESS will be to provide a single mechanism to support test invocation for variety of differing test frameworks.  The necessary [ASDF definition][3] will be quite simple, merely naming the test framework, and the package to test: 

(defsystem "asdf-test-harness-example"
  :defsystem-depends-on (asdf-test-harness)
  :class :harness-test
  :test-framework :lisp-unit
  :test-package :lisp-unit-example
  :components ((:file "lisp-unit-example")))

[3]: https://github.com/emotiq/asdf-test-harness/blob/master/eg/asdf-test-harness-example.asd

Almost all test frameworks which I have reviewed support a method to introspect a package for running the tests contained therein. All one will have to do to support a new framework will be write a specialization on a generic method which given the package name, runs the tests, collects the results, and returns a boolean truth value as to whether that test invocation should be considered pass (“Green”) or fail (“Red”).

Initially, I plan to support LISP-UNIT, PROVE, and FIVEAM.

Now, if I can only get Quicklisp to work again with :DEFSYSTEM-DEPENDS-ON clauses, I will be a pig in slop heaven…

Thanks very much for the help. 


-- 
"A screaming comes across the sky.  It has happened before but there is nothing 
to compare to it now."