[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Mark Evenson]

> On Sep 25, 2019, at 21:06, Vladimir Sedach <vas at oneofus.la> wrote:
> 
> 
> Mark Evenson <evenson at panix.com> writes:
> 
>> Each flavor of testing framework needs to write a simple adaptor
>> that returns a condition containing a boolean indicating success or
>> failure and the testing framework specific results.

> Unfortunately, it has the same downsides. There is only a subset of
> the test libraries supported. The code depends on the internal
> workings of the test libraries and is often out of date. The projects
> cannot be used as-is to test arbitrary systems (with cl-test-grid,
> you have to manually add configuration for each system to
> testsuites/testsuites.lisp, asdf-test-harness seems to depend on
> emotiq configuration files).
> 
> This approach has not worked well in the past 8 years (cl-test-grid
> was started in 2011), and I do not think it is viable.
> 
> What you have also done in asdf-test-harness, adding an
> ASDF-TEST-SYSTEM-FAILURE condition to ASDF, is a better approach. I
> have shown how this can be used by test libraries (with working
> implementations for FiveAM³ and rove⁴). This removes the need for
> asdf-test-harness, cl-test-grid, etc. to each have to implement
> custom adapters for every different test library, completely
> eliminating the dependency between test harnesses and particular test
> libraries. The test libraries themselves do not need any new
> dependencies either.
> 
> This decouples the systems and will make projects like
> asdf-test-harness much easier to extend and maintain.

Thanks for taking the time to understand my code enough to cogently critique
the two approaches.

As I understand it, both approaches share the need to add a condition to ASDF
that indicates that an invocation of an ASDF test-op has failed.

The difference is that you propose getting changes to all the various test
framework to signal this error condition with a subclass that contains
meaningful information for that given framework.  Whereas, I (and
cl-test-grid)--who was in a hurry to get something working without being able
to wait for modifications to the test suites--needs to write an adaptor for
each test suite.  And consequently these adaptors will be often be “brittle”
depending on implementation details that may not be constant over time to work.

If this is indeed an accurate summary, then I would endorse your patch to ASDF
as necessary for both approaches with the following suggestions for
improvements:

1.  Have a slot in your base condition class TEST-OP-TEST-FAILURE in which  one
can record the ASDF component which caused the failure.  It is probably
possible to dig this information out of the stack, but that will be messy. This
would also allow for distinguishing when multiple TEST-OP-TEST-FAILURES are
signaled from a single ASDF:TEST-OP invocation, as will be the case when one
“chains” test invocation over many ASDF systems.

2.  Provide an implementation of the subclass of TEST-OP-TEST-FAILURE that
contains the basic structure of a reporter class for the  information that
should be present in all test frameworks, namely the total number of tests run,
the number of failed tests, the identities of the failed tests, and a slot for
a human readable error message, along with a reporter function that displays
this information.  Having an implementation class to work from would make it
easier for test frameworks to adapt.

3.  Go ahead and define the subclass of this condition when no tests have been
run.

4.  As for adoption by test framework your strategy, we will have the problem
that a given test framework won’t want to adopt the conditions because it isn’t
in the version of ASDF they are using, or can easily get a hold of.  To solve
this, we might somehow define the code within the ASDF source tree so that one
can make a standalone ASDF system (“ASDF-TEST-CONDITIONS” or some such) that
one may include separately from actually upgrading ASDF.

Sincerely, 
Mark Evenson

-- 
"A screaming comes across the sky.  It has happened before but there is nothing 
to compare to it now."