[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Vladimir Sedach]

Mark Evenson <evenson at panix.com> writes:

> Each flavor of testing framework needs to write a simple adaptor
> that returns a condition containing a boolean indicating success or
> failure and the testing framework specific results.

I am not sure if you are aware of cl-test-grid¹ - it takes the
same approach.

Unfortunately, it has the same downsides. There is only a subset of
the test libraries supported. The code depends on the internal
workings of the test libraries and is often out of date. The projects
cannot be used as-is to test arbitrary systems (with cl-test-grid,
you have to manually add configuration for each system to
testsuites/testsuites.lisp, asdf-test-harness seems to depend on
emotiq configuration files).

This approach has not worked well in the past 8 years (cl-test-grid
was started in 2011), and I do not think it is viable.

What you have also done in asdf-test-harness, adding an
ASDF-TEST-SYSTEM-FAILURE condition to ASDF, is a better approach. I
have shown how this can be used by test libraries (with working
implementations for FiveAM³ and rove⁴). This removes the need for
asdf-test-harness, cl-test-grid, etc. to each have to implement
custom adapters for every different test library, completely
eliminating the dependency between test harnesses and particular test
libraries. The test libraries themselves do not need any new
dependencies either.

This decouples the systems and will make projects like
asdf-test-harness much easier to extend and maintain.

As an example, writing a script to test systems in different
implementations now becomes trivial.⁵

¹ https://github.com/cl-test-grid/cl-test-grid
² https://github.com/easye/asdf-test-harness/blob/master/asdf-test-harness.lisp
³ https://github.com/sionescu/fiveam/pull/58
⁴ https://github.com/vsedach/rove/commit/f6f8822eedc61d131c3b1d37b45c6d48cefcf633
⁵ https://gitlab.common-lisp.net/uri-template/uri-template2/blob/master/run-tests.lisp

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la