[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Vladimir Sedach]

Thank you for the specific suggestions Mark.

Mark Evenson <evenson at panix.com> writes:

> 1. Have a slot in your base condition class TEST-OP-TEST-FAILURE in
> which one can record the ASDF component which caused the failure.
> It is probably possible to dig this information out of the stack,
> but that will be messy. This would also allow for distinguishing
> when multiple TEST-OP-TEST-FAILURES are signaled from a single
> ASDF:TEST-OP invocation, as will be the case when one “chains” test
> invocation over many ASDF systems.

This is really easy to do with a special variable in an initarg, but
are there any systems that you know of that do this? I would
definitely like to test with them, because I thought that nested
TEST-OP was not supposed to work. From the "More Elaborate Testing"
section of the best practices document¹:

"You MUST NOT call asdf:operate or any of its derivatives, such as
asdf:load-system or asdf:test-system from within a perform method."

Unfortunately it looks like that is what ROVE:RUN-SYSTEM-TESTS does
exactly that.

> 2. Provide an implementation of the subclass of
> TEST-OP-TEST-FAILURE that contains the basic structure of a
> reporter class for the information that should be present in all
> test frameworks, namely the total number of tests run, the number
> of failed tests, the identities of the failed tests, and a slot for
> a human readable error message, along with a reporter function that
> displays this information. Having an implementation class to work
> from would make it easier for test frameworks to adapt.

I tried to avoid enforcing required slots, but as both
asdf-test-harness and cl-test-grid want a list of failed tests, that
is a strong case to make the above slots required in
TEST-OP-TEST-FAILURE itself.

cl-test-grid wants a list of test names as strings (it wants them
down-cased, but that is a detail that can be left to cl-test-grid). A
list of strings is a requirement that any test library should be able
to satisfy (worst case, it could be a list of random names), and
looks to me specific enough for most test harness use cases.

The length of the list of failed test names gives the count of failed
tests.

It seems to me like having a slot for an error message is redundant
with the reporter function, given that I think it should be up to the
test library to define the reporter function, and not for
TEST-OP-TEST-FAILURES to dictate how it is printed. That way, if a
test library has a flag to print results in machine readable format,
the flag will work without any changes if the overridden reporter
function re-uses the library's output facilities, and as long as the
test harness PRINCs the condition, the test harness does not need to
do anything either.

> 3. Go ahead and define the subclass of this condition when no tests
> have been run.

I thought about doing this, but with the above slots, there is no
need to - the test library can signal TEST-OP-TEST-FAILURE with a 0
count of total number of tests run.

> 4. As for adoption by test framework your strategy, we will have
> the problem that a given test framework won’t want to adopt the
> conditions because it isn’t in the version of ASDF they are using,
> or can easily get a hold of. To solve this, we might somehow define
> the code within the ASDF source tree so that one can make a
> standalone ASDF system (“ASDF-TEST-CONDITIONS” or some such) that
> one may include separately from actually upgrading ASDF.

That is something that Robert brought up on the merge request
discussion.² It looks like this can be handled with the #+ #- feature
macros or #. read macro to provide CL:WARNING as a fallback
super-class. I am open to any ideas.

I went ahead and added the slots to ASDF², the FiveAM³, and the rove⁴
implementations. Up next, I am going to work on adding support to
cl-test-grid for a library that uses rove, which cl-test-grid does
not support yet.

¹ https://github.com/fare/asdf/blob/master/doc/best_practices.md
² https://gitlab.common-lisp.net/asdf/asdf/merge_requests/124
³ https://github.com/sionescu/fiveam/pull/58
⁴ https://github.com/fukamachi/rove/pull/29

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la