[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Vladimir Sedach]

Robert Goldman <rpgoldman at sift.info> writes:

>> - success should also be signaled, so we can distinguish a version
>> where
>>   this new protocol is not implemented from the version where tests
>> pass
>
> This requires a protocol where ASDF can "know" when the test op is
> done, so that it can distinguish "has succeeded" from "has not failed
> yet."  It's possible that this could be managed by the test op on the
> system as a whole, but the design must consider the possibility that
> the test-op may be implemented so that:
>
> 1. Multiple calls are made into the test library (possibly even
> multiple test libraries are used)
> 2. TEST-OP may involve invoking test operation on systems depended on
> (e.g., where a system has subsystems).

There would be three situations here:

1. OPERATE TEST-OP returns with no relevant conditions signaled. You
can infer that the condition protocol is not implemented.
2. OPERATE TEST-OP returns and one or more test failure conditions
are signaled.
3. OPERATE TEST-OP returns and only test success conditions are
signaled.

>> - the minimal requirement is a success / failure designator, the
>> failed
>>   test names can be optional
>
> - Additional requirement: the condition should support accessing both
> a long and a short form report.  In degenerate implementations, these
> can, of course, be identical.

What would the long and short form reports look like?

>>   some corner cases. Still, even if we expect test results being
>> signalled
>>   multiple times during a test-op, it would be good to provide a
>> wrapper
>>   which aggregates them into a single return value.
>>
>>         (common-test-results:collect (asdf:test-system "my-system"))
>
> This would require that the test library provide an extensible
> protocol for fusing together multiple test results.

It is simpler than that: take all of the conditions, add the number
of tests executed, and append all of the test failure lists. No need
for library-specific code.

> And note that the above suggestion will not work, because ASDF does
> not ever *return* a value from operating. This has to do with the
> way ASDF creates a plan and then executes it. The plan doesn't
> support a notion of "return value," so the only way to get
> information out of ASDF is through conditions.

What COMMON-TEST-RESULTS:COLLECT would do is handle and coalesce
multiple conditions and re-signal a single condition.

> I agree -- I think `TRIVIAL-TEST-INTERFACE` might be a better first
> step.  I suppose the alternative rationale is that a test interface
> that was *not* intended for incorporation into ASDF would be able to
> just *return* things, instead of *signaling* them.

The point is, systems already define TEST-OP. I am trying to use
that. The code that uses TEST-OP can do whatever it needs to, but the
communication between that code and the test libraries has to be done
by stack-based mechanisms like conditions or special variables.

> That is true, but it's also true that it would require special
> condition-handling to fit test results into continuous integration
> -- programmers would no longer be able to just use `quit-on-error`,
> which is a very handy way to turn a lisp test into something that
> works in Jenkins or, for that matter, any infrastructure based on
> shell scripting.

Right now errors are not signaled on test failures in most
definitions of TEST-OP I looked at, so this is not something that is
currently going on. Neither is this something that would stop working
for anyone that has the signal-error-on-failure flags set for their
test library, or is throwing errors explicitly.

> I'd rather have to write code to handle errors when I *don't* want
> them, than have test failure not be an error.
>
> If I'm running interactively, it's not a bother to deal with this as
> an error condition -- I can easily get out of the debugger.  But
> writing a lot of code to catch `TEST-FAILURE` conditions and
> translate them into exit with non-zero status would be a pain.

Test libraries already have flags whether to signal errors on test
failures or not. Having the condition be a sub-class of error would
not only be annoying in the REPL, it would break whatever test
automation code uses these flags, and it would change the behavior of
TEST-OP, most of whose callers do not expect it to signal errors on
test failures right now. This is a lot of breakage of thousands of
existing systems, just to avoid doing the following in a few test
automation scripts:

(handler-case
    (asdf:test-system "some-system")
  (asdf:test-op-test-failures (condition)
    (princ condition uiop:*stderr*)
    (uiop:quit 1)))

>> - slot for the failing asdf system could probably be avoided,
>>   the list failed test names could be enough, if the names are
>> "fully qualified"
>>   i.e. include package or system name.
>
> I don't think we can make any assumptions about the above -- there's
> no rule about how a programmer can assign test names in a library
> like FiveAM to packages.

FiveAM test names are symbols, so they already get printed with their
package name in the implementation I did for FiveAM, without any
extra work.

> I would note also that getting a new library into Quicklisp for this
> is going to be a lot easier than getting a new ASDF into Quicklisp:
> Xach has for years refused to update the ASDF version in Quicklisp,
> and I don't see any reason to believe this will change.

As I mentioned before, I would like to avoid creating a whole library
out of something that is a work-around to OPERATE not returning
results.

Unfortunately, it seems there are both social and technical problems
with updating ASDF. In particular I do not see a good mechanism for
advertising the availability of this condition protocol to test
libraries (there does not seem to be an established way of
advertising new ASDF functionality other than the :ASDF3.3 :ASDF3.2
etc. keywords in *FEATURES*).

As Anton pointed out, this necessitates the libraries signaling a
condition for test success, which necessitates a function like
COMMON-TEST-RESULTS:COLLECT. While writing an implementation of that
function, I realized I would need to add continue restarts (the only
way to handle a condition without affecting control flow).

Take all of this together, and it becomes apparent that avoiding ASDF
to provide a more useful TEST-OP is, ironically, the way to go. It is
simpler to drive communications down the stack by binding special
variables to act as accumulators, than it is communicating up the
stack with signals, handlers, and restarts. Putting this into a
library means TEST-OP would still retain a use as a way to trigger
test runs without knowing details about either the tests or the test
library (the first half of this proposal), but it will unfortunately
mean that ASDF will continue to have no say about what the effects of
TEST-OP are (the second half of this proposal).

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la