[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Anton Vodonosov]

Vladmir, the library content is not just the signal trick (or special vars).

It defines the unified common test result representation, provides a bridge
between the protocol test libraries "signal" their result and the protocol
the user running tests consumes the results. The library can also
provide means to create the common test results as a normal return value;
or maybe deliver it through a stack-based mechanism, but explicitly,
without relying on test frameworks to do it automatically (in some cases
I anticipate explicit creation of the result will be more convenient)

And yes, the most important, adding such functionality into the public ASDF API
requires too much upfront thinking. After separate library experiments with
the approach and stabilizes, ASDF can incorporate, or officially refer it.

As Robert mentions in the pull request, having this functionality in new
versions of ASDF would require the consumer to check feature flags
in their code. In the case of a separate library, the consumer can just load
this library, and it will work even with old versions of ASDF.




04.10.2019, 06:39, "Vladimir Sedach" <vas at oneofus.la>:
> Robert Goldman <rpgoldman at sift.info> writes:
>
>>>  - success should also be signaled, so we can distinguish a version
>>>  where
>>>    this new protocol is not implemented from the version where tests
>>>  pass
>>
>>  This requires a protocol where ASDF can "know" when the test op is
>>  done, so that it can distinguish "has succeeded" from "has not failed
>>  yet." It's possible that this could be managed by the test op on the
>>  system as a whole, but the design must consider the possibility that
>>  the test-op may be implemented so that:
>>
>>  1. Multiple calls are made into the test library (possibly even
>>  multiple test libraries are used)
>>  2. TEST-OP may involve invoking test operation on systems depended on
>>  (e.g., where a system has subsystems).
>
> There would be three situations here:
>
> 1. OPERATE TEST-OP returns with no relevant conditions signaled. You
> can infer that the condition protocol is not implemented.
> 2. OPERATE TEST-OP returns and one or more test failure conditions
> are signaled.
> 3. OPERATE TEST-OP returns and only test success conditions are
> signaled.
>
>>>  - the minimal requirement is a success / failure designator, the
>>>  failed
>>>    test names can be optional
>>
>>  - Additional requirement: the condition should support accessing both
>>  a long and a short form report. In degenerate implementations, these
>>  can, of course, be identical.
>
> What would the long and short form reports look like?
>
>>>    some corner cases. Still, even if we expect test results being
>>>  signalled
>>>    multiple times during a test-op, it would be good to provide a
>>>  wrapper
>>>    which aggregates them into a single return value.
>>>
>>>          (common-test-results:collect (asdf:test-system "my-system"))
>>
>>  This would require that the test library provide an extensible
>>  protocol for fusing together multiple test results.
>
> It is simpler than that: take all of the conditions, add the number
> of tests executed, and append all of the test failure lists. No need
> for library-specific code.
>
>>  And note that the above suggestion will not work, because ASDF does
>>  not ever *return* a value from operating. This has to do with the
>>  way ASDF creates a plan and then executes it. The plan doesn't
>>  support a notion of "return value," so the only way to get
>>  information out of ASDF is through conditions.
>
> What COMMON-TEST-RESULTS:COLLECT would do is handle and coalesce
> multiple conditions and re-signal a single condition.
>
>>  I agree -- I think `TRIVIAL-TEST-INTERFACE` might be a better first
>>  step. I suppose the alternative rationale is that a test interface
>>  that was *not* intended for incorporation into ASDF would be able to
>>  just *return* things, instead of *signaling* them.
>
> The point is, systems already define TEST-OP. I am trying to use
> that. The code that uses TEST-OP can do whatever it needs to, but the
> communication between that code and the test libraries has to be done
> by stack-based mechanisms like conditions or special variables.
>
>>  That is true, but it's also true that it would require special
>>  condition-handling to fit test results into continuous integration
>>  -- programmers would no longer be able to just use `quit-on-error`,
>>  which is a very handy way to turn a lisp test into something that
>>  works in Jenkins or, for that matter, any infrastructure based on
>>  shell scripting.
>
> Right now errors are not signaled on test failures in most
> definitions of TEST-OP I looked at, so this is not something that is
> currently going on. Neither is this something that would stop working
> for anyone that has the signal-error-on-failure flags set for their
> test library, or is throwing errors explicitly.
>
>>  I'd rather have to write code to handle errors when I *don't* want
>>  them, than have test failure not be an error.
>>
>>  If I'm running interactively, it's not a bother to deal with this as
>>  an error condition -- I can easily get out of the debugger. But
>>  writing a lot of code to catch `TEST-FAILURE` conditions and
>>  translate them into exit with non-zero status would be a pain.
>
> Test libraries already have flags whether to signal errors on test
> failures or not. Having the condition be a sub-class of error would
> not only be annoying in the REPL, it would break whatever test
> automation code uses these flags, and it would change the behavior of
> TEST-OP, most of whose callers do not expect it to signal errors on
> test failures right now. This is a lot of breakage of thousands of
> existing systems, just to avoid doing the following in a few test
> automation scripts:
>
> (handler-case
>     (asdf:test-system "some-system")
>   (asdf:test-op-test-failures (condition)
>     (princ condition uiop:*stderr*)
>     (uiop:quit 1)))
>
>>>  - slot for the failing asdf system could probably be avoided,
>>>    the list failed test names could be enough, if the names are
>>>  "fully qualified"
>>>    i.e. include package or system name.
>>
>>  I don't think we can make any assumptions about the above -- there's
>>  no rule about how a programmer can assign test names in a library
>>  like FiveAM to packages.
>
> FiveAM test names are symbols, so they already get printed with their
> package name in the implementation I did for FiveAM, without any
> extra work.
>
>>  I would note also that getting a new library into Quicklisp for this
>>  is going to be a lot easier than getting a new ASDF into Quicklisp:
>>  Xach has for years refused to update the ASDF version in Quicklisp,
>>  and I don't see any reason to believe this will change.
>
> As I mentioned before, I would like to avoid creating a whole library
> out of something that is a work-around to OPERATE not returning
> results.
>
> Unfortunately, it seems there are both social and technical problems
> with updating ASDF. In particular I do not see a good mechanism for
> advertising the availability of this condition protocol to test
> libraries (there does not seem to be an established way of
> advertising new ASDF functionality other than the :ASDF3.3 :ASDF3.2
> etc. keywords in *FEATURES*).
>
> As Anton pointed out, this necessitates the libraries signaling a
> condition for test success, which necessitates a function like
> COMMON-TEST-RESULTS:COLLECT. While writing an implementation of that
> function, I realized I would need to add continue restarts (the only
> way to handle a condition without affecting control flow).
>
> Take all of this together, and it becomes apparent that avoiding ASDF
> to provide a more useful TEST-OP is, ironically, the way to go. It is
> simpler to drive communications down the stack by binding special
> variables to act as accumulators, than it is communicating up the
> stack with signals, handlers, and restarts. Putting this into a
> library means TEST-OP would still retain a use as a way to trigger
> test runs without knowing details about either the tests or the test
> library (the first half of this proposal), but it will unfortunately
> mean that ASDF will continue to have no say about what the effects of
> TEST-OP are (the second half of this proposal).
>
> --
> Vladimir Sedach
> Software engineering services in Los Angeles https://oneofus.la