[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Anton Vodonosov]

Also, the library can provide serialization of the common test result to files,
thus simplifying implementation of the test-report-op, as Fare mentions.

08.10.2019, 07:17, "Anton Vodonosov" <avodonosov at yandex.ru>:
> Vladmir, the library content is not just the signal trick (or special vars).
>
> It defines the unified common test result representation, provides a bridge
> between the protocol test libraries "signal" their result and the protocol
> the user running tests consumes the results. The library can also
> provide means to create the common test results as a normal return value;
> or maybe deliver it through a stack-based mechanism, but explicitly,
> without relying on test frameworks to do it automatically (in some cases
> I anticipate explicit creation of the result will be more convenient)
>
> And yes, the most important, adding such functionality into the public ASDF API
> requires too much upfront thinking. After separate library experiments with
> the approach and stabilizes, ASDF can incorporate, or officially refer it.
>
> As Robert mentions in the pull request, having this functionality in new
> versions of ASDF would require the consumer to check feature flags
> in their code. In the case of a separate library, the consumer can just load
> this library, and it will work even with old versions of ASDF.
>
> 04.10.2019, 06:39, "Vladimir Sedach" <vas at oneofus.la>:
>>  Robert Goldman <rpgoldman at sift.info> writes:
>>
>>>>   - success should also be signaled, so we can distinguish a version
>>>>   where
>>>>     this new protocol is not implemented from the version where tests
>>>>   pass
>>>
>>>   This requires a protocol where ASDF can "know" when the test op is
>>>   done, so that it can distinguish "has succeeded" from "has not failed
>>>   yet." It's possible that this could be managed by the test op on the
>>>   system as a whole, but the design must consider the possibility that
>>>   the test-op may be implemented so that:
>>>
>>>   1. Multiple calls are made into the test library (possibly even
>>>   multiple test libraries are used)
>>>   2. TEST-OP may involve invoking test operation on systems depended on
>>>   (e.g., where a system has subsystems).
>>
>>  There would be three situations here:
>>
>>  1. OPERATE TEST-OP returns with no relevant conditions signaled. You
>>  can infer that the condition protocol is not implemented.
>>  2. OPERATE TEST-OP returns and one or more test failure conditions
>>  are signaled.
>>  3. OPERATE TEST-OP returns and only test success conditions are
>>  signaled.
>>
>>>>   - the minimal requirement is a success / failure designator, the
>>>>   failed
>>>>     test names can be optional
>>>
>>>   - Additional requirement: the condition should support accessing both
>>>   a long and a short form report. In degenerate implementations, these
>>>   can, of course, be identical.
>>
>>  What would the long and short form reports look like?
>>
>>>>     some corner cases. Still, even if we expect test results being
>>>>   signalled
>>>>     multiple times during a test-op, it would be good to provide a
>>>>   wrapper
>>>>     which aggregates them into a single return value.
>>>>
>>>>           (common-test-results:collect (asdf:test-system "my-system"))
>>>
>>>   This would require that the test library provide an extensible
>>>   protocol for fusing together multiple test results.
>>
>>  It is simpler than that: take all of the conditions, add the number
>>  of tests executed, and append all of the test failure lists. No need
>>  for library-specific code.
>>
>>>   And note that the above suggestion will not work, because ASDF does
>>>   not ever *return* a value from operating. This has to do with the
>>>   way ASDF creates a plan and then executes it. The plan doesn't
>>>   support a notion of "return value," so the only way to get
>>>   information out of ASDF is through conditions.
>>
>>  What COMMON-TEST-RESULTS:COLLECT would do is handle and coalesce
>>  multiple conditions and re-signal a single condition.
>>
>>>   I agree -- I think `TRIVIAL-TEST-INTERFACE` might be a better first
>>>   step. I suppose the alternative rationale is that a test interface
>>>   that was *not* intended for incorporation into ASDF would be able to
>>>   just *return* things, instead of *signaling* them.
>>
>>  The point is, systems already define TEST-OP. I am trying to use
>>  that. The code that uses TEST-OP can do whatever it needs to, but the
>>  communication between that code and the test libraries has to be done
>>  by stack-based mechanisms like conditions or special variables.
>>
>>>   That is true, but it's also true that it would require special
>>>   condition-handling to fit test results into continuous integration
>>>   -- programmers would no longer be able to just use `quit-on-error`,
>>>   which is a very handy way to turn a lisp test into something that
>>>   works in Jenkins or, for that matter, any infrastructure based on
>>>   shell scripting.
>>
>>  Right now errors are not signaled on test failures in most
>>  definitions of TEST-OP I looked at, so this is not something that is
>>  currently going on. Neither is this something that would stop working
>>  for anyone that has the signal-error-on-failure flags set for their
>>  test library, or is throwing errors explicitly.
>>
>>>   I'd rather have to write code to handle errors when I *don't* want
>>>   them, than have test failure not be an error.
>>>
>>>   If I'm running interactively, it's not a bother to deal with this as
>>>   an error condition -- I can easily get out of the debugger. But
>>>   writing a lot of code to catch `TEST-FAILURE` conditions and
>>>   translate them into exit with non-zero status would be a pain.
>>
>>  Test libraries already have flags whether to signal errors on test
>>  failures or not. Having the condition be a sub-class of error would
>>  not only be annoying in the REPL, it would break whatever test
>>  automation code uses these flags, and it would change the behavior of
>>  TEST-OP, most of whose callers do not expect it to signal errors on
>>  test failures right now. This is a lot of breakage of thousands of
>>  existing systems, just to avoid doing the following in a few test
>>  automation scripts:
>>
>>  (handler-case
>>      (asdf:test-system "some-system")
>>    (asdf:test-op-test-failures (condition)
>>      (princ condition uiop:*stderr*)
>>      (uiop:quit 1)))
>>
>>>>   - slot for the failing asdf system could probably be avoided,
>>>>     the list failed test names could be enough, if the names are
>>>>   "fully qualified"
>>>>     i.e. include package or system name.
>>>
>>>   I don't think we can make any assumptions about the above -- there's
>>>   no rule about how a programmer can assign test names in a library
>>>   like FiveAM to packages.
>>
>>  FiveAM test names are symbols, so they already get printed with their
>>  package name in the implementation I did for FiveAM, without any
>>  extra work.
>>
>>>   I would note also that getting a new library into Quicklisp for this
>>>   is going to be a lot easier than getting a new ASDF into Quicklisp:
>>>   Xach has for years refused to update the ASDF version in Quicklisp,
>>>   and I don't see any reason to believe this will change.
>>
>>  As I mentioned before, I would like to avoid creating a whole library
>>  out of something that is a work-around to OPERATE not returning
>>  results.
>>
>>  Unfortunately, it seems there are both social and technical problems
>>  with updating ASDF. In particular I do not see a good mechanism for
>>  advertising the availability of this condition protocol to test
>>  libraries (there does not seem to be an established way of
>>  advertising new ASDF functionality other than the :ASDF3.3 :ASDF3.2
>>  etc. keywords in *FEATURES*).
>>
>>  As Anton pointed out, this necessitates the libraries signaling a
>>  condition for test success, which necessitates a function like
>>  COMMON-TEST-RESULTS:COLLECT. While writing an implementation of that
>>  function, I realized I would need to add continue restarts (the only
>>  way to handle a condition without affecting control flow).
>>
>>  Take all of this together, and it becomes apparent that avoiding ASDF
>>  to provide a more useful TEST-OP is, ironically, the way to go. It is
>>  simpler to drive communications down the stack by binding special
>>  variables to act as accumulators, than it is communicating up the
>>  stack with signals, handlers, and restarts. Putting this into a
>>  library means TEST-OP would still retain a use as a way to trigger
>>  test runs without knowing details about either the tests or the test
>>  library (the first half of this proposal), but it will unfortunately
>>  mean that ASDF will continue to have no say about what the effects of
>>  TEST-OP are (the second half of this proposal).
>>
>>  --
>>  Vladimir Sedach
>>  Software engineering services in Los Angeles https://oneofus.la