[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Vladimir Sedach]

Robert Goldman <rpgoldman at sift.info> writes:
> Don't a lot of these CI systems just rely on the exit status of a
> build step?  I typically just wrap a trivial script around running
> the test-op from fiveam-asdf, and make lisp exit with a non-zero
> status when it encounters an error.  That's been sufficient for me,
> but then I tend to just look at success or failure, and read the
> console output on failures.  I don't do anything fancy like generate
> XML...

That is what all of the CI scripts I have looked at in various
projects do (there may be others I have not seen; please post them if
you know of any). FiveAM itself provides a typical example:
https://github.com/sionescu/fiveam/blob/master/.travis.yml#L40

(uiop:quit (if (some (lambda (x) (typep x '5am::test-failure))
                     (5am:run :it.bese.fiveam))
               1 0))

With my proposal becomes:

(handler-case (asdf:test-system "any-system")
  (asdf:test-op-test-failure (condition)
    (princ condition uiop:*stderr*)
    (uiop:quit 1)))

And now the lowest common denominator case for CI systems is
automatic.

To expand on my previous statement that this proposal will not have
backward-compatibility problems for CI systems because signals can be
sub-classed, here is what that could look like:

Someone writes a GitLab continuous integration library, call it
cl-gitlab-ci, and wants to add support to test libraries.
cl-gitlab-ci defines a condition:

(define-condition cl-gitlab-ci-test-failure ()
  ((test-failure-info …)
   (test-skip-info …))
  …)

Libraries like FiveAM then add this condition as a subclass and
fill in the necessary slots:

(define-condition test-spec-failure
    (cl-gitlab-ci-test-failure asdf:test-op-test-failure)
  …)

This does not break backward compatibility.

Same thing for a Travis integration library.

When someone decides to write a CI meta-library that abstracts over
cl-gitlab-ci, cl-travis-ci, etc. (which as a basis will have what
Stelian is asking for), they can define a condition class,
generic-ci-test-failure, then send patches to the CI libraries to
handle that condition, and send patches to the test libraries to
subclass generic-ci-test-failure and fill out the slots (this is just
like adding support for another CI library to the test library).
Again, this does not break backward compatibility.

Currently popular CI programs include Buildbot, GitLab, Jenkins,
Travis CI (essentially limited to GitHub). If someone has a proposal
for a protocol that will cover the needed features for all of these
systems, please post it.

Right now the only library for any of these programs that I know of
is cl-travis. The hypothetical CI meta-library will ideally be built
from experience with several (as of now non-existing) libraries for
the various CI programs. Not only do I think that this is outside the
scope of ASDF, I also suspect that designing a protocol for CI
integration now, without the needed experience from actual libraries
for various CI systems, is what is going to result in backward
compatibility problems.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la