[PATCH] Add a TEST-OP-TEST-FAILURE condition for test libraries to sub-class

[Vladimir Sedach]

Stelian, I think there is something wrong with the reply you sent.
The Subject header is mangled in the Mailman archives where I spotted
it. I never received this email:

https://mailman.common-lisp.net/pipermail/asdf-devel/2019-September/006356.html

> I'd like to have something that gathers enough information to allow
> building a suitable integration with continuous integration systems
> (there are already a few ad-hoc ones around).

What exactly would that look like?

> For example, a common taxonomy of test status is: SUCEEDED, FAILED,
> BROKEN, SKIPPED, NOTSTARTED.

I do not see what this has to do with my proposal. FiveAM does not
even distinguish failed from broken tests (I do not know if any
Common Lisp test libraries do - a broken test would generally be a
file compilation error). NOTSTARTED means having asynchronous updates
go from the test library to some external system; that is completely
out of scope of ASDF.

> Whatever support ends up in ASDF should be sufficient to allow
> outputting common rest report formats, such as JUnit, TAP, etc...

You can easily do this with my proposal by having the condition
reporter output TAP or JUnit XML. Formatting test results is the
responsibility of the test library.

There are test libraries for test libraries to help with this
formatting:
https://github.com/e-user/testbild
https://github.com/brobinson9999/cl-tap-producerX
in case you want to add JUnit XML output to FiveAM.

> otherwise we risk ending up with something that cannot be improved
> because users start relying on it and we don't want to break
> backwards compatibility.

This is not a problem because conditions can be sub-classed.

--
Vladimir Sedach
Software engineering services in Los Angeles https://oneofus.la