[asdf-devel] About regression tests and libraries

[Nikodemus Siivola]

2009/8/5 Robert Goldman <rpgoldman at sift.info>:

>> I would propose that as a matter of good style implementations of
>> TEST-OP should signal an error is tests do not pass. It is the
>> simplest way to make sure the result is not misinterpreted...

> I suppose, but typically a regression test suite should attempt to run
> ALL of its tests, collecting a set of failing tests, and then returning
> that information.  And demanding interactive attention ("I'm going to
> throw you into the debugger if I fail") may also be undesirable in
> practice.  I suppose handler-case would be enough for that, but I'm
> still uncomfortable with the idea.

My rationale for suggesting this was that there are two mental modes
for running tests: as a user ("Do I have version that seems to be
working?") and as a developer ("What did I break now?").

I would expect TEST-SYSTEM to be used mostly by the end users of ASDF
systems, whereas presumably developers run their own test-suites more
directly -- I'm assuming that the typical TEST-OP definition would
look like this:

(defmethod perform ((op test-op) (sys (eql (find-system :my-system))))
  (foo-unit:run-tests :my-system-tests :on-failure :error))

and developers of MY-SYSTEM would normally use RUN-TESTS directly,
taking advantage of whatever reporting and debugging features the
test-suite provides.

Cheers,

 -- Nikodemus