[asdf-devel] About regression tests and libraries

[Robert Goldman]

Nikodemus Siivola wrote:
> 2009/8/5 Robert Goldman <rpgoldman at sift.info>:
> 
>>> I would propose that as a matter of good style implementations of
>>> TEST-OP should signal an error is tests do not pass. It is the
>>> simplest way to make sure the result is not misinterpreted...
> 
>> I suppose, but typically a regression test suite should attempt to run
>> ALL of its tests, collecting a set of failing tests, and then returning
>> that information.  And demanding interactive attention ("I'm going to
>> throw you into the debugger if I fail") may also be undesirable in
>> practice.  I suppose handler-case would be enough for that, but I'm
>> still uncomfortable with the idea.
> 
> My rationale for suggesting this was that there are two mental modes
> for running tests: as a user ("Do I have version that seems to be
> working?") and as a developer ("What did I break now?").
> 
> I would expect TEST-SYSTEM to be used mostly by the end users of ASDF
> systems, whereas presumably developers run their own test-suites more
> directly -- I'm assuming that the typical TEST-OP definition would
> look like this:
> 
> (defmethod perform ((op test-op) (sys (eql (find-system :my-system))))
>   (foo-unit:run-tests :my-system-tests :on-failure :error))
> 
> and developers of MY-SYSTEM would normally use RUN-TESTS directly,
> taking advantage of whatever reporting and debugging features the
> test-suite provides.

I see.  However, I think it would be a mistake to force /anyone/ to
encounter an immediate error on running the test-op as in the snippet above.

I work on a couple of large systems where, in fact, the developers run
their own test suites through ASDF, because many of them are not
familiar with the test tool (maybe they are working on the java side).
They don't want to encounter the lisp debugger.  For that matter, we
actually package up the results of the tests into an XML file that can
be slurped by junit, and assembled into an overall test result.  In
fact, in some cases there isn't even someone available to interact with
the build process --- it's run at a high level by scripts.

For relatively small libraries, doing one thing, the "error immediately"
behavior might be appropriate.  But for large, complex artifacts (e.g.,
a web site), this behavior is not appropriate.  It would be appropriate
to /offer/ error-immediately behavior, but not to offer /only/ that
behavior.

Best,
Robert

Best,
R