[asdf-devel] About regression tests and libraries
rpgoldman at sift.info
Wed Aug 5 19:39:34 UTC 2009
Gary King wrote:
>> Part of the problem with test-op is that the desired behavior has not
>> been specified by the ASDF community. Because of the nature of ASDF, it
>> is impossible for
>> (asdf:test-system <system>)
>> to return a value indicating whether or not <system> has passed its
> FWIW, I recently changed operate-on-system (and therefore
> asdf:test-system, asdf:load-system, etc.) to return the operation
> instance that they processed.
> If ASDF also added and documented (shocking!) a few canonical slots in
> the test-op class then these could be used to transmit information. As a
> first pass, I'd suggest
> test-properties - a property list of test system specific stuff
> test-result - a keyword from the list :success, :failures,
> :errors, errors-and-failures
>> Some time ago, I proposed that ASDF provide a stream argument to the
>> test-op, providing a place into which the testing tool could dump its
>> test report for human inspection. I can't say that this suggestion
>> received universal approbation. Or disapprobation. It merely received
>> near-universal lack of interest! ;-)
> Sigh. I think this is a good idea; if we added a test-stream slot to the
> test-op, then test systems could find it there.
> A tricky part is that test systems may not want to rely on the existence
> of ASDF while running. So they'll need some sort of adaptor that adds
> stuff to the ASDF operation if it exists. Would it make sense to define
> and export *current-operation* to make this easier?
You'd still need the asdf stuff to get at this. The alternative would
be to define a special variable that gives direct access to the stream,
which is not an asdf-dependent entity.
The problem with that approach is that you're pretty much doomed to crap
up the CL-USER namespace.
>> Related to the question of what the test-op should provide to its
>> invoker is the question of how dependencies should be propagated.
> I think that the default is that test-op depends on load-op.
Right, but what about testing system <foo> --- do we test <foo>'s
depended on systems? My guess is that the correct answer to this is "it
depends." If you have a standard set of libraries, the answer will
typically be "no." OTOH, if you have a very large application (e.g., a
web site with many components providing different features), you may
well want to test the subsystems that are part of the top-level system.
> other thoughts?
We have found in practice that we end up creating a test-<foo> system
definition to complement our <foo> system definitions. The reason for
this is that the existing ASDF structures have only components that are
unconditional. For the test op we typically want to load new files. A
second item is that the test system typically :depends-on the test
This suggests a question: is there a nice way to specify additional
files that need to be loaded ONLY for the test-op? I can think of some
ways to achieve this with the existing ASDF framework, but they would be
cumbersome and non-standard. Would it be possible to simply add a
:test-components argument to defsystem?
Note that the use of additional <test>-foo systems also helps us tailor
Another note --- I found in an earlier attempt to tweak the test-op that
it worked poorly to subclass operations (as opposed to components).
IIRC, this was because of the way dependent operation instances are
built. Alas, I didn't take good notes when I did this, and have
garbage-collected much of what I learned.
More information about the asdf-devel