[asdf-devel] About regression tests and libraries

[Robert Goldman]

Gary King wrote:
>> Part of the problem with test-op is that the desired behavior has not
>> been specified by the ASDF community.  Because of the nature of ASDF, it
>> is impossible for
>>
>> (asdf:test-system <system>)
>>
>> to return a value indicating whether or not <system> has passed its
>> tests.
> 
> FWIW, I recently changed operate-on-system (and therefore
> asdf:test-system, asdf:load-system, etc.) to return the operation
> instance that they processed.
> 
> If ASDF also added and documented (shocking!) a few canonical slots in
> the test-op class then these could be used to transmit information. As a
> first pass, I'd suggest
> 
>     test-properties - a property list of test system specific stuff
>     test-result         - a keyword from the list :success, :failures,
> :errors, errors-and-failures
> 
>> Some time ago, I proposed that ASDF provide a stream argument to the
>> test-op, providing a place into which the testing tool could dump its
>> test report for human inspection.  I can't say that this suggestion
>> received universal approbation.  Or disapprobation.  It merely received
>> near-universal lack of interest! ;-)
> 
> Sigh. I think this is a good idea; if we added a test-stream slot to the
> test-op, then test systems could find it there.
> 
> A tricky part is that test systems may not want to rely on the existence
> of ASDF while running. So they'll need some sort of adaptor that adds
> stuff to the ASDF operation if it exists. Would it make sense to define
> and export *current-operation* to make this easier?

You'd still need the asdf stuff to get at this.  The alternative would
be to define a special variable that gives direct access to the stream,
which is not an asdf-dependent entity.

The problem with that approach is that you're pretty much doomed to crap
up the CL-USER namespace.
> 
>> Related to the question of what the test-op should provide to its
>> invoker is the question of how dependencies should be propagated.
> 
> 
> I think that the default is that test-op depends on load-op.

Right, but what about testing system <foo> --- do we test <foo>'s
depended on systems?  My guess is that the correct answer to this is "it
depends."  If you have a standard set of libraries, the answer will
typically be "no."  OTOH, if you have a very large application (e.g., a
web site with many components providing different features), you may
well want to test the subsystems that are part of the top-level system.
> 
> other thoughts?

We have found in practice that we end up creating a test-<foo> system
definition to complement our <foo> system definitions.  The reason for
this is that the existing ASDF structures have only components that are
unconditional.  For the test op we typically want to load new files.  A
second item is that the test system typically :depends-on the test
framework library.

This suggests a question:  is there a nice way to specify additional
files that need to be loaded ONLY for the test-op?  I can think of some
ways to achieve this with the existing ASDF framework, but they would be
cumbersome and non-standard.  Would it be possible to simply add a
:test-components argument to defsystem?

Note that the use of additional <test>-foo systems also helps us tailor
subcomponent testing.

Another note --- I found in an earlier attempt to tweak the test-op that
it worked poorly to subclass operations (as opposed to components).
IIRC, this was because of the way dependent operation instances are
built.  Alas, I didn't take good notes when I did this, and have
garbage-collected much of what I learned.

r