[asdf-devel] About regression tests and libraries

[Gary King]

> Part of the problem with test-op is that the desired behavior has not
> been specified by the ASDF community.  Because of the nature of  
> ASDF, it
> is impossible for
>
> (asdf:test-system <system>)
>
> to return a value indicating whether or not <system> has passed its  
> tests.

FWIW, I recently changed operate-on-system (and therefore asdf:test- 
system, asdf:load-system, etc.) to return the operation instance that  
they processed.

If ASDF also added and documented (shocking!) a few canonical slots in  
the test-op class then these could be used to transmit information. As  
a first pass, I'd suggest

     test-properties - a property list of test system specific stuff
     test-result         - a keyword from the  
list :success, :failures, :errors, errors-and-failures

> Some time ago, I proposed that ASDF provide a stream argument to the
> test-op, providing a place into which the testing tool could dump its
> test report for human inspection.  I can't say that this suggestion
> received universal approbation.  Or disapprobation.  It merely  
> received
> near-universal lack of interest! ;-)

Sigh. I think this is a good idea; if we added a test-stream slot to  
the test-op, then test systems could find it there.

A tricky part is that test systems may not want to rely on the  
existence of ASDF while running. So they'll need some sort of adaptor  
that adds stuff to the ASDF operation if it exists. Would it make  
sense to define and export *current-operation* to make this easier?

> Related to the question of what the test-op should provide to its
> invoker is the question of how dependencies should be propagated.


I think that the default is that test-op depends on load-op.

other thoughts?
--
Gary Warren King, metabang.com
Cell: (413) 559 8738
Fax: (206) 338-4052
gwkkwg on Skype * garethsan on AIM * gwking on twitter