[asdf-devel] ASDF test-op question

[Robert Goldman]

Daniel Herring wrote:
> On Mon, 5 Oct 2009, Robert Goldman wrote:
>> Gary King wrote:
>>> Hi Robert,
>>>
>>>> I don't believe that this is a general solution, for two reasons:
>>> I agree that it isn't a general solution especially since there is no
>>> interface/API for clients to do anything with an ASDF operation! It
>>> might, however, be a small step in the right direction.
>> An alternative solution would be to provide a :stream or :filename init
>> argument for the test-op operation class and bind a dynamic variable
>> around every perform, making the stream or filename available for
>> writing....
> 
> Why serialize the data?  Could we design a structured API to be used by 
> other tools?
> 
> What if each test logged messages to asdf:*test-stream* and finished by 
> calling
> (asdf:test-result test-description-string result-keyword)
> 
> DejaGnu has a good description of test results; the keywords from good to 
> bad could be :xpass, :xfail, :untested, :unresolved, :unsupported, :pass, 
> and :fail.
> http://www.gnu.org/software/dejagnu/manual/x47.html
> 
> Then asdf:test-op could return (values worst-result results-list).
> 
> Of course ASDF doesn't need to reinvent testing; there are plenty of 
> existing frameworks to choose from.
> http://www.cliki.net/test_framework
> 

What I am after is an ASDF test-op that will adapt as well as possible
to the widest possible set of testing frameworks.  I agree that we
should not attempt to reinvent testing.

That is why I have been suggesting that we provide a test operation that
binds a stream --- because most of the test frameworks I have worked
with provide a test report, rather than returning results.

I don't believe that having the test-op return a result will work, given
the current ASDF execution model.  If we are testing system X that has A
and B dependencies, then what will happen will be something like

.... testing A's components... test-op A....testing B's components ...
test-op B .... test other X components ... test-op X

It's not obvious to me how we take the results from test-op A and
test-op B and roll them up into test-op X.  Nor do I see how, in
general, a test-op done in LIFT for A, FiveAM for B and NST for X, can
be made to communicate with each other.

It's not ideal, but I think a separate stream may be about as good as we
can do.

Best,
r