[asdf-devel] ASDF test-op question

[Robert Goldman]

Daniel Herring wrote:
> On Mon, 5 Oct 2009, Robert Goldman wrote:
>> Daniel Herring wrote:
>>> On Mon, 5 Oct 2009, Robert Goldman wrote:
>>>> Gary King wrote:
>>>>> Hi Robert,
>>>>>
>>>>>> I don't believe that this is a general solution, for two reasons:
>>>>> I agree that it isn't a general solution especially since there is no
>>>>> interface/API for clients to do anything with an ASDF operation! It
>>>>> might, however, be a small step in the right direction.
>>>> An alternative solution would be to provide a :stream or :filename init
>>>> argument for the test-op operation class and bind a dynamic variable
>>>> around every perform, making the stream or filename available for
>>>> writing....
>>> Why serialize the data?  Could we design a structured API to be used by
>>> other tools?
> ...
>>> Of course ASDF doesn't need to reinvent testing; there are plenty of
>>> existing frameworks to choose from.
>>> http://www.cliki.net/test_framework
>>>
>> What I am after is an ASDF test-op that will adapt as well as possible
>> to the widest possible set of testing frameworks.  I agree that we
>> should not attempt to reinvent testing.
>>
>> That is why I have been suggesting that we provide a test operation that
>> binds a stream --- because most of the test frameworks I have worked
>> with provide a test report, rather than returning results.
>>
>> I don't believe that having the test-op return a result will work, given
>> the current ASDF execution model.
> ...
> 
> Given that, how about deprecating/removing test-op?  Then people can 
> continue to simply load sysname-test.asd and run tests however they want.
> 
> I see value in creating a widely accepted testing framework.  I don't see 
> the benefit in simply binding a stream variable.
> 
Let me clarify.  I /do/ see the value of providing a test-op.

We have our own testing framework at my company, NST (see presentation
hosted at tc-lispers.org for more information).  We have provided a
back-end for it that hooks into ASDF, so that people can do

(test-system <foo>)

and our NST tests will run and a report will be printed.

This isn't fully ideal, though, since the results of tests will be
mingled in the console output together with compilation results, which
makes finding the test outputs a little more difficult than it should
be.  So I think providing a standardized output stream would be helpful,
and probably generally helpful, since no matter what your unit-testing
library, if it writes a report to the console, and is invoked from ASDF,
it's likely to suffer from the same problem of output interleaving.

I don't think that ASDF needs to provide a test framework; all it needs
to do is to provide hooks for invoking existing test framework(s).

Best,
r