[asdf-devel] ASDF test-op question

[Robert Goldman]

Tobias C. Rittweiler wrote:
> Robert Goldman writes:
> 
>> Juan Jose Garcia-Ripoll wrote:
>>  
>>> You are just imposing too much complexity. If I want to test package
>>> Cl-UNICODE, I do nont want to test FLEXI-STREAMS or U-SOCKETS. Tests
>>> should be atomic and not generate a tree of actions like ASDF does not
>>> for everything.
>> Please see earlier discussion about this topic.
>>
>> The point I made there is that systems may have COMPONENT subsystems
>> such that you want to test the entire system together.
>>
>> Consider, e.g., a DB library with multiple backends, each described in a
>> separate system.  In order to do the test-op on the DB library you want
>> to do the test on all active backends.  Similarly, McCLIM has multiple
>> graphics display backends, not all of which are loaded at a given time.
>>    I work actively on three (or four, depending on how you count) large
>> CL-based applications.  Each one of them is made up of multiple ASDF
>> subsystems.
> 
> Yes, that is a valid use case. No one disputes that.
> 
> What if we start with the simple case of only carring one bit of
> meta-information (successp), merging is nothing more than ORing together
> the results.
> 
> This can be done in a way to satiesfy current needs without restricting
> future improvement (which then can be based on actual experience in the
> field.)

Actually, I don't think that the hard part is deciding how to merge
things together.  I agree with you that the merge operation can be made
a generic function, solving this problem.

The greater problem, I believe, is determining /what/ operations to
merge together.

Consider the case of a system, X, on which I invoke the TEST-OP.  Doing
so can create a plan (through TRAVERSE) which contains an arbitrary
number of operations necessary to perform TEST-OP on X, including
potentially many LOAD-OPs and COMPILE-OPs.  How many of these operations
need to be inspected for success or failure?

One might think "all and only the test-op"s, but I can say from sad
experience that this is not so --- a large program on which I am now
working came to a halt for several hours because it did TEST-OP, and the
TEST-OP didn't yield a report because some subsystem failed to build.
We spent quite a while looking for the test-op's output which we, of
course, did not find.  So certainly, one should make sure that the
test-op handle this properly.  For interactive use, this is not a
problem --- if the load fails, an exception will be raised.  But for
NON-interactive use (in our case on a build-and-test server) it's not so
obvious....  A build-and-test server is very handy on medium to large
size projects to verify that someone committing a patch is not breaking
the build.

As I said, perhaps the ancestor links among operations may give a
solution to this problem:  I simply haven't had time to investigate
thoroughly.

I will try to get some time in the near future to discuss this matter
with my colleague, John Maraist, who has done the most recent work on
integrating the NST unit test library and ASDF.  I will report back
anything useful I learn.

best,
Robert