[asdf-devel] ASDF test-op question

Robert Goldman rpgoldman at sift.info
Sun Oct 18 22:00:49 UTC 2009


Tobias C. Rittweiler wrote:
> Robert Goldman writes:
> 
>> Faré wrote:
>>
>>> Maybe ASDF is the wrong place to try to standardize testing infrastructure?
>> This is the conclusion I have reached, as well.  I was hoping that some
>> very weak standard could be arrived at that would make the test-op more
>> generally useful to people installing systems, so that they could simply
>> run the test-op and easily tell whether or not the test operation was
>> successful.
>>
>> However, it may be that it's just a combination of features of our test
>> framework and the way we have built our systems that makes it difficult
>> for us (the original problem --- we find that we get compilation and
>> loading output mixed together with the test output), so this is not a
>> place where the ASDF community can readily get consensus.
>>
>> So please consider the suggest dropped.
> 
> Please excuse me that I cannot grow at peace with this conclusion.
> 
> The missed utility is far too great to let this rest:
> 
> Juanjo's automatic regression testing work to check for regressions in
> ECL is dead on, and I'm sure other implementators would be interested in
> such infrastructure as well.
> 
> The same is true for D Herring and his LibCL project which looks very
> promising, but is also a candidate for automatic regression testing. I
> think that Daniel is the right guy for the job and I hope he'll have the
> necessary perseverance for what he has started.
>   
> 
> I have to excuse myself again because I'm not familiar with ASDF in any
> great detail; on the other hand, ASDF -- unlike most other Lisp project
> -- consists of over half a dozen of people who do know it pretty well,
> so I cannot believe that we can not come up with some solution.
> 
> In particular because it's my impression that the problems have been
> over-stated. 

I think there are some problems which are real, but perhaps not insuperable:

1.  Not all regression test frameworks are functional, returning values.
 Some write reports instead.

2.  One needs to come up with a means of combining operation results
that takes into account the structure of the plan that traverse produces
and that operate then executes.  If in order to test-op X, I must
test-op A and B, how do I combine together the test results from A, B,
and X into a top-level operation result.  Perhaps OPERATION-ANCESTOR can
be pressed into service.

3.  Is there some way to do this such that the regression-testing
framework need not be made aware of ASDF?  Ideally, ASDF and the
regression-testing framework would be defined independently, and some
additional combination code would be defined that would link the two
together, and that would be loaded downstream.  I'd be reluctant to see
a solution that demanded our regression-testing frameworks become
dependent on ASDF.  I believe that this problem could be easily
circumvented by defining new systems like ASDF-FIVEAM, ASDF-NST,
ASDF-STEFIL.  At my company, we have already been working on ASDF-NST,
but it has not reached a fully satisfactory state.

best,
r




More information about the asdf-devel mailing list