[asdf-devel] ASDF test-op question

[Robert Goldman]

Faré wrote:
> Maybe ASDF is the wrong place to try to standardize testing infrastructure?

This is the conclusion I have reached, as well.  I was hoping that some
very weak standard could be arrived at that would make the test-op more
generally useful to people installing systems, so that they could simply
run the test-op and easily tell whether or not the test operation was
successful.

However, it may be that it's just a combination of features of our test
framework and the way we have built our systems that makes it difficult
for us (the original problem --- we find that we get compilation and
loading output mixed together with the test output), so this is not a
place where the ASDF community can readily get consensus.

So please consider the suggest dropped.

I may try to come back to the list with a proposal for modifying the
MAKE-SUB-OPERATION function so that it makes specializing operations
(including the test-op) easier.  That would be a more general proposal
that might make many things easier for ASDF system specifiers.

> 
> I mean, maybe instead the authors of various test infrastructures
> should have a common list where they discuss interoperability,
> reporting, and a declarative way of specifying dependencies between
> test suites, between files and test suites, etc.?
> 
> Since for instance XCVB doesn't have testing support yet, but testing
> support is amongst our eventual goals, I would be eager to have
> feedback on how a new build system could "do things right" in this
> regard -- or perhaps avoid doing anything, and just plugging into
> something else.

Let me suggest a few desiderata:

1.  Make it possible for operation performance to return values.  Think
about what the types of these values should be while designing this
facility.

2.  Provide a standard means for specifying how sub-operation values
should roll up to the values of their parents.

3.  If you are taking the plan then act design framework, think about
how to keep the (direct) child operations of a parent in some kind of
procedure call envelope (dynamic scope extent).

4.  Deal with the difference between how different operations should
handle values and failure.  E.g., a test operation should simply
accumulate failures.  A build operation should almost always fail at the
first point of failure.

Hope that helps,

r