[asdf-devel] ASDF test-op question

[Tobias C. Rittweiler]

Juan Jose Garcia-Ripoll writes:

> On Mon, Oct 19, 2009 at 12:00 AM, Robert Goldman wrote:

[for the record; the topmost paragraph came from me:]

> > > In particular because it's my impression that the problems have been
> > > over-stated.
>
> I think so. And this is indicated by the following paragraphs
>
> > 1.  Not all regression test frameworks are functional, returning values.
> >  Some write reports instead.
>
> That does not mean anything. ASDF may impose one behavior w.r.t to the
> output and the package maintainer may decide to output T saying "hey,
> there might be errors, but I do not know." Later on other software and
> test suites will catch up. With RT, for instance it is trivial to grab
> the number of failed tests.
>
> > 2.  One needs to come up with a means of combining operation results
> > that takes into account the structure of the plan that traverse produces
> > and that operate then executes.  If in order to test-op X, I must
> > test-op A and B, how do I combine together the test results from A, B,
> > and X into a top-level operation result.  Perhaps OPERATION-ANCESTOR can
> > be pressed into service.
>
> You are just imposing too much complexity. If I want to test package
> Cl-UNICODE, I do nont want to test FLEXI-STREAMS or U-SOCKETS. Tests
> should be atomic and not generate a tree of actions like ASDF does not
> for everything.

As an interim solution, there can be some merge operation which, for the
moment, just bails out if multiple results really have to be combined.


> > 3.  Is there some way to do this such that the regression-testing
> > framework need not be made aware of ASDF?
>
> No, there is none. Regression tests are libraries which are used by
> people using ASDF. Eventually they may provide their own operations,
> extending what ASDF provides, but currently it is better to have some
> bare minimum from ASDF and let each package maintainer decide how they
> compute the success of their test suite.

Right. 

In particular: between ASDF and the regression test framework always
stands a _human being_, the maintainer of the software package. It's his
duty to interconnect his choice of third-party test framework with ASDF
-- that is if he really wants to. If he doesn't, well, he just
shouldn't.

Existing practise does luckily not matter much in this case. It's about
meta-information that nobody can depend on at the moment because it does
not exist. And if it exists, it'll matter to people who care and will do
the right thing. (Which may include fixing other people's code; but see
that it's in _their_ own interests to do so, so they will.)



> Blocking this development just because there are 5 test suites and you
> do not know how to combine them with ASDF is really absurd. ASDF's
> specifications can not depend on what your company or other people's
> companies are setting up for their workflow.

Blocking is way too strong a word. After all, both of us want to
piggy-back onto work of others. We can just hope to make our case clear
to those who contribute code---or contribute ourselves.

  -T.