Testing ASDF with ABCL under Windows (was Re: [asdf-devel] Releasing asdf 3.1.1 ?)

[Anton Vodonosov]

31.12.2013, 10:31, "Faré" <fahree at gmail.com>:
> Dear Anton,
>
> On Tue, Dec 31, 2013 at 1:25 AM, Anton Vodonosov <avodonosov at yandex.ru> wrote:
>
>>  I have started cl-test-grid tests with the latest ASDF (git revision 38337a5)
>>  After SBCL and CCL tests complete we can run tests on the previous ASDF
>>  and compare results.
>
> Thanks a whole lot for cl-test-grid, it has been extremely valuable in the past,
> especially when initially releasing ASDF 3.
>

You are welcome.

>>  The question: what is the previous stable and tested ASDF version?
>
> I suppose the real question is: which version did YOU last use?

Yes, but I thought you may have some additional information
form non cl-test-grid testing.

When we tested with cl-test-grid, 
ABCL, CMUCL, CCL, CLISP, ECL, SBCL were tested on 2.31.8
http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-21.html

The latest cl-test-grid cycle was for ASDF 2.32.35, it was tested
on CCL, CLISP, ECL, SBCL on Linux.
http://common-lisp.net/project/cl-test-grid/asdf/asdf-diff-23.html

So, as you see we ended up testing last fixes only on some lisps,
not all lisps. Probably that's because it was stabilizing period and
the fixes were for concrete bugs.

Still, we didn't make a firm "rollback" point - an ASDF version where we
ensured we have no regressions on all lisps. For now I suggest to consider
2.32.35 as more or less reliable previous comparison point, where
all regressions were proved to be not ASDF bugs, but bugs in libraries,
and were reported to the library maintainers.

I hope in this ASDF release, we will test the released version
on maximum number of lisps, so that in future we will have reliable
comparison point.

> But then, you'd have to check with both old and new Quicklisp.

I am going to run tests of both previous stable ASDF version and the
current HEAD on the same Quicklisp - 2013-12-13.

> I suppose that you should compare with whatever comes with the implementation.

OK, we can do this too - we will compare result for the latest ASDF
with the results for the "main", unpatched quicklisp, which uses ASDF
coming with the implementation.

It also would be nice to compare with some concrete previous stable
ASDF version, as described above, to be sure we cover full ASDF evolution
on all lisps.

> Therefore, for SBCL, that would be 3.0.2, and for CCL, 3.0.3.
> If you need the same in both cases, make it ASDF 3.0.3.

I think the further back in time we go, the better, for full coverage.
I suggest compare the HEAD with 2.32.35. Then we can also test with some
intermediate version, like 3.0.3, to ensure we not only don't have regressions
since 2.32.35, but also don't lose improvements introduced in 3.0.3 (if any).

Best regards,
- Anton