[Ecls-list] Project for newbies: enlarged testing suite

[Juan Jose Garcia-Ripoll]

Dear lispers,

this is not only for new ECL users looking for a place to contribute,
but also for developers of libraries that want them to be tested with
ECL.

We have reached a point where the ANSI compatibility test suite is not
enough. Undocumented or undefined behavior that changes between
releases may cause some libraries to break in unexpected ways -- we
have seen several messages in the mailing list about them.

So my idea is to add new directories to ecl-test for automating the
downloading and testing of other libraries on a daily, weekly or
monthly basis, depending on how things move on.

As a guidance you may have a look at how ecl-test/ansi is implemented:
Makefile checks for the existence of that directory and otherwise
checks out with cvs a fresh new copy of the test suite. Tests are then
loaded and run, and the output is logged into output/ecl/ansi.log. We
could have similar directories for cffi, babel, flexi-streams, and
other packages. It should not be that difficult.

However, given that this is to be run on at least five different
platforms, a few restrictions apply:
1) Source code can only be retrieved using cvs, ftp or wget/curl
2) Running the tests should not take more than half an hour on a
computer with few resources: 512Mb, P4 (this is a typical
configuration for my virtual machines)
3) The test suite should be comprehensive enough.
4) If possible, libraries should not have other dependencies.
5) One should only test stable releases.

If neither of these points is satisfied, then I do not see the point
of adding those directories. In particular, points 1 and 3 are
crucial, and 3 most so: why running a test suite when it is not able
to produce a meaningful answer? Point 4 is not so relevant. If it
cannot be satisfied by all libraries then the design might become a
bit more complex.

Juanjo

-- 
Instituto de Física Fundamental
CSIC, Serrano, 113, Madrid 28040 (Spain)
http://juanjose.garciaripoll.googlepages.com