Inclusion into cl-test-grid

[Raymond Toy]

>>>>> "Mark" == Mark Evenson <evenson at panix.com> writes:

    Mark> On 2016/4/29 04:31, Ian Tegebo wrote:
    >> In reading through the history and issues of ASDF [1], I came across
    >> complaints that various LISP implementations were broken in ways that
    >> complicated its development.  Thinking that there must be a conformance
    >> test suite out there, I found ansi-test.
    Mark> […]
    >> Do you think it makes sense to include ansi-test into cl-test-grid?

    Mark> As an ABCL "implementor"--more a maintainer of the work of others at
    Mark> this point--I have always found ANSI-TEST to be helpful and would love
    Mark> to see its output included in CL-TEST-GRID.

    Mark> But I get the sense that my enthusiasm for ANSI-TEST is not necessarily
    Mark> shared by other contributors to implementations.   When the current ABCL
    Mark> maintainers "took over" in 2006, we committed a fair amount of energy to
    Mark> understanding and fixing why we were failing various parts of ANSI-TEST.
    Mark> For me personally, who was more-or-less learning Common Lisp at the
    Mark> same time, studying the ANSI-TEST code was an invaluable aid to
    Mark> understanding the finer points of the specification.

    Mark> But as I expanded my knowledge, I wondered why other open source CL
    Mark> implementations didn't seem to devote equal energy to fixing problems
    Mark> flagged by ANSI-TEST.  I remember asking an SBCL contributor--whose name
    Mark> I have since forgotten--at a "Lisp Gathering" organized by Andreas Fuchs
    Mark> in Vienna in 2008 (?) about why the SBCL community, normally a
    Mark> hyper-competitive bunch, didn't seem to care that ABCL failed fewer
    Mark> parts of ANSI-TEST than SBCL.  The essential gist of his reply was along
    Mark> the lines that after a certain point of conformance, chasing down all
    Mark> the individual "little" points wasn't as effective as working on core
    Mark> parts of the implementation.  My reply was along the lines that, sure,
    Mark> an individual test doesn't tell you much about the quality of an
    Mark> implementation, but as one adds more and more "dumb" tests, at some
    Mark> point the test suite crosses a threshold where it appears quite "smart".
    Mark> At the time, ANSI-TEST was a pretty static entity, seeing the
    Mark> occasional bugfix.

As a lisp maintainer, I do care about the tests that fail.
Unfortunately, some of the failures are really hard to fix, so they
don't get fixed.  Some of the tests, while correct, I refuse to do
anything about, like arrays of type nil are strings (or something like
that).

When I do run ansi-tests on the rare occassion, it's mostly now as a
regression to make sure I didn't do anything totally stupid.  (Mostly
because I haven't set up some kind of continuous integration scheme.)

That's a rather sad state, but there it is.

--
Ray