Inclusion into cl-test-grid

[Mark Evenson]

On 2016/4/29 04:31, Ian Tegebo wrote:
> In reading through the history and issues of ASDF [1], I came across
> complaints that various LISP implementations were broken in ways that
> complicated its development.  Thinking that there must be a conformance
> test suite out there, I found ansi-test.
[…]
> Do you think it makes sense to include ansi-test into cl-test-grid?

As an ABCL "implementor"--more a maintainer of the work of others at
this point--I have always found ANSI-TEST to be helpful and would love
to see its output included in CL-TEST-GRID.

But I get the sense that my enthusiasm for ANSI-TEST is not necessarily
shared by other contributors to implementations.   When the current ABCL
maintainers "took over" in 2006, we committed a fair amount of energy to
understanding and fixing why we were failing various parts of ANSI-TEST.
 For me personally, who was more-or-less learning Common Lisp at the
same time, studying the ANSI-TEST code was an invaluable aid to
understanding the finer points of the specification.

But as I expanded my knowledge, I wondered why other open source CL
implementations didn't seem to devote equal energy to fixing problems
flagged by ANSI-TEST.  I remember asking an SBCL contributor--whose name
I have since forgotten--at a "Lisp Gathering" organized by Andreas Fuchs
in Vienna in 2008 (?) about why the SBCL community, normally a
hyper-competitive bunch, didn't seem to care that ABCL failed fewer
parts of ANSI-TEST than SBCL.  The essential gist of his reply was along
the lines that after a certain point of conformance, chasing down all
the individual "little" points wasn't as effective as working on core
parts of the implementation.  My reply was along the lines that, sure,
an individual test doesn't tell you much about the quality of an
implementation, but as one adds more and more "dumb" tests, at some
point the test suite crosses a threshold where it appears quite "smart".
 At the time, ANSI-TEST was a pretty static entity, seeing the
occasional bugfix.

While it is not at all clear that my little anecdote accurately
describes a widely held contemporary opinion of the ANSI-TEST suite
relevance, I think that the nature of software development has changed
enough over the past decade to the point that ANSI-TEST could become
quite useful both for implementors and ASDF system creators.  In
contrast with a decade ago the effort needed to add a new test or to
refer to a specific result of a test via tools like CL-TEST-GRID has
been dramatically reduced, increasing the chances of collaboration.

ANSI-TEST could become a communication mechanism for ASDF package
developers to address the implementation community:  when an actual
conformance issue arises a specific facet of ANSI-TEST could be
referenced (or first contributed if it didn't exist) in the process of
working with implementations towards resolution.


-- 
"A screaming comes across the sky.  It has happened before, but there
is nothing to compare to it now."