Pointing out ABCL "extensions" to ANSI-TEST

[Mark Evenson]

Congratulations on getting the divergent ANSI-TEST suite codebases back
together in one place, as well as establishing this mailing list.  As a
Common Lisp implementer who only really started studying the language a
decade ago, I have profited immeasurably from figuring out why ABCL
fails a given ANSI-TEST while other implementations don't.

I would like to point out a few "extensions" to the ANSI-TEST suite that
ABCL has added over the years so that other implementations might profit
from them and/or that some of the ideas might be included in the
ANSI-TEST suite itself.  Upon reviewing the implementation of these
ideas, parts of me cringe at how my younger self implemented things so I
wouldn't necessarily recommend adoption of our code wholesale.

1)  ASDF

By the nature of how ANSI-TEST loads and runs itself, encapsulation as a
"true" ASDF system (i.e. one where all the source is declared) seems a
little farfetched.  And maybe it is useful to be able to run the suite
in places where ASDF is not available, such as when bootstrapping a new
implementation.  Nevertheless, we put some work into encapsulating the
ANSI-SUITE into ASDF to the point that it can be loaded and the tests
run that might be of interest.

[ABCL defines three ASDF systems][1], namely  ANSI-RT, ANSI-COMPILED,
and ANSI-INTERPRETED.  ANSI-RT encapsulates the regression test
framework, while ANSI-COMPILED and ANSI-INTERPETED provide definitions
to the point that ASDF:TEST-SYSTEM can run the compiled or interpreted
tests respectively.  This helps with automation of the tests in ABCL's
build mechanism.  The one caveat is that one cannot run tests in
"parallel" on, say, different versions of your implementation that you
are regression testing due to the way that ANSI-TEST uses the filesystem
of the ANSI-TEST itself to keep fasls and record state for tests in the
course of running the suite.

[1]: http://abcl.org/trac/browser/trunk/abcl/abcl.asd#L76


2)  ABCL.TEST.ANSI

In order to support the necessary shenanigans to make the ANSI-COMPILED
and ANSI-INTERPRETED definitions work, we [created a small set of
utility routines][2].  CLEAN-TESTS performs the removal of intermediate
files that the old Makefile used to do.  DO-TESTS-MATCHING will run only
those tests whose symbols match a regular expression, which is useful
for repeatedly running a subset of tests.

[2] http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/abcl-ansi.lisp


3)  ANSI-TEST error "database"

We [created a simple s-expr "database"][4] for recording test failures
and then [reporting on the results][3].  ABCL is a unique CL
implementation because it is hosted on a JVM for which there are
multiple implementations and whose behavior often varies by operating
system so running the ANSI-TEST suite has proven to be fairly sensitive
to hosting JVM implementation version and operating system.  Thus, such
a database of regressions has proven to be of special importance to
ABCL.  That being said, my implementation of this idea is pretty
amateurish in hindsight, so I think what is important here is the idea
rather than the implementation.

[3]:
http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/parse-ansi-errors.lisp
[4]:
http://abcl.org/trac/browser/trunk/abcl/test/lisp/ansi/ansi-test-failures

-- 
"A screaming comes across the sky.  It has happened before, but there
is nothing to compare to it now."