Pointing out ABCL "extensions" to ANSI-TEST

[Raymond Toy]

>>>>> "Mark" == Mark Evenson <evenson at panix.com> writes:

[snip]

    Mark> 1)  ASDF

    Mark> By the nature of how ANSI-TEST loads and runs itself, encapsulation as a
    Mark> "true" ASDF system (i.e. one where all the source is declared) seems a
    Mark> little farfetched.  And maybe it is useful to be able to run the suite
    Mark> in places where ASDF is not available, such as when bootstrapping a new
    Mark> implementation.  Nevertheless, we put some work into encapsulating the
    Mark> ANSI-SUITE into ASDF to the point that it can be loaded and the tests
    Mark> run that might be of interest.

This is interesting.  It sounds nicer than running a makefile where I
always have to look to see what the magic variables are to run what I
want.

    Mark> 2)  ABCL.TEST.ANSI

    Mark> In order to support the necessary shenanigans to make the ANSI-COMPILED
    Mark> and ANSI-INTERPRETED definitions work, we [created a small set of
    Mark> utility routines][2].  CLEAN-TESTS performs the removal of intermediate
    Mark> files that the old Makefile used to do.  DO-TESTS-MATCHING will run only
    Mark> those tests whose symbols match a regular expression, which is useful
    Mark> for repeatedly running a subset of tests.

This also sounds interesting because I often do want to run just a
subset.

    Mark> 3)  ANSI-TEST error "database"

    Mark> We [created a simple s-expr "database"][4] for recording test failures
    Mark> and then [reporting on the results][3].  ABCL is a unique CL
    Mark> implementation because it is hosted on a JVM for which there are
    Mark> multiple implementations and whose behavior often varies by operating
    Mark> system so running the ANSI-TEST suite has proven to be fairly sensitive
    Mark> to hosting JVM implementation version and operating system.  Thus, such
    Mark> a database of regressions has proven to be of special importance to
    Mark> ABCL.  That being said, my implementation of this idea is pretty
    Mark> amateurish in hindsight, so I think what is important here is the idea
    Mark> rather than the implementation.

Again, this sounds pretty cool too.  Cmucl has a few extensions that
cause the test suite to fail.  It would be nice to annotate these test
as expected failures.  Idealy, a reason for the failure would be part
of the annotation.  I don't run the test suite that often anymore, but
when I do, I always forget exactly which tests are known expected
failures and which are new.  Having some kind of annotation (other
than comments on the test itself) would be awesome!

--
Ray