default method for test-system?

[Robert P. Goldman]

Faré wrote:
> Dear ASDF users,
> 
> what about this default method (or something similar) for test-op?
> 
> (defmethod perform ((o test-op) (s system))
>   (loop :with name = (coerce-name s)
>         :for suffix :in '("" "/test" "-test")
>         :for test-system-name = (strcat name suffix)
>         :for test-system = (find-system test-system-name nil)
>         :when test-system :do
>           (load-system test-system)
>           ;; Would be string-upcase, if not to accommodate for "modern" syntax.
>           ;; UIOP probably needs to offer a standard way to abstract over
>           ;; the string-upcase vs string-downcase vs neither default.
>           (flet ((symbol-namify (x) (string (safe-read-from-string x))))
>             (if-let (test-suite (find-symbol* '#:test-suite
> (symbol-namify test-system-name) nil))
>               (return (funcall test-suite))))
>         :finally (error "No tests found for system ~A" name)))
> 


I like this, but I'd like a clear statement of its objective.  I believe
I can reconstruct it as follows:

By default, look for a test system with the name <system-name>/test or
<system-name>-text.  Load that system, and then look for a function
called "test-suite" in the package with the same name as the test system.

I'd like to suggest a modification to the above, that teases things apart.

1. Look for the test system as you do, but add "-tests" and "/tests".
2. Do not try to do the PERFORM on the main system, but instead delegate
it to the test system.  I.e., do the equivalent of
(in-order-to (test-op system) (test-op test-system))
3. Add a new class: TEST-SYSTEM, and make the code that does TEST-SUITE
be the default TEST-OP method on TEST-SYSTEM.
4. We could additionally allow the users to specify the TEST-SYSTEM
corresponding to any system for themselves, and use that name in place
of the auto-search when it's available.

A slight variation would be to make the invocation of an appropriately
named TEST-SUITE function be the default TEST-OP for *all* systems.

We could also have the TEST-SYSTEM class have an optional TEST-PACKAGE
initarg that would allow users to specify where to look for TEST-SUITE.

I would bump the version numbering of ASDF to either 3.2 or 4 and add a
#+ to allow users to easily add this to their systems without breakage.

Best,
r