default method for test-system?

[Faré]

On Sat, Apr 25, 2015 at 4:52 PM, Robert P. Goldman <rpgoldman at sift.net> wrote:
> Faré wrote:
>> Dear ASDF users,
>>
>> what about this default method (or something similar) for test-op?
>>
>> (defmethod perform ((o test-op) (s system))
>>   (loop :with name = (coerce-name s)
>>         :for suffix :in '("" "/test" "-test")
>>         :for test-system-name = (strcat name suffix)
>>         :for test-system = (find-system test-system-name nil)
>>         :when test-system :do
>>           (load-system test-system)
>>           ;; Would be string-upcase, if not to accommodate for "modern" syntax.
>>           ;; UIOP probably needs to offer a standard way to abstract over
>>           ;; the string-upcase vs string-downcase vs neither default.
>>           (flet ((symbol-namify (x) (string (safe-read-from-string x))))
>>             (if-let (test-suite (find-symbol* '#:test-suite
>> (symbol-namify test-system-name) nil))
>>               (return (funcall test-suite))))
>>         :finally (error "No tests found for system ~A" name)))
>>
>
>
> I like this, but I'd like a clear statement of its objective.  I believe
> I can reconstruct it as follows:
>
> By default, look for a test system with the name <system-name>/test or
> <system-name>-text.  Load that system, and then look for a function
> called "test-suite" in the package with the same name as the test system.
>
> I'd like to suggest a modification to the above, that teases things apart.
>
> 1. Look for the test system as you do, but add "-tests" and "/tests".
> 2. Do not try to do the PERFORM on the main system, but instead delegate
> it to the test system.  I.e., do the equivalent of
> (in-order-to (test-op system) (test-op test-system))
> 3. Add a new class: TEST-SYSTEM, and make the code that does TEST-SUITE
> be the default TEST-OP method on TEST-SYSTEM.
> 4. We could additionally allow the users to specify the TEST-SYSTEM
> corresponding to any system for themselves, and use that name in place
> of the auto-search when it's available.
>
Yeah, something like that would possibly be a better API.
a :test-system slot to specify the test system, and/or
a :test-suite slot to specify the test suite function.
The user-visible reader function would guess the results based on heuristics
only when said slots are not specified.

> A slight variation would be to make the invocation of an appropriately
> named TEST-SUITE function be the default TEST-OP for *all* systems.
>
> We could also have the TEST-SYSTEM class have an optional TEST-PACKAGE
> initarg that would allow users to specify where to look for TEST-SUITE.
>
Or have TEST-SUITE be a string that specifies a function, just like
for other similar user-configurable functions in ASDF.

> I would bump the version numbering of ASDF to either 3.2 or 4 and add a
> #+ to allow users to easily add this to their systems without breakage.
>
That's a good idea — but please only do it after the feature is stable
enough. The #+feature is useless if the functionality moves a lot
afterwards. And yes, the functionality isn't as useful until the
feature exists, so there's a bootstrap issue for it to become popular.
But in the world of ASDF, it's common to have to wait for two years or
more before users can assume a feature is ubiquitous and rely on it.

Can we declare ASDF3 to have fully replaced ASDF2 yet, and encourage
users to wholly drop support for ASDF2, just like we encourage them to
actively stop supporting ASDF1? Probably not until LispWorks 7.0 is
released with ASDF3. And we're nowhere near able to declare ASDF3.1 as
universally supported. CLISP is notably still using 3.0.2.26 (!).

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
A student who changes the course of history is probably taking an exam.