set variables before executing the "test-op"

[Robert Goldman]

On 9/6/16 Sep 6 -8:03 AM, Stelian Ionescu wrote:
>> It's probably wrong to set those settings from your .asd file, since they may be set or reset before your project runs, or in between two runs.
>> If you actually care about those variables, define a function that sets them,
>> and call it at the beginning of those files.
>>
>> If you have a lot of files, define a class for those files that does it in its perform method for basic-load-op.
>>
>> As for defining accessors before the packages are interned,
>> to be executed by a function run *after* they are interned,
>> you can use such idioms as:
>>    (setf (symbol-value (find-symbol* :*enable-colors* :prove.color) nil)
>> Note that find-symbol* is defined by uiop, which is :use'd by :asdf-user.
>>
>> Alternatively, you could (load-system :prove) in your .asd file,
>> but it's ugly.
> 
> This sounds like a good moment to come up with an interface between test-system and the test suite runner, so that you can pass arguments to the test runner directly through asdf:test-system. Using dynamic variables for this use case is a bad idea.

Offhand, I don't see how this is possible.  If a test library decides to
use dynamic variables to control its function, how can ASDF fix that?

Mostly I try to handle this by writing methods on

PERFORM :around ((op test-op) (system (eql (find-system ...))))

Actually, Alexey's case looks more complex, since it involves some call
to CLEAR-SYSTEM.

This suggests that there's some setup and teardown that isn't being
handled by the test library itself.

What I have done in my work on testing with 5AM is to add special error
classes for test-failures, an unexpected number of tests (we had
problems once where due to a bug tests were not run, which didn't yield
errors but.... wasn't working, either), and unexpected test passes.

That addresses the problem of ASDF not returning values from its operations.

Best,
r