[asdf-devel] About regression tests and libraries

[Robert Goldman]

Tobias C. Rittweiler wrote:
> Robert Goldman writes:
> 
>> Tobias C. Rittweiler wrote:
>>
>>> There's (ASDF:OOS 'ASDF:TEST-OP <system>), and on very recent revisions
>>> even (ASDF:TEST-SYSTEM <system>) as an abbeviation.
>>>
>>> I don't think this has been properly documented in the manual of ASDF
>>> yet. I hence CC'd the asdf-devel mailing list so this will hopefully get
>>> promoted with the due it deserves.
>>>
>> Part of the problem with test-op is that the desired behavior has not
>> been specified by the ASDF community.  Because of the nature of ASDF, it
>> is impossible for
>>
>> (asdf:test-system <system>)
>>
>> to return a value indicating whether or not <system> has passed its tests.
> 
> What is the reason that OPERATE can not return the return value of
> PERFORM?
> 

At any rate, right now OPERATE does not return the return value of
PERFORM.  It returns the operation.  [I believe someone suggested
stashing test success/failure information on the operation in the course
of this discussion.]  Interestingly, OPERATE seems to go out of its way
to discard the return value of PERFORMs (see ll 1152-1153 of ASDF in git).

Given the plan-then-perform structure of ASDF, I'm not sure it's
entirely obvious how to restructure it so that values are appropriately
returned.  It's not clear to me how values of subsidiary tests (e.g.,
other test operations invoked by a top-level TEST-OP) would be combined
into values for the top level operation, since there's no
result-catching stack frame corresponding to the top-level operation
(this is for the same reason that it's tricky to, e.g., use ASDF to wrap
operations on a system in a dynamic variable binding).

I'm not saying that this couldn't be done, but it's not a simple change
to make ASDF do this.