A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[Robert Goldman]

On 2/11/16 Feb 11 -8:12 PM, Faré wrote:
> On Wed, Nov 18, 2015 at 8:59 AM, Robert Goldman <rpgoldman at sift.net> wrote:
>> Commonly, one wants to extend ASDF with new operation and component classes.
>>
>> But the support in ASDF for referencing such classes all involves
>> automagically interpreting keyword symbols (and some unqualified
>> symbols?) in the ASDF package.
>>
>> If I understand this correctly, this leaves the ASDF extender with the
>> alternatives of either jamming their new symbols into the ASDF package
>> late, and exporting them, or having them live in a new package, which is
>> cleaner, but relegates these new components to second-class status.
>>
>> Consider the following
[...snip....]
> The problem is that the defsystem form is read before the
> :defsystem-depends-on clause can be processed, and hence any symbol
> used in the defsystem form MUST be in a package that is already
> defined by the time the defsystem form is read. If you stick to a
> purely declarative style of not having definitions in your .asd file,
> that pretty much means only the asdf and keyword packages are allowed.
> 
> defsystem-depends-on is also broken in that ASDF assumes that it
> doesn't mark such extensions as needing to be reloaded and the .asd
> files that use them reprocessed should these extensions or their
> dependencies be modified.
> 
> On the one hand, ASDF is very wrong. On the other hand, side effects
> everywhere mean that even if we ever fix that wrong (which will be a
> lot of work), there's still a lot of wrong in trusting extensions to
> not have persistent side-effects.

This is very frustrating to me.

In the old days, we would just put

(asdf:load-system "extension")

at the top of the file.  Everybody said "oh, noes! side effects!"

So we added DEFSYSTEM-DEPENDS-ON.

This adds a truckload of complex code to ASDF that we must maintain, but
basically doesn't fix anything:

* The extensions don't work like native-to-ASDF constructs, unless you
put them in the ASDF package, so we are encouraging programmers to
perpetrate name collisions.

Note that the "bad" side-effecting construct is *BETTER* than this,
because the load form can  be put upstream of a defpackage in the asd
file, so that the symbols will exist before the DEFSYSTEM is read.  This
means that ASDF extenders are *not* encouraged to mess up the ASDF package.

* The dependencies, as you point out, are broken.  Indeed, they may be
*more* broken than when we put ASDF:LOAD-SYSTEM forms in the file.

The only thing that DEFSYSTEM-DEPENDS-ON fixes, then, is introspection.
 But a *declaration*, instead of a semantic form would solve this
problem, in a much simpler way.

I suggest that DEFSYSTEM-DEPENDS-ON should turn into a declaration.
I.e., it should provide introspection semantics for, e.g., quicklisp,
but that's it.  To keep stupid things from happening, it should error
out if the depended-on system is not loaded at the time the follow-on
system is loaded.  That's it.  End of story.

Simplification ensues, as opposed to us having to maintain fussy
propagation code.

Also, we stop encouraging people to bork the ASDF package, which is A
Bad Thing to do.

Indeed, it's tempting to package lock ASDF on lisps that support that
operation....