A modest proposition: DEFSYSTEM-DEPENDS-ON should die [was Re: What's the right way to extend ASDF with new symbols?]

[Robert Goldman]

On 2/15/16 Feb 15 -10:26 AM, Stelian Ionescu wrote:
>> On 2/12/16 Feb 12 -3:15 PM, Stelian Ionescu wrote:
>>> On Fri, 2016-02-12 at 16:07 -0500, Faré wrote:
>>>> I'm OK with declaring DEFSYSTEM-DEPENDS-ON a failure, and load-system
>>>> (or load-systems) the official way to go. But
>>>>
>>>> 1- This of course requires heads up, updating all users before
>>>> retiring the feature, etc. From my experience, if you start seriously
>>>> deprecating today and sending patches to all authors who use it in
>>>> quicklisp, you can expect to be removing that part of the code in two
>>>> years or so.
>>>>
>>>> 2- To make these dependencies work properly still requires modifying
>>>> ASDF to add explicit plan nodes for loading ASD files, that will
>>>> contain the systems loaded by load-system. The same trick will also
>>>> automatically make the :defsystem-depends-on work, since it itself
>>>> calls load-system.
>>>>
>>>> 3- Yes, defining things in the ASDF package is ugly, but extensions
>>>> are few enough, and using a prefix is a good enough namespace
>>>> management strategy. Not the most horrible thing that working with CL
>>>> does to you.
>>>
>>> Please don't. It's a net improvement compared to the previous situation.
>>> It's easy to simply name your class with a keyword :package/foo-file.
>>>
>>
>> With all due respect, no, it is not.  It is a net *negative* with
>> respect to the previous situation.  Here are the reasons why, briefly
>> reiterated:
>>
>> 1. It effectively forces you to stick new symbols into the ASDF namespace.
> 
> Technically yes, but that's not the essential requirement. Because the CL reader interns eagerly, ASDF extension classes need to be interned into a package that is owned by ASDF. Currently that's the ASDF package itself, but it would be a good idea to add a special package for extensions, and start encouraging people to use that by showing appropriate deprecation messages.

This doesn't do anything to resolve the underlying problem, AFAICT.
I.e., instead of getting name collisions in ASDF (I believe that's
ASDF/INTERFACE), we get name collisions in ASDF/EXTENSIONS.

Am I missing something about this suggestion?
> 
> 
>> I don't understand your proposed rebuttal, involving slash-named
>> packages. I don't see any evidence of this being legal ASDF syntax,
>> looking at FIND-CLASS*, and trying an experiment.  If it works, it needs
>> documentation.  If it does not work, it will not be added -- ASDF must
>> get simpler, not more complex.  Please amplify, thanks!
> 
> It's not "syntax", it's just manual namespacing. The symbol 'asdf::pkg/class is symply a legal symbol. And I don't agree that ASDF must get simpler at all costs, rather that it should have as simple an implementation as possible, while still allowing the use cases that its users require.

With all due respect, I believe ASDF to be unmaintainable now, except
for a Hail Mary Pass to Faré at regular intervals.  Again, you are
welcome to have a whack at it.  I won't be upset if you prove me wrong!
> 
> 
>> 2. If you are arguing that we can just solve this with a naming
>> convention, I don't buy it.  Consider different libraries, each of which
>> hook ASDF to normal "make" by creating MAKE-FILE and MAKE-OP classes.
>> This is not a far-fetched example; I have seen it.  They both try to jam
>> these symbols into ASDF.
> 
> See my previous reply.
> 
> 
>> This behavior should be *strongly* discouraged, even to the extent of
>> (as I said earlier) package-locking ASDF when possible.  Currently, it
>> is actively *ENCOURAGED* -- close to mandated, in fact.
> 
> By dedicating a package for naming extension classes, we can even lock ASDF while still making QA easier.

Per earlier response, this seems to me to just kick the problem from one
namespace/package to another.
> 
>> 3. If we make DEFSYSTEM-DEPENDS-ON into a declaration, a lot of
>> simplification ensues, including eliminating the complex double-parsing
>> of DEFSYSTEM.  ASDF is currently over-complicated and over-long.
> 
> In what sense is it not currently a declaration ?

What I meant is that it's not *only* a declaration.  It has extra
operational import.   I would like to change D-D-O to be only
declarative (except that we will also check it).

> 
> 
>> 4.  The double-parsing doesn't even work, because the packages don't
>> exist at the right time.  That's why, even with DEFSYSTEM-DEPENDS-ON,
>> you must either mess with the ASDF package, or put in a LOAD-SYSTEM call
>> to get the symbols created, and stuck into *PACKAGE* before the
>> defsystem form is parsed.
> 
> I think this might be an issue with the current implementation of DEFSYSTEM-DEPENDS-ON, but that's not a necessity. E.g.:
> 
> (defsystem foo
>   :defsystem-depends-on (foo/asd)
>   :components ((:foo/file "foo")))
> 
> This ought to mean that LOAD-SYSTEM of "foo" depends on LOAD-SYSTEM of "foo/asd", and that the exact class object named by "foo/file" should be fetched right after loading "foo/asd".

Fetched from where?  Do you want to further extend the syntax so that
FOO/FILE is interpreted as FOO:FILE?  That might be feasible (suggest
you try).  It would certainly be preferable to adding an ASDF/EXTENSIONS
package and fighting over its contents.

*HOWEVER* that doesn't address the operation-naming problem, which would
require further syntactic extensions.

This is what really concerns me -- doing DEFSYSTEM-DEPENDS-ON *properly*
requires a lot more work, which has not been well thought out.  And it's
not something I want to do (I maintain ASDF because I rely on it, not
because I have a fundamental interest in system-building systems -- Faré
was more interested in this as a research problem).  I'd rather remove a
mostly-broken implementation than maintain it.

>> 4. backwards compatibility involves nothing more than adding a
>> LOAD-SYSTEM form to the top of a file.
> 
> This breaks one of the most important use cases: doing large scale static analysis of dependencies, like in the case of Quicklisp. At the moment, one cannot even be sure of parsing a .asd without compiling code, unless using a custom reader and/or regular expressions.

I get your point, but this won't break anything that isn't already
broken.  If you want to refer to new components as first class entities
(i.e., without syntactic mess),  the only thing you can do is the following:

(1)
(load-system <extension-system>)
(defpackage my-system-asd
  (:use asdf extension-package common-lisp))
<system declaration>

... or (2) shovel stuff into the ASDF package.

So if programmers do (1) already, your important use case is already
broken.  And there is no way to prevent them from doing so.

My guess is that Quicklisp simply doesn't contain libraries with that
construct (which is fine for QL).  For example, CFFI takes method (2)
and jams stuff into the ASDF package to make GROVEL-FILE work, rather
than keeping its extensions in a separate package.

If ASDF was a different system, and it had special files, instead of
files that are lisp source, and can contain arbitrary forms, then this
argument would be stronger.  But it's not that way.  Maybe it *should*
be, but that's a different system.

As an aside: I think ASDF has gone as far as it can go on gut, rather
than spec.  The discussion here shows that we have reached those limits,
because we have features that are poorly understood, not well-documented
anywhere, and that become critical while still only lightly baked.
E.g., for me the namespace pollution is a critical pain-point, which
doesn't seem to bother others.  Contrariwise, I have no idea what
requirements Quicklisp exerts on ASDF (especially now that Xach has fled
this mailing list).  I don't believe either of us well understands the
current design intent.

> 
> 
>> 5. DEFSYSTEM-DEPENDS-ON as introspection will persist, so that
>> introspection continues to be supported.
> 
> I'm not sure what you mean by this.

You will still be able to load systems and find out which ones depend on
which other ones.  In the old days, before DEFSYSTEM-DEPENDS-ON, method
(1) was mandatory for extending ASDF, and there was no way to
interrogate the lisp image t o tell what extensions were required by
what systems.
> 
> 
>> TL;DR: We have a "declarative" construct which is not declarative at
>> all.  Worse, it almost forces poisonous namespace pollution.  Killing it
>> would simplify the code.  Minimal backwards-compatibility issues.
> 
> I think I showed how it can be improved without removing it. Killing it would be a disaster for those who rely on it.

I don't think we are there yet.  Showing how it can be improved requires
an approach that covers *both* component types *and* operations, for one
thing.  It should also support use of normal syntax (possibly extended,
like your "slashy" syntax) to reference extensions *without* shoving new
symbols into ASDF.

If someone wants to provide this, I would take such in place of removing
the current behavior.  *BUT* "providing" does not mean a patch only.  It
would have to be accompanied by a list of requirements -- nothing too
fancy, a bulleted list in Markdown or something like it would be fine --
and a paragraph or two explaining how the proposed approach meets those
requirements.

If you are interested in that, I'd be happy to help you assemble the
requirements: I think they are hidden in this email exchange.
> 
> 
>> Alternative: if you, or someone else, will take over ASDF
>> maintainership, you can keep DEFSYSTEM-DEPENDS-ON.  I will happily leave
>> it to you!
> 
> I'm considering this.
> 

Offer is still open.

Cheers,
r