Impact on modifying the current planning code (was Re: Plans for ASDF 3.3)

[Mark Evenson]

On 5/1/17 03:45, Faré wrote:
> I took the content of this thread and edited it into a blog post:
> http://fare.livejournal.com/188940.html
>
> Robert, can you tell me what the plan should be for ASDF 3.3, the
> "plan" (phase separation) branch and the "syntax-control" branch?

Thanks for the detailed write-up, Faré, as such exposition from your
ASDF wrangling helps me greatly in understanding why things are the
way they are in ASDF.  Bonus points for the snark in "Common Lisp is a
hippie language that disallows disallowing", at which I am still
chuckling.

In the JVM ecosystem, the standard way to identify packaged binary
artifacts ("jar files") and their dependencies is via the Maven
["coordinate system" in the POM model][1] which associates every
artifact with a group-id:artifact-id:version triple.  An example of
such a triple would be ["org.abcl:abcl:1.4.0" ][2].  The use of the
POM model has been adopted by almost (?) all other current "Java the
platform" build systems, including Clojure's Leiningen and ABCL's
extensions to ASDF, [abcl-asdf][3].

[1]: https://maven.apache.org/pom.html
[2]:
https://search.maven.org/#artifactdetails%7Corg.abcl%7Cabcl%7C1.4.0%7Cjar
[3]:
https://gitlab.common-lisp.net/abcl/abcl/blob/master/contrib/abcl-asdf/abcl-asdf.lisp

Unfortunately, the current implementation of ABCL-ASDF took a naive
shortcut that needs fixing, as it works on a per-dependency basis as
opposed to properly sorting and reducing declared dependencies to a
minimal working set for a given ASDF system definition.  Alan
Ruttenberg has [proposed a short term fix][5] that we are currently
experimenting with, but the proper way forward as I understand it,
would be to push more of this dependency logic into ASDF where the
"plan" phase would be able to properly reason about dependencies.

[4]:
https://mailman.common-lisp.net/pipermail/armedbear-devel/2017-April/003810.html
[5]:
https://mailman.common-lisp.net/pipermail/armedbear-devel/2017-April/003838.html

An example to illuminate the problem as I understand it:

Currently, when ABCL encounters the following definition

    (asdf:defsystem :log4j
      :defsystem-depends-on (abcl-asdf)
      :components ((:module log4j.jar
                            :components ((:mvn "log4j/log4j/1.2.17")))))

a request is made to the Maven code to satisfy the dependency for
"log4j/log4j/1.2.17" and add all transitive dependencies to the JVM's
classpath.  This graph is discovered at request time by parsing all
the POM models from the network.  Issuing this request at an ABCL repl
just returned the following for me:


"/Users/evenson/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/evenson/.m2/repository/javax/mail/mail/1.4.3/mail-1.4.3.jar:/Users/evenson/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/Users/evenson/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.0/geronimo-jms_1.1_spec-1.0.jar"

which means a request for a logging framework dependency
(`log4j-1.2.17.jar`) has also pulled in an SMTP implementation
(`mail-1.4.3.jar`) amongst other things (a common problem with the
JVM, apparently not easily solved as the "resolution" of the packaging
in jar files is too coarse grained).  ABCL-ASDF currently only creates
a single ASDF:MVN object that represents the entire request, not
representing the other binary artifacts it introduces in a way that is
in any way amendable to "planning".

I figure that the right way forward would be to create additional
ASDF:MVN objects for each discrete dependency in-memory that would not
have a corresponding ASDF:MVN definition in a file on the filesystem.
To continue the example, the ABCL-ASDF code would create the
discovered "javax.mail/mail/1.4.3",
"javax.activation/activation/1.4.3", and
"org.apache.geronimo.specs/gerionimo-jms/1.1" ASDF:MVN objects, using
them to reason about what to load.

Questions for consideration by the ASDF community:

1.  As I understand it, my problem doesn't have the problems of
    redefining ASDF behavior during ASDF's load phase that Faré wishes
    to eliminate in asdf-3.3.  If I were to slog through asdf-3.2.x to
    implement the planning which I need would such an effort be
    impacted in any way that you can foresee by what you need to change
    for asdf-3.3?

2.  Does anyone know if there an existing analogy for the ASDF:MVN
    component in an ASDF extension that I could profitably study?
    Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT
    association with a pathname, mainly because in the current
    implementation a given ASDF:MVN component results in one or more
    jar archives.  Does creating additional ASDF:MVN (a subclass of
    ASDF:COMPONENT) instances in the in-memory ASDF that aren't
    reflected in an *.asd file raise any problems that anyone is aware
    of?

3.  In the last few months, I think I remember that there has been
    discussion around the possible use of ASDF to locate and download
    shared objects for CFFI definitions (or maybe this was within the
    CFFI community?).  The ABCL-ASDF case is a little simpler in that
    the needed Maven binary objects are identical, unlike the CFFI
    problem which has per-operating systems and (possibly) per-dynamic
    linker implementation dependencies.  But still, as I remember the
    outcome of that discussion, the general feeling was that such a
    mechanism does not belong in ASDF.  Does the ASDF cognoscenti
    think that what I am proposing here for ABCL-ASDF also seem to be
    "too much"?  Note, that ABCL-ASDF will only ever work on ABCL
    (unless, of course, we get another JVM CL implementation), and as
    such, is intended to be written as an ASDF extension that should
    have no impact on other's usage of ASDF.  Still, what has been
    implemented (and what I am proposing), seems to violate Faré's
    description of ASDF as a build system that should only deal with
    Common Lisp artifacts.

Thanks for the attention and the solid re-engineering of ASDF,
Mark <evenson at panix.com>

-- 
"A screaming comes across the sky.  It has happened before, but there
is nothing to compare to it now."