Impact on modifying the current planning code (was Re: Plans for ASDF 3.3)

[Alan Ruttenberg]

Here is what I see a problem is: ASDF only resolves dependencies on a
system by system basis. The cross-system link is that if system B depends
on system A we can avoid loading A if it has already been loaded. If an
identical source file happens to be in both B and A that isn't recognized,
but also doesn't have consequence - the functions in it are redefined.

Resolving the maven dependencies in java needs to be a globally done across
ASDF systems. If one ASDF system A depends on log4j version 2.1 or higher
and another B depends on 2.9, and the range of versions is 2.1-2.12 then if
A is loaded first there is no way to satisfy the second dependence in a
predictable way. If you choose the highest available version A can take it
is too high for B. If you choose the lowest A can choose it is too low for
B.

Java systems manage this because they are built from a single root POM
file. That POM can include other POMs which have additional dependencies,
but solving the full set of dependencies consistently is done at one fell
swoop. While there may be seeming conflicts, there are mechanisms for
making choices - the including POM can specify a version for a dependency
an included POM specifies - "managed dependencies" and exclusions. There's
also a fundamental difference in that what maven is doing with the
dependencies is collecting them into a classpath, and then the process is
run with the classpath. Because of this there aren't load-time side
effects, as one has with lisp. There are static initializers that depend on
classes in other jars, but these are never called for until the process is
started.

I'm honestly not sure how to handle this generally, or specifically with
ASDF.  One piece could be having a persistent registry of components (as
opposed to just systems). Then, instead of just reloading a component that
happens to be the same, we could detect whether the requested component's
version is compatible with what was previously loaded. Another piece could
be to distinguish between dependencies needed for loading the system versus
runtime, and then handling the planning for a system and all the systems it
depends on at once, instead of one planning one system at a time, with the
default being load-time independence. That default would be analogous to
the need to make exceptions for within file by use eval-when
:compile-toplevel.

Hope this helps. Perhaps both these capabilities are already easily
available. My use of ASDF doesn't extend far enough to know, and I haven't
spent a lot of time studying ASDF.

Any ideas would certainly be welcome.

Alan

On Thu, May 4, 2017 at 11:08 AM, Faré <fahree at gmail.com> wrote:

> Dear Mark,
>
> thanks for your request. I'm not sure I understand how your ASDF
> dependencies do or don't map to MVN entities, though. I also don't
> understand the maven model very well.
>
> > Questions for consideration by the ASDF community:
> >
> > 1.  As I understand it, my problem doesn't have the problems of
> >     redefining ASDF behavior during ASDF's load phase that Faré wishes
> >     to eliminate in asdf-3.3.  If I were to slog through asdf-3.2.x to
> >     implement the planning which I need would such an effort be
> >     impacted in any way that you can foresee by what you need to change
> >     for asdf-3.3?
> >
> If you're going to go deep inside ASDF's planning code, I would
> strongly recommend starting off the 'plan' branch (due to be committed
> in ASDF 3.3) rather than off 3.2, as there was significant refactoring
> and merging would be a huge pain.
>
> Also, if some of your dependencies are from defsystem-depends-on, you
> *definitely* need to start off the 'plan' branch.
>
>
> > 2.  Does anyone know if there an existing analogy for the ASDF:MVN
> >     component in an ASDF extension that I could profitably study?
> >     Currently, ABCL-ASDF hackishily neuters the ASDF:COMPONENT
> >     association with a pathname, mainly because in the current
> >     implementation a given ASDF:MVN component results in one or more
> >     jar archives.  Does creating additional ASDF:MVN (a subclass of
> >     ASDF:COMPONENT) instances in the in-memory ASDF that aren't
> >     reflected in an *.asd file raise any problems that anyone is aware
> >     of?
> >
> The current ASDF model is that every component has only one pathname,
> though that pathname can be NIL, or can be a directory. If you need
> multiple files, you can create multiple components indeed.
>
> I don't understand what an MVN component is or does, but I don't see
> any problem a priori with having multiple of them. Is it supposed to
> represent (a) a dependency on an external maven package? Or is it
> supposed to represent (b) a jar that is included as pre-built in the
> current source? Or (c) the ability to build a jar from source in the
> current system, that can then be pushed to maven?
>
> In case (a), it probably should be a special subclass of SYSTEM. In
> case (b) a component with a single file pathname should be fine. In
> case (c) I would probably also have some subclass of SYSTEM do it, and
> use one or many secondary systems.
>
>
> > 3.  In the last few months, I think I remember that there has been
> >     discussion around the possible use of ASDF to locate and download
> >     shared objects for CFFI definitions (or maybe this was within the
> >     CFFI community?).  The ABCL-ASDF case is a little simpler in that
> >     the needed Maven binary objects are identical, unlike the CFFI
> >     problem which has per-operating systems and (possibly) per-dynamic
> >     linker implementation dependencies.  But still, as I remember the
> >     outcome of that discussion, the general feeling was that such a
> >     mechanism does not belong in ASDF.  Does the ASDF cognoscenti
> >     think that what I am proposing here for ABCL-ASDF also seem to be
> >     "too much"?  Note, that ABCL-ASDF will only ever work on ABCL
> >     (unless, of course, we get another JVM CL implementation), and as
> >     such, is intended to be written as an ASDF extension that should
> >     have no impact on other's usage of ASDF.  Still, what has been
> >     implemented (and what I am proposing), seems to violate Faré's
> >     description of ASDF as a build system that should only deal with
> >     Common Lisp artifacts.
> >
> I don't remember any discussion about downloading CFFI shared objects
> (though I have vague recollections of discussions about cl-test-grid
> and/or Quicklisp doing automatic installation of Ubuntu packages or
> such.)
>
> > Thanks for the attention and the solid re-engineering of ASDF,
> Thanks for your appreciation.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics•
> http://fare.tunes.org
> Anyone who says he can see through women is missing a lot. — Groucho Marx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20170504/d882892f/attachment.html>