Proposal to improve the loading of the abcl-contrib dependency.

[Ralph Ritoch]

Mark,

   I have reviewed your suggestions and they are certainly progress in the
right direction.  I have two concerns, the first is with the path-spec for
locating configuration files.

jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf

It generally considered a bad practice for two libraries to contribute to
the same package and while the META-INF folder falls into system dependent
functionality I believe a better pathspec would be accommodate this to
"namespace" the configuration files. I'm aware that this isn't the standard
practice, but if the namespace is properly named it is will be easier to
track configuration files back to the library they came from.

jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*/*.conf

My second concern is the method of configuring the contrib search. The
default should certainly support the "legacy" search methods and fallback
to the classpath, but to avoid imposing limitations on future systems I
believe a better algorithm would be to allow registration of custom
configuration search handlers such as having a
sys:*configuration-providers* which contains and ordered list of
configuration provider functions. In the case of web based apps, this opens
the potential to cache remote configuration files.

I am looking forward to the progress of this, expanding this functionality
beyond the configuration of one specific library could greatly improves the
application lifecycle for ABCL applications.

As for the maintenance of the maven build system, the pom's I submitted
compile the java files, and builds the jar, but is calling ant targets.
While ant can call maven it still comes down to the key issue that maven
doesn't pick-up a jar created by an external process. Maven central
requires a source release, a binary release, and javadoc release, which all
must be GPG signed. The complexity of trying to make maven a slave to ant
is comparable to the complexity of the current "manual" system and I
believe it will lead to the same situation you have now where no one does
releases to maven. While there may be room for improvement with the maven
system, there's normally little to no maintenance needed, once it is
properly configured. The only thing that must be changed are the version
numbers, and any dependencies. If you can make maven a slave to ant such
that it will deploy to central and local repositories, that would be fine,
but other than creating a custom maven plugin I don't know any way it would
be possible without allowing maven to package and sign the jars.

I was advised that it would be possible to do away with the ant system all
together. What was suggested to me by java developers was to have the java
sources built in one module, which has a post-build script to compile the
lisp files, and a second module which uses the target (generated) files of
the first module as it's resource path. Calling ant is much easier since
the ant system is already built, but it is possible to completely deprecate
ant in favor of maven.

Best Regards,
  Ralph Ritoch

Date: Mon, 27 Jul 2015 10:42:30 +0200
From: Mark Evenson <evenson at panix.com>
To: armedbear-devel at common-lisp.net
Subject: Re: Proposal to improve the loading of the abcl-contrib
        dependency.
Message-ID: <55B5EEF6.5030601 at panix.com>
Content-Type: text/plain; charset=utf-8



On 2015/7/26 14:29, Ralph Ritoch wrote:
> Hi,
>
>    I have run into an issue with making executable abcl jars when the jar
> depends on, and provides, abcl-contrib. As far as I can tell abcl looks
for
> abcl-contrib in the following locations.
>
> 1. Looks for a jar named abcl-contrib.jar in the classpath determined by
> probing the classloader for the jars it provides.
> 2. Looks in the directory of the jars in the classpath for a file named
> abcl-contrib.jar
>
> Neither of these solutions work if the currently executing jar includes
> abcl-contrib and isn't named abcl-contrib.jar.

The contrib file may also be named via a versioning suffix something
like 'abcl-contrib-m.n.o.jar'.

Some background on the current design: abcl-contrib.jar was explicitly
separated from abcl.jar in order to have a clean separation between the
code providing an ANSI Common Lisp, and the ABCL-specific extensions to
this.  The goal was to bound the size of the abcl.jar to some reasonably
constant (over time) amount.  In addition, function symbols in abcl.jar
are subject to an "autoload" mechanism which delays the loading of their
implementation until their first invocation.  Consequently, the Lisp
code in abcl.jar is loadable by ABCL-specific SYS:LOAD-SYSTEM-FILE
(which is used by the ANSI REQUIRE mechanism), while abcl-contrib.jar
may assume the full ANSI environment so may be addressed via CL:LOAD.

> The following code proves that it is relatively easy to add this
capability.
>
> (in-package 'sys)
> (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref
> (java:jcall "getURLs" (sys:boot-classloader)) 0))))
>
> This code properly loads abcl-contrib when the executing jar provides
> abcl-contrib, while (require 'abcl-contrib) does not.

A better implementation would be to add this capability to the mechanism
used by (require :abcl-contrib) which corresponds to directly to [code
contained in abcl-contrib.lisp][1].

[1]:
http://abcl.org/trac/browser/trunk/abcl/src/org/armedbear/lisp/abcl-contrib.lisp

> My proposal is to add a file named abcl-contrib/version.lisp to the
> abcl-contrib.jar, possibly in the /META-INF directory to avoid any
> potential conflicts with java, and to have sys:find-contrib look for this
> file in the resources provided by the classpath. The contents of this file
> is not important.

I think 'version.lisp' is a little too generic, and having its contents
be irrelevant missing a design opportunity.

All SYS:ADD-CONTRIB essentially does is add locations to ASDF.
Therefore it would be better to use the facilities of ASDF with [ABCL's
implementation of JAR-PATHNAME][2] to search for an ASDF source-registry
configuration at a well known path within a JAR, something like
<jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf>.
 The files in this zip entry would then further specify specific ASDF
locations within the jar.  This has an advantage over the current
mechanism which just looks for *.asd files in the second level
directories, as supporting much more flexible packaging.

[2]:
http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.markdown

While I don't think we lose anything, I would like to note that
implementing this proposal would weaken the difference between an
ABCL-CONTRIB provided system and any other system packaged by ASDF.
With the proposed (on the asdf development mailing list) strengthening
of the rules and facilities for metadata in ASDF system descriptions, we
should be able to (eventually) differentiate an ABCL-CONTRIB system from
any other ASDF system if there were a valid need for this.

As part of the modification of SYS:ADD-CONTRIB, the current use of
ASDF:*CENTRAL-REGISTRY* should be re-written in favor of the explicit
configuration API exported by ASDF.

[?]

> It would be relatively easy for me to produce a patch to provide this
> feature as part of ABCL. It may be possible to provide this feature as an
> add-on, but the functions needed within the system package are private
(not
> exported) so it would be difficult, if not impossible, to implement this
> feature as an add-on.

The argument that the symbols one needs in SYS are not exported makes no
functional difference to implement anything:  you can still call them.
What difficulty/impossibility do you see here?

That the symbols which are exported from SYS and EXT should be done in a
cleaner, more consistent, manner is a point I readily accept.  Ideally
over time, I wish to only external symbols in the EXT package to
constitute the API for extending ABCL.

[?]

In order to sponsor these changes to ABCL trunk, I would like to see:

1.  The build instructions for Ant and Maven must share the same
instructions.  I do not want to chase the maintenance of two build
processes.  The path of least resistance to the rest of the current ABCL
community would be to have Maven slave into the Ant build process at all
necessary points, including sharing intermediate artifact locations and
packaging instructions.

2.  SYS:ADD-CONTRIB should use the ASDF configuration API as the
mechanism to add new system locations based on the presence of a special
directory such as
<jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf>.

3.  The code in SYS:ADD-CONTRIB should initially try to locate
abcl-contrib{,-m.n.o}.jar files, falling back to searching the
classpath.  Since searching the classpath may be a potentially expensive
operation--especially if it contains locations on a remote network--we
may want to consider a runtime option and/or special variable to limit
the search strategy.

4.  Since the classpath can be modified, there should be a mechanism to
explicitly request the ABCL-CONTRIB initialization strategy be re-run.
I'm not sure the current code does this very well.

5.  We should to test the new loading strategy in a) Java Servlet
(Tomcat/Jetty) b) Google App Engine and c) OSGi containers.

Otherwise, good work!  I think with a little back-and-forth on design,
we can get something like this into ABCL in the near term.

--
"A screaming comes across the sky.  It has happened before, but there
is nothing to compare to it now."


On Sun, Jul 26, 2015 at 8:29 PM, Ralph Ritoch <rritoch at gmail.com> wrote:

> Hi,
>
>    I have run into an issue with making executable abcl jars when the jar
> depends on, and provides, abcl-contrib. As far as I can tell abcl looks for
> abcl-contrib in the following locations.
>
> 1. Looks for a jar named abcl-contrib.jar in the classpath determined by
> probing the classloader for the jars it provides.
> 2. Looks in the directory of the jars in the classpath for a file named
> abcl-contrib.jar
>
> Neither of these solutions work if the currently executing jar includes
> abcl-contrib and isn't named abcl-contrib.jar.
>
> The following code proves that it is relatively easy to add this
> capability.
>
> (in-package 'sys)
> (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref
> (java:jcall "getURLs" (sys:boot-classloader)) 0))))
>
> This code properly loads abcl-contrib when the executing jar provides
> abcl-contrib, while (require 'abcl-contrib) does not.
>
> My proposal is to add a file named abcl-contrib/version.lisp to the
> abcl-contrib.jar, possibly in the /META-INF directory to avoid any
> potential conflicts with java, and to have sys:find-contrib look for this
> file in the resources provided by the classpath. The contents of this file
> is not important.
>
> This change makes it possible to include the abcl-contrib "dependency"
> directly inside the jar application that is currently running, regardless
> of the name of the jar file. Therefore, applications which make use of
> abcl-contrib can be distributed as a single jar.
>
> For maven users, this means that their uberjars will work by just
> including abcl and abcl-contrib as dependencies.
>
> For ant users they would probably need to extract abcl-contrib and include
> the extracted files in their generated jar which I believe can be fully
> automated.
>
> It would be relatively easy for me to produce a patch to provide this
> feature as part of ABCL. It may be possible to provide this feature as an
> add-on, but the functions needed within the system package are private (not
> exported) so it would be difficult, if not impossible, to implement this
> feature as an add-on.
>
> I would like your feedback on this issue. If this is a feature you want,
> or if you have any specific objections to including this feature in ABCL,
> please let me know.
>
> Best Regards,
>    Ralph Ritoch
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/armedbear-devel/attachments/20150801/1839f3d3/attachment.html>