Proposal to improve the loading of the abcl-contrib dependency.

[Mark Evenson]

On 2015/7/26 14:29, Ralph Ritoch wrote:
> Hi,
> 
>    I have run into an issue with making executable abcl jars when the jar
> depends on, and provides, abcl-contrib. As far as I can tell abcl looks for
> abcl-contrib in the following locations.
> 
> 1. Looks for a jar named abcl-contrib.jar in the classpath determined by
> probing the classloader for the jars it provides.
> 2. Looks in the directory of the jars in the classpath for a file named
> abcl-contrib.jar
> 
> Neither of these solutions work if the currently executing jar includes
> abcl-contrib and isn't named abcl-contrib.jar.

The contrib file may also be named via a versioning suffix something
like 'abcl-contrib-m.n.o.jar'.

Some background on the current design: abcl-contrib.jar was explicitly
separated from abcl.jar in order to have a clean separation between the
code providing an ANSI Common Lisp, and the ABCL-specific extensions to
this.  The goal was to bound the size of the abcl.jar to some reasonably
constant (over time) amount.  In addition, function symbols in abcl.jar
are subject to an "autoload" mechanism which delays the loading of their
implementation until their first invocation.  Consequently, the Lisp
code in abcl.jar is loadable by ABCL-specific SYS:LOAD-SYSTEM-FILE
(which is used by the ANSI REQUIRE mechanism), while abcl-contrib.jar
may assume the full ANSI environment so may be addressed via CL:LOAD.

> The following code proves that it is relatively easy to add this capability.
> 
> (in-package 'sys)
> (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref
> (java:jcall "getURLs" (sys:boot-classloader)) 0))))
> 
> This code properly loads abcl-contrib when the executing jar provides
> abcl-contrib, while (require 'abcl-contrib) does not.

A better implementation would be to add this capability to the mechanism
used by (require :abcl-contrib) which corresponds to directly to [code
contained in abcl-contrib.lisp][1].

[1]:
http://abcl.org/trac/browser/trunk/abcl/src/org/armedbear/lisp/abcl-contrib.lisp

> My proposal is to add a file named abcl-contrib/version.lisp to the
> abcl-contrib.jar, possibly in the /META-INF directory to avoid any
> potential conflicts with java, and to have sys:find-contrib look for this
> file in the resources provided by the classpath. The contents of this file
> is not important.

I think 'version.lisp' is a little too generic, and having its contents
be irrelevant missing a design opportunity.

All SYS:ADD-CONTRIB essentially does is add locations to ASDF.
Therefore it would be better to use the facilities of ASDF with [ABCL's
implementation of JAR-PATHNAME][2] to search for an ASDF source-registry
configuration at a well known path within a JAR, something like
<jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf>.
 The files in this zip entry would then further specify specific ASDF
locations within the jar.  This has an advantage over the current
mechanism which just looks for *.asd files in the second level
directories, as supporting much more flexible packaging.

[2]:
http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.markdown

While I don't think we lose anything, I would like to note that
implementing this proposal would weaken the difference between an
ABCL-CONTRIB provided system and any other system packaged by ASDF.
With the proposed (on the asdf development mailing list) strengthening
of the rules and facilities for metadata in ASDF system descriptions, we
should be able to (eventually) differentiate an ABCL-CONTRIB system from
any other ASDF system if there were a valid need for this.

As part of the modification of SYS:ADD-CONTRIB, the current use of
ASDF:*CENTRAL-REGISTRY* should be re-written in favor of the explicit
configuration API exported by ASDF.

[…]

> It would be relatively easy for me to produce a patch to provide this
> feature as part of ABCL. It may be possible to provide this feature as an
> add-on, but the functions needed within the system package are private (not
> exported) so it would be difficult, if not impossible, to implement this
> feature as an add-on.

The argument that the symbols one needs in SYS are not exported makes no
functional difference to implement anything:  you can still call them.
What difficulty/impossibility do you see here?

That the symbols which are exported from SYS and EXT should be done in a
cleaner, more consistent, manner is a point I readily accept.  Ideally
over time, I wish to only external symbols in the EXT package to
constitute the API for extending ABCL.

[…]

In order to sponsor these changes to ABCL trunk, I would like to see:

1.  The build instructions for Ant and Maven must share the same
instructions.  I do not want to chase the maintenance of two build
processes.  The path of least resistance to the rest of the current ABCL
community would be to have Maven slave into the Ant build process at all
necessary points, including sharing intermediate artifact locations and
packaging instructions.

2.  SYS:ADD-CONTRIB should use the ASDF configuration API as the
mechanism to add new system locations based on the presence of a special
directory such as
<jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf>.

3.  The code in SYS:ADD-CONTRIB should initially try to locate
abcl-contrib{,-m.n.o}.jar files, falling back to searching the
classpath.  Since searching the classpath may be a potentially expensive
operation--especially if it contains locations on a remote network--we
may want to consider a runtime option and/or special variable to limit
the search strategy.

4.  Since the classpath can be modified, there should be a mechanism to
explicitly request the ABCL-CONTRIB initialization strategy be re-run.
I'm not sure the current code does this very well.

5.  We should to test the new loading strategy in a) Java Servlet
(Tomcat/Jetty) b) Google App Engine and c) OSGi containers.

Otherwise, good work!  I think with a little back-and-forth on design,
we can get something like this into ABCL in the near term.

-- 
"A screaming comes across the sky.  It has happened before, but there
is nothing to compare to it now."