Proposal to improve the loading of the abcl-contrib dependency.

Sat Aug 1 10:00:40 UTC 2015

Mark,

   I have one additional issue to add to the configuration discovery
process.  The original concept of using a
/META-INF/abcl-contrib/version.lisp file accounts for web based classpaths
while the *-conf.d pattern does not due to the fact that it is often not
possible to get a list of files for web based classpaths. To accommodate
this a default configuration file list would also be needed, such as
/META-INF/abcl-config-sources.lisp . I am aware that this creates a huge
potential for conflict as various libraries could each contain this file,
but the web deployment process would need to ensure that all of the
configuration files in the *-conf.d are included in this file when the
classpath will be accessed remotely but not from within a jar.

Best Regards,
  Ralph Ritoch

On Sat, Aug 1, 2015 at 4:20 PM, Ralph Ritoch <rritoch at gmail.com> wrote:

> Mark,
>
>    I have reviewed your suggestions and they are certainly progress in the
> right direction.  I have two concerns, the first is with the path-spec for
> locating configuration files.
>
> jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf
>
> It generally considered a bad practice for two libraries to contribute to
> the same package and while the META-INF folder falls into system dependent
> functionality I believe a better pathspec would be accommodate this to
> "namespace" the configuration files. I'm aware that this isn't the standard
> practice, but if the namespace is properly named it is will be easier to
> track configuration files back to the library they came from.
>
> jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*/*.conf
>
> My second concern is the method of configuring the contrib search. The
> default should certainly support the "legacy" search methods and fallback
> to the classpath, but to avoid imposing limitations on future systems I
> believe a better algorithm would be to allow registration of custom
> configuration search handlers such as having a
> sys:*configuration-providers* which contains and ordered list of
> configuration provider functions. In the case of web based apps, this opens
> the potential to cache remote configuration files.
>
> I am looking forward to the progress of this, expanding this functionality
> beyond the configuration of one specific library could greatly improves the
> application lifecycle for ABCL applications.
>
> As for the maintenance of the maven build system, the pom's I submitted
> compile the java files, and builds the jar, but is calling ant targets.
> While ant can call maven it still comes down to the key issue that maven
> doesn't pick-up a jar created by an external process. Maven central
> requires a source release, a binary release, and javadoc release, which all
> must be GPG signed. The complexity of trying to make maven a slave to ant
> is comparable to the complexity of the current "manual" system and I
> believe it will lead to the same situation you have now where no one does
> releases to maven. While there may be room for improvement with the maven
> system, there's normally little to no maintenance needed, once it is
> properly configured. The only thing that must be changed are the version
> numbers, and any dependencies. If you can make maven a slave to ant such
> that it will deploy to central and local repositories, that would be fine,
> but other than creating a custom maven plugin I don't know any way it would
> be possible without allowing maven to package and sign the jars.
>
> I was advised that it would be possible to do away with the ant system all
> together. What was suggested to me by java developers was to have the java
> sources built in one module, which has a post-build script to compile the
> lisp files, and a second module which uses the target (generated) files of
> the first module as it's resource path. Calling ant is much easier since
> the ant system is already built, but it is possible to completely deprecate
> ant in favor of maven.
>
> Best Regards,
>   Ralph Ritoch
>
> Date: Mon, 27 Jul 2015 10:42:30 +0200
> From: Mark Evenson <evenson at panix.com>
> To: armedbear-devel at common-lisp.net
> Subject: Re: Proposal to improve the loading of the abcl-contrib
>         dependency.
> Message-ID: <55B5EEF6.5030601 at panix.com>
> Content-Type: text/plain; charset=utf-8
>
>
>
> On 2015/7/26 14:29, Ralph Ritoch wrote:
> > Hi,
> >
> >    I have run into an issue with making executable abcl jars when the jar
> > depends on, and provides, abcl-contrib. As far as I can tell abcl looks
> for
> > abcl-contrib in the following locations.
> >
> > 1. Looks for a jar named abcl-contrib.jar in the classpath determined by
> > probing the classloader for the jars it provides.
> > 2. Looks in the directory of the jars in the classpath for a file named
> > abcl-contrib.jar
> >
> > Neither of these solutions work if the currently executing jar includes
> > abcl-contrib and isn't named abcl-contrib.jar.
>
> The contrib file may also be named via a versioning suffix something
> like 'abcl-contrib-m.n.o.jar'.
>
> Some background on the current design: abcl-contrib.jar was explicitly
> separated from abcl.jar in order to have a clean separation between the
> code providing an ANSI Common Lisp, and the ABCL-specific extensions to
> this.  The goal was to bound the size of the abcl.jar to some reasonably
> constant (over time) amount.  In addition, function symbols in abcl.jar
> are subject to an "autoload" mechanism which delays the loading of their
> implementation until their first invocation.  Consequently, the Lisp
> code in abcl.jar is loadable by ABCL-specific SYS:LOAD-SYSTEM-FILE
> (which is used by the ANSI REQUIRE mechanism), while abcl-contrib.jar
> may assume the full ANSI environment so may be addressed via CL:LOAD.
>
> > The following code proves that it is relatively easy to add this
> capability.
> >
> > (in-package 'sys)
> > (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref
> > (java:jcall "getURLs" (sys:boot-classloader)) 0))))
> >
> > This code properly loads abcl-contrib when the executing jar provides
> > abcl-contrib, while (require 'abcl-contrib) does not.
>
> A better implementation would be to add this capability to the mechanism
> used by (require :abcl-contrib) which corresponds to directly to [code
> contained in abcl-contrib.lisp][1].
>
> [1]:
>
> http://abcl.org/trac/browser/trunk/abcl/src/org/armedbear/lisp/abcl-contrib.lisp
>
> > My proposal is to add a file named abcl-contrib/version.lisp to the
> > abcl-contrib.jar, possibly in the /META-INF directory to avoid any
> > potential conflicts with java, and to have sys:find-contrib look for this
> > file in the resources provided by the classpath. The contents of this
> file
> > is not important.
>
> I think 'version.lisp' is a little too generic, and having its contents
> be irrelevant missing a design opportunity.
>
> All SYS:ADD-CONTRIB essentially does is add locations to ASDF.
> Therefore it would be better to use the facilities of ASDF with [ABCL's
> implementation of JAR-PATHNAME][2] to search for an ASDF source-registry
> configuration at a well known path within a JAR, something like
> <jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf>.
>  The files in this zip entry would then further specify specific ASDF
> locations within the jar.  This has an advantage over the current
> mechanism which just looks for *.asd files in the second level
> directories, as supporting much more flexible packaging.
>
> [2]:
>
> http://abcl.org/trac/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.markdown
>
> While I don't think we lose anything, I would like to note that
> implementing this proposal would weaken the difference between an
> ABCL-CONTRIB provided system and any other system packaged by ASDF.
> With the proposed (on the asdf development mailing list) strengthening
> of the rules and facilities for metadata in ASDF system descriptions, we
> should be able to (eventually) differentiate an ABCL-CONTRIB system from
> any other ASDF system if there were a valid need for this.
>
> As part of the modification of SYS:ADD-CONTRIB, the current use of
> ASDF:*CENTRAL-REGISTRY* should be re-written in favor of the explicit
> configuration API exported by ASDF.
>
> [?]
>
> > It would be relatively easy for me to produce a patch to provide this
> > feature as part of ABCL. It may be possible to provide this feature as an
> > add-on, but the functions needed within the system package are private
> (not
> > exported) so it would be difficult, if not impossible, to implement this
> > feature as an add-on.
>
> The argument that the symbols one needs in SYS are not exported makes no
> functional difference to implement anything:  you can still call them.
> What difficulty/impossibility do you see here?
>
> That the symbols which are exported from SYS and EXT should be done in a
> cleaner, more consistent, manner is a point I readily accept.  Ideally
> over time, I wish to only external symbols in the EXT package to
> constitute the API for extending ABCL.
>
> [?]
>
> In order to sponsor these changes to ABCL trunk, I would like to see:
>
> 1.  The build instructions for Ant and Maven must share the same
> instructions.  I do not want to chase the maintenance of two build
> processes.  The path of least resistance to the rest of the current ABCL
> community would be to have Maven slave into the Ant build process at all
> necessary points, including sharing intermediate artifact locations and
> packaging instructions.
>
> 2.  SYS:ADD-CONTRIB should use the ASDF configuration API as the
> mechanism to add new system locations based on the presence of a special
> directory such as
> <jar:file:SOME.jar!/META-INF/common-lisp/source-registry-conf.d/*.conf>.
>
> 3.  The code in SYS:ADD-CONTRIB should initially try to locate
> abcl-contrib{,-m.n.o}.jar files, falling back to searching the
> classpath.  Since searching the classpath may be a potentially expensive
> operation--especially if it contains locations on a remote network--we
> may want to consider a runtime option and/or special variable to limit
> the search strategy.
>
> 4.  Since the classpath can be modified, there should be a mechanism to
> explicitly request the ABCL-CONTRIB initialization strategy be re-run.
> I'm not sure the current code does this very well.
>
> 5.  We should to test the new loading strategy in a) Java Servlet
> (Tomcat/Jetty) b) Google App Engine and c) OSGi containers.
>
> Otherwise, good work!  I think with a little back-and-forth on design,
> we can get something like this into ABCL in the near term.
>
> --
> "A screaming comes across the sky.  It has happened before, but there
> is nothing to compare to it now."
>
>
> On Sun, Jul 26, 2015 at 8:29 PM, Ralph Ritoch <rritoch at gmail.com> wrote:
>
>> Hi,
>>
>>    I have run into an issue with making executable abcl jars when the jar
>> depends on, and provides, abcl-contrib. As far as I can tell abcl looks for
>> abcl-contrib in the following locations.
>>
>> 1. Looks for a jar named abcl-contrib.jar in the classpath determined by
>> probing the classloader for the jars it provides.
>> 2. Looks in the directory of the jars in the classpath for a file named
>> abcl-contrib.jar
>>
>> Neither of these solutions work if the currently executing jar includes
>> abcl-contrib and isn't named abcl-contrib.jar.
>>
>> The following code proves that it is relatively easy to add this
>> capability.
>>
>> (in-package 'sys)
>> (sys:add-contrib (make-pathname :defaults (java:jcall "toString" (svref
>> (java:jcall "getURLs" (sys:boot-classloader)) 0))))
>>
>> This code properly loads abcl-contrib when the executing jar provides
>> abcl-contrib, while (require 'abcl-contrib) does not.
>>
>> My proposal is to add a file named abcl-contrib/version.lisp to the
>> abcl-contrib.jar, possibly in the /META-INF directory to avoid any
>> potential conflicts with java, and to have sys:find-contrib look for this
>> file in the resources provided by the classpath. The contents of this file
>> is not important.
>>
>> This change makes it possible to include the abcl-contrib "dependency"
>> directly inside the jar application that is currently running, regardless
>> of the name of the jar file. Therefore, applications which make use of
>> abcl-contrib can be distributed as a single jar.
>>
>> For maven users, this means that their uberjars will work by just
>> including abcl and abcl-contrib as dependencies.
>>
>> For ant users they would probably need to extract abcl-contrib and
>> include the extracted files in their generated jar which I believe can be
>> fully automated.
>>
>> It would be relatively easy for me to produce a patch to provide this
>> feature as part of ABCL. It may be possible to provide this feature as an
>> add-on, but the functions needed within the system package are private (not
>> exported) so it would be difficult, if not impossible, to implement this
>> feature as an add-on.
>>
>> I would like your feedback on this issue. If this is a feature you want,
>> or if you have any specific objections to including this feature in ABCL,
>> please let me know.
>>
>> Best Regards,
>>    Ralph Ritoch
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/armedbear-devel/attachments/20150801/163701a7/attachment-0001.html>