Finding abcl-contrib packages inside a JAR: approach and suggestions

Sun Apr 30 14:00:08 UTC 2017

Hello,

Thank you for your comprehensive answer, much appreciated.

Mark Evenson <evenson at panix.com> wrote:
> On 4/29/17 21:48, Frederico Munoz wrote:
> > Hello,
(...)
> > Note that I need to have only one jar file and with specific
> > requirements around the main method; the above allows me to have it
> > working fine, a single  JAR which is essentially standalone.
> 
> If you could detail the requirements you have around the main method a
> little bit, that would help understand more exactly where you are
> heading.  From reading between the lines based on your [comments around
> your Watson IoT example][1], I *think* you are trying to make a WAR that
> will deploy to Jetty, so I am going to take an educated guess that that
> is what you are trying to create.
> 
> [1]:
> https://developer.ibm.com/recipes/tutorials/watson-iot-with-common-lisp/

Ahh, yes, I can see how it could appear that way, I should have been
clearer but I didn't want to add to much information; currently I'm
taking pieces from my recent articles in dW - Common Lisp and OpenWhisk
- and using the ability to write Java functions to use ABCL:

https://console.ng.bluemix.net/docs/openwhisk/openwhisk_actions.html#openwhisk_actions_java

So not a WAR but a compiled Java class with a certain main method (must
receive and return a JsonObject), which mustn't depend on things which
are not guaranteed to exist remotely.

> A couple pointers on making WAR with ABCL:
> 
> 1) [abcl-servlet][2] provides infrastructure for making an example WAR
> archive which has ABCL-CONTRIB available.  The WAR only works for Java
> Servlet containers that provide an "exploded WAR" deployment strategy,
> i.e. at deployment the WAR archive is unzipped onto the local filesystem.
> 
> [2]: https://bitbucket.org/easye/abcl-servlet

I actually read that during my investigations and it was exactly what
would latter lead me to loadSystemFile(): I can't use loadLispResource()
since the jar isn't unzipped at the server. This lead me to some initial
strange errors because locally it works since I'm running it in a place
where the directory structure supports it, and this is directly related
to your second point.

> 2) Mucking with Load.loadSystemFile() and *LISP-HOME* should not be
> necessary, as these are really meant to be internal interfaces to boot
> the Common Lisp environment.  Of course, you are welcome to experiment,
> but it would be best not to rely on these interfaces if at all possible.
> The values for the logical pathname SYS *should* be the "right way"
> forward to identify what *LISP-HOME* points to.  Logical pathnames are
> better as they easily allow us to "update" the directory with the
> runtime value without having to explicitly re-merge the whole slew of
> PATHNAME objects derived from the build-time value.

I'm using loadSystemFile() exactly because I need to work within the
constraints of an (unzipped) jar file, and after some time trying to
find how to make loadLispResources() work for files within the jar I
found
http://www.didierverna.net/blog/index.php?post/2011/01/22/Towards-ABCL-Standalone-Executables
which allowed me to have it working. *LISP-HOME* was the first thing I
found that could be used when I started looking around for an answer to
"how do I get the path of the jar file so that I can use it to load
abcl-contrib or the included packages?".

I tried finding information on the "logical pathname SYS" without
success, if you could point to the right documentation about this it
would be great.

> 3) Gradle recipes are cool.  We would like to eventually include Gradle
> support in the ABCL build mechanism as it seems the most progressive of
> current ecosystem (especially now that Groovy has matured a bit).  We
> would like to "call into" Ant for parts of the build as the current
> mechanism for controlling the artifacts and contents of 'abcl.jar' and
> 'abcl-contrib.jar' reside there and duplication of code paths would
> incur maintenance overhead.

I'm new to Java development toolchain (I used Eclipse here and there to
make some trivial things but that automates most of this), I used gradle
because it was being used by the OpenWhisk team and also because it
allowed me to quickly achieve my goal: compared with ant XML files it
was a breeze to have something that produced a single JAR file with the
lisp file in place and the ABCL classes inside. This from my perspective
which is not tied with building ABCL itself, of course. I will share all
of it in some repo of course.

> > My only remaining doubt (apart from other possible solutions which could
> > also work and be more elegant, I'm open to suggestions of course) is
> > around loading abcl-contrib.jar or more specifically the included
> > packages (JSS and JFLI specifically). I've spent some hours tracking the
> > loading mechanism and learned a bit about the process but I'm still
> > missing some pieces.
> 
> The as yet unreleased abcl-1.5.0 contains [a top-level Ant target
> 'abcl-aio.jar'][4] that creates a single archive containing both
> abcl.jar, abcl-contrib.jar, plus whatever has been added to the
> build-time <file:abcl/contrib/> directory.
> 
> [4]: http://abcl.org/trac/browser/trunk/abcl/build.xml#L521
> 
> With [abcl-aio][] we introduced a mechanism that introspects the jar
> manifest for declarations of ASDF systems to load when (require
> :abcl-contrib) is issued.
> 
> [abcl-aio]: http://abcl.org/trac/changeset/14908
> 
> We intend for 'abcl-aio' to serve as the mechanism for the
> end-user/developer to craft custom jar files.

Oh, this looks good. I will look into it. Currently I'm not building
ABCL itself, I'm using the existing jar files as dependencies (and, btw,
it seems that the ones in the Maven repository are rather old, 1.2 IIRC,
would be good to update them).

> Separately, there has been a long-standing mechanism to [include
> arbitrary code at system boot in a 'system.lisp' file][5] that could
> plausibly act as a hook for system intitialization, but I have never
> needed it to get WAR archives loaded in Java Servlet contexts.
> 
> [5]: http://abcl.org/trac/browser/trunk/abcl/build.xml#L281

Since I'm not building ABCL (and I think that going that route could
appear a bit to complex to others, but maybe I'm wrong, more on that
further down) I could use that if system.lisp is being automatically
loaded, but OTOH using loadSystemFile() is not overly complex. BTW,
system.lisp has this which seems related to the SYS logical pathname.

(SETF (LOGICAL-PATHNAME-TRANSLATIONS "sys")
      '(("SYS:SRC;**;*.*"
               "/Users/evenson/work/abcl/src/org/armedbear/lisp/**/*.*")
	               ("SYS:JAVA;**;*.*"
		                "/Users/evenson/work/abcl/src/org/armedbear/lisp/../../../**/*.*")))

> In short, if you were to build from ABCL trunk, I think you should have
> the mechanisms to do what you want.  ABCL trunk (aka 'abcl-1.5.0-dev')
> is quite stable, essentially representing the current release candidate
> for abcl-1.5.0.  I am still working on a couple things (Java 6 support
> for UIOP/RUN-PROGRAM, using ABCL-BUILD to install Ant and/or Maven via
> interactive restart, better tests for PATHNAME shennigans), but I keep
> the patches for this work separate from the ABCL trunk source until it
> passes tests.

The way I've done it this far was from a different angle: instead of
building ABCL and making changes to the ABCL build mechanism to produce
my desired outcome, I'm using the binary ABCL distribution as a
dependency, making changes during the construction of the JAR file. This
can be the wrong approach but given my lack of familiarity with ABCL's
(and Java in general) build process it seemed the most direct way
because I started from an available Java sample and went from there.

That said I will investigate using ABCL build itself to do this, using
AIO. Since I currently have it working I can showcase something based on
what I have and then do a second part using that approach.

> > I think there is something obvious I'm missing, the pieces are all there
> > (the jar handling code works fine, etc) but I can't seem to find the
> > right way to put them together.
> 
> ABCL-AIO isn't currently documented (a bug), so I would not consider it
> obvious.  [ABCL-JAR][] essentially uses the same mechanism (i.e. pushing
> PATHNAMEs to ASDF:*CENTRAL-REGISTRY*) you are prototyping here, but of
> course you are trying to get ABCL-CONTRIB to work in the first place
> here so it won't be much help for the immediate task at hand.
> 
> [abcl-jar]:
> http://abcl.org/trac/browser/trunk/abcl/contrib/asdf-jar/README.markdown

That's mildly reassuring to hear :) I generally go towards a goal by
learning what I need as I go which can lead me to some less-than-obvious
solutions. Still, I have learned quite a bit about how things work under
the hood in the process.

Thank you for your help and support!

Best regards,

-- 
Frederico Munoz

fsmunoz at sdf.org
SDF Public Access UNIX System - http://sdf.org