[armedbear-devel] ABCL packaging ASDF system in jars (was Re: Disabling "built-in" output translation)

Tue May 17 20:24:11 UTC 2011

On Tue, May 17, 2011 at 9:47 PM, Mark Evenson <evenson.not.org at gmail.com> wrote:
> On 5/17/11 3:28 PM, Faré wrote:
> […]
>>
>> Hope this helps.
>
> I'm gonna need a bit more time to chew over the ASDF code, but thanks for
> the general direction.  I must confess that in spite of the reasonable
> looking documentation and having contributed the function translation code
> to ASDF, its whole output translation API has never really gelled in my
> understanding as a totality.  Maybe I can contribute some examples to the
> texinfo file when the fuzz in my understanding resolves a bit.
>
>> BTW, for cl-launch and XCVB, I indeed am looking for a way to create
>> bundles from compiled stuff. How do I create a jar from ABCL and a set
>> of lisp files, precompiled or not?
>
> A reasonable packaging mechanism for ABCL plus "compiled stuff" composed of
> ASDF system definitions that works well in a Java ecosystem is precisely the
> itch I'm in the process of scratching for ABCL right now.  The base unit of
> packaging in Java is the jar file, which is nothing more than a ZIP archive
> with (possibly) some Java-the-language specific metadata.  Currently, one
> can package an ASDF definition in a jar file, push the location of the asd
> file into ASDF:*CENTRAL-REGISTRY* using the [ABCL JAR-PATHNAME
> conventions][1] for which a subsequent ASDF:LOAD-SYSTEM will do the right
> thing.
>
> [1]:
> http://trac.common-lisp.net/armedbear/browser/trunk/abcl/doc/design/pathnames/jar-pathnames.markdown
>
> For instance, if you were to package up cl-ppcre installed via
> [quicklisp][10^9monkey-wants-this] in a jar file via
>
>  unix$ cd ~/quicklisp/dists/quicklisp/software && jar cfv
> /tmp/cl-ppcre-2.0.3.jar cl-ppcre-2.0.3
>
> [10^9monkey-wants-this]: http://www.quicklisp.org
>
> one could subsequently enable an ASDF controlled load of this system via
>
>  CL-USER> (push "jar:file:/tmp/cl-ppcre-2.0.3.jar!/cl-ppcre-2.0.3/"
> asdf:*central-registry)
>
> When loaded, ASDF will compile the system to the user cache.
>
> Some problems here that I'm working through:
>
> 1)  You can't immediately load FASL out of the jar.  In my current hackish
> way—i.e. without comprehending your advice yet—one has first ASDF compile
> the system with output translations disabled, and then
>
>  (defmethod asdf:perform ((o asdf:compile-op) (c asdf:cl-source-file)))
>  (setf (asdf::output-translations) '((t t))))
>
> in the target JVM to load the ABCL FASLS from the jar.
>
> 2) One has to specify the absolute path on the local filesystem (or
> potentially via a URI) for the jar, which makes things a bit fragile in the
> typical Java ecosystem usage which shuffles jars around like win32 DLLs (or,
> to be fair, Unix dynamic libraries) relying merely on the filename to keep
> things straight.  My current insight into a way around would be to define
> another PATHNAME extension in ABCL for CLASSPATH entries, i.e.
> "classpath:cl-ppcre-2.0.3.jar!/cl-ppcre-2.0.3/" would refer to the named jar
> in the JVM CLASSPATH if it exists.

I think this is wrong. Not everyone here is Java-savvy, so I'm going
to explain some things you probably already know:

a. CLASSPATH (the listing of directories and jar files where to look
for classes) is not a runtime concept; it's only a variable
(environment variable or command line switch) used to init the JVM's
default class loader. Once the JVM is started, CLASSPATH doesn't make
any more sense.
b. At runtime there's a hierarchy of class loaders, that, without
going too much in detail, define a sort of limited virtual filesystem.
Resources in that filesystem might come from plain files, from
contents of jars, from the network, from memory... So,
classpath:foo.jar!/bar does not make much sense: the fact that bar
comes from foo.jar is no longer important (and possibly not even known
anymore) inside the virtual filesystem. This is not just theory, I can
cite at least two examples: ABCL's very own functions compiled at
runtime and loaded from memory, and anything running on JBoss >= 5,
where applications are loaded with a classloader that does not expose
jar files in any way. So, I would suggest simply
classpath:/foo/bar/baz and leave it to the classloader to go and find
/foo/bar/baz. This is precisely how the Spring framework does it, for
example[1]. What you lose is the ability to enumerate the contents of
the "file system": you can only ask for a resource by name and get
back a stream if it exists. But I don't think we currently need the
capability to enumerate.

[1] http://static.springsource.org/spring/docs/2.0.1/reference/resources.html#resources-resource-strings

> 3)  ABCL [has a bug][#149] that currently prevents ASDF systems located in
> the top-level entry of a JAR from being accessed.
>
> [#149]: http://trac.common-lisp.net/armedbear/ticket/149
>
> 4)  The extremely nice use of [JSS][jss] and [ABCLD's slight
> modification][abcld] of ASDF to also refer to jar files to dynamically load
> into the JVM probably needs to be rewritten, otherwise we run into the
> situation whereby we have jars (i.e. the packaged Java code) within the ASDF
> packaged jar which A) needs changes within ABCL to completely work and B)
> would be rather inefficient in that the naive implementation  each request
> for a new entry in the JAR within a JAR would require a complete "reseek"
> through the enclosed ZIP file.

Another benefit of going through classloaders is that they (at least
the default ones) already cache resources and don't reseek the jar
every time. And I wouldn't put Java dependencies as jars inside
another jar: either you should distribute them as separate jars, or
you should "sum" them to a single fat jar or "uberjar" (the latter is
typically generated by the Clojure build tool, Leiningen:
<https://github.com/technomancy/leiningen/blob/stable/TUTORIAL.md>).

> [jss]: http://code.google.com/p/lsw2/source/browse/#svn%2Ftrunk%2Fjss
> [abcld]:
> http://code.google.com/p/abcl-dynamic-install/source/browse?repo=abcld
>
> 5)  A fear of mine: if I enable all this, I presume that people would start
> going around creating 'abcl.jar' files with different inclusions of
> different ASDF packagings.  Without a real smart dynamic introspection
> system that essentially solves the problems in JVM's classpath we would end
> up with, at the minimum, a rather hostile situation for the end user.  "Put
> 'abcl.jar' in your classpath."  "I did, and it still didn't load Maxima."
>  "Well, what's the checksum of your abcl.jar?" "c48d359a23ee"  "Oh, you need
> 846f78c279cb".  Ideally, I would like to come up with a mechanism that would
> require that 'abcl.jar' come from "official" ABCL packaging, but would
> somehow be able to introspect the JVM classpath to include ASDF definitions.

Why should everything go into abcl.jar? Applications and libraries
should go inside their own jars. It's the same as with, e.g.,
clisp.exe.

Alessio