[armedbear-devel] In-memory compilation status

[Alessio Stalla]

On Tue, Sep 8, 2009 at 2:19 AM, John Pallister<john at synchromesh.com> wrote:
> Hello list,
>
> I'm interested in trying to get ABCL running (and preferably REPL-ing
> & compiling) on a certain cloud computing platform that has a JVM with
> no access to a file system.
>
> The Wiki page for JSR-223 and the sample code both mention that
> "compilation using the runtime compiler has been removed due to
> inconsistencies with evaluation and file-based compilation", since May
> '09. Browsing the mailing list archives suggests that people have been
> working on removing the need for temporary files since then, but I
> can't find any mention of "inconsistencies". The thread "Save-image
> w/Serialization - progress report" from April ends with Sig. Stalla in
> the process of implementing a byte-array-output-stream, then
> silence...

Hello John,

Sig. Stalla sounds pompous, you can call me Alessio ;)
Those you cite are really 2 separate issues.

1. For the JSR-223 part, it used to use the cl function COMPILE to
compile code using the runtime compiler. However its behavior differs
from COMPILE-FILE in ways that could result in unexpected behavior, so
now the ABCL script engine uses COMPILE-FILE on its own, by saving the
code to compile to temporary files. This could be fixed by using a
primitive that works like COMPILE-FILE but is able to read from any
stream, included from a string in memory. I don't exclude such a thing
already exists, and only needs to be polished and made public.

2. However in general, JSR-223 or not, the ABCL compiler currently
always uses files at least when compiling local functions, even if you
use the runtime compiler. That is what you see in the serialization
thread. Changing this is more problematic, since it probably requires
changing the way compiled Lisp code is loaded and the instructions
emitted by the compiler to load compiled local functions, and I'm
afraid also the order in which the compiler does some things (wild
guess here!). I'm also interested in this change, but I don't have
much time to work on it.

> Can someone comment on the state of runtime compilation, preferably
> entirely in-memory without reference to a file system?

> Basically I'd like to be able to compile stuff to a byte array, store
> the array as a blob in a memcache and/or other datastore, then pull it
> out again as required and load it via a custom classloader. Is this
> doable, or would it be easier to port ParenScript to Clojure (which is
> currently my Plan B)?

A relatively quick fix would be to implement a virtual file API that
writes either on disk or in memory (in a dummy, virtual filesystem).
The compiler would be only changed slightly to use the more high-level
API instead of direct file access. Primitives to load compiled code
would need to look at some variable to determine where to load code
from. The virtual filesystem could be exposed to the user so it could
be persisted & restored across sessions.

I think this approach is not enough to solve the problems with
serialization (because the virtual fs needs to be manually
saved/restored, since loaded code does not know where it is loaded
from), but it's probably ok for situations like yours. Note that you'd
have to use COMPILE-FILE and not COMPILE with the virtual fs, in order
to have a known entry point (the virtual file name) when loading code
later. Also maybe a finer-grained control on the virtual fs could be
given to the user, so you'd be able to decide the life span of the fs.

If others have ideas in this regard, please share. I'd like to avoid
Plan B (having users resort to the competition :D)

Bye,
Alessio

> Cheers, and it's great to see the ABCL project still going strong!
>
> John :^P
>
> P.S. If anyone is going to be in Hamburg at the weekend and wants to
> chat about this stuff, that'd be great!
> --
> John Pallister
> john at johnp.net
> john at synchromesh.com
>
> _______________________________________________
> armedbear-devel mailing list
> armedbear-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel
>