[armedbear-devel] In-memory compilation status

[Alessio Stalla]

On Tue, Sep 8, 2009 at 1:05 PM, John Pallister<john at synchromesh.com> wrote:
> Hi Alessio, thanks for your quick response.
>
> So, from reading your message the options would seem to be:
>
> 1. Fix COMPILE so it no longer produces "unexpected behaviour". In
> what way was it (mis-)behaving, and how was this unexpected?

Sorry, I wasn't clear enough. COMPILE behaves differently from
COMPILE-FILE as per the CL standard, to the point that (load
(compile-file "file.lisp")) might not have the same effects as
something like (loop :for x :in (read-all-forms-in "file.lisp") :do
(funcall (compile `(lambda () ,x)))) - I don't remember the exact
details, but I explicitly asked in comp.lang.lisp and I was advised
against using COMPILE for that sort of thing. (the loop is only
pseudo-code, just to make an example). Of the two, COMPILE-FILE is the
one that is closer to LOADing the uncompiled source, so I chose that
for uniformity, that way evaluate(script) behaves the same as (LOAD
script) and compile(script).evaluate(), modulo compilation of course.

> 2. Alter COMPILE so it just uses the some code as COMPILE-FILE, only
> with output to a byte stream instead of a file stream. Is this what
> you were doing? If so, how did you get on?

No, IIRC COMPILE and COMPILE-FILE both use lower-level functions and
it's those functions that presently require temporary files for some
of their work. I actually just read some parts of the compiler sources
and discussed a couple of things with Erik, but never got to the point
of modifying the compiler.

> 3. Alter COMPILE-FILE so that it doesn't use files for local
> functions. (This would be required for 2 above, I guess.) From your
> comments, this sounds like it might be tricky. Are the files just used
> as a cache, or are they re-loaded again at a later date?

Heh, it depends. When COMPILE-FILE is called, files for local
functions are kept, because you'll need them when you'll load the main
FASL. When COMPILE is called, they are removed after compilation, as
they are only used when loading the compiled function, before COMPILE
returns. This again is from memory, but I'm pretty sure about it.

> 4. Add some sort of "VFS factory" that can provide either the real
> file I/O calls or some "in-memory JAR" thing, and then modify
> COMPILE-FILE to use that. This would imply a performance penalty for
> the current "use real files" case due to the additional layer of
> indirection.

Yes, but only for the compiler; I don't think this is much of a
problem performance wise.

> This boils down to "remove the need for files during compilation" vs.
> "remove the need for a file system during compilation". Obviously one
> would be much easier than the other.
>
> It sounds like you and I would be interested in an ABCL that didn't
> rely on files for compilation, but I don't know whether that would be
> considered a Good Thing by everyone else.

I believe at least Erik wanted this, too. Generally I don't think it
has negative side effects - it would be just a matter of adding new
possibilities, not removing anything.

> Re: Clojure, it looks OK, but I'd be happy to stick with good old Lisp
> for now (especially since my JVM target has no threads as well as no
> file system, so all its concurrency benefits are moot). And having to
> hack Java to get ABCL to work is OK, as long as I'm on the right path
> (and not inadvertently forking the project).

Hehe I was only half serious about Clojure. As always, the right thing
is to use the tool that is best suited for the job. Still, if ABCL has
a problem that makes it not be a viable option in certain cases, I'd
like to remove the problem if it can be done (and if it doesn't
introduce worse problems, of course).

> P.S. I'm talking about Google App Engine, in case anyone hasn't guessed yet.

Interesting, since GAE is one of the platforms ABCL is advertised to work on ;)

Bye,
Alessio