[armedbear-devel] Idea for new FASL format: smaller, faster

[Erik Huelsmann]

>> 2. Because lisp functions have become Java functions, we can't
>> directly expose them to the Lisp world anymore. This basically creates
>> the option to have an 'internal function signature' and an 'external
>> function signature'. SBCL has this: XEPs -- eXternal Entry Points.
>> These entry points into a function sort the arguments into the right
>> order before calling the internal entry point. Code which is compiled
>> into the same fasl pre-sorts the arguments at compile time (when
>> possible) and calls the internal entry point -- eliminating the need
>> to sort keyword parameters.
>
> That's awesome. To clarify, do you intend these XEPs to be method
> overloads in the same FASL-class?

Yes: the XEPs would be methods accepting an array argument, just like
we accept now. The XEP would use the Closure.processArgs() code to
figure out the optional and keyword arguments, just like today. Unlike
today, the XEP would then forward the call unpacking the argument
array into individual parameters and call the internal entry point.


[...]

>> c. We need to find a way to correctly handle the interaction between
>> the successive IN-PACKAGE, DEFPACKAGE, EVAL-WHEN, etc, forms appearing
>> in the input file and the initialization of fields in the resulting
>> class file.
>
> Off the top of my head (but the standard might imply otherwise) the
> only problem is IN-PACKAGE (and DEFPACKAGE) and that can be solved by
> 1) adding a static initializer to the fasl-class that pre-installs all
> the packages and 2) serializing all symbols explicitly as
> package::symbol. Everything else (mostly EVAL-WHEN combined with stuff
> that affects the reader) is handled by isolating the reader used to
> parse serialized stuff in the class from the reader used to read forms
> in the fasl, and I believe it's already like that - isn't it?

Yes, the reader fasl reader is indeed separated from the compile-time
reader. One thing that got us in the past when serializing symbols was
EXPORT, which was first evaluated and then serialized. When reading
the resulting fasl, the export command tries to export a non-existing
symbol.

However, serializing all symbols with their package prefix breaks a
use case I know of to be actually in use - which may not be explicitly
supported by the CLHS: when loading our current fasls while having a
different current package than during compilation, we simply load
everything into the new current package. This is because the
preconditions before loading are assumed to be the same as the
preconditions before compilation.

What I'm thinking now is that I might try to compile the top level
forms into an initialization function in the class. That
initialization function could interleave class field initialization
with "lisp world initialization". That way, I think, we should be able
to get our dependencies ordered correctly.

>> d. We need a way to expose the external entry points to the lisp world.
>
> We could lazily construct (by generating bytecode at runtime) a
> LispFunction instance that calls the right method. It would be
> generated the first time someone calls, or otherwise references, a
> function in the fasl, and then associated with the symbol as its
> symbol-function. Hopefully these runtime-generated classes will be in
> much smaller numbers than if we compiled each and every function to a
> separate class. Additionally with invokedynamic the LispFunction will
> only need to be generated when the function is reified (basically the
> first time someone directly reads a symbol's function slot), while
> regular function calls could be directed to the target method.

This is one method. The other - which might have roughly equal
performance characteristics - that I was thinking about is to create a
class which uses introspection to look up the right method to call.
The initial introspection lookup has bad performance drawbacks.
However, after the initial lookup, I've been told, the performance
drawbacks have been fixed for quite some time (Java 1.4?).

That way, we would not even need to increase the number of classes in
the system when the number of entry points into the fasl grows. The
same approach could apply when a function reference is to be returned:
the function reference could simply be an encapsulating function
object.


Thanks for your comments!


Bye,


Erik.