[armedbear-devel] Autoloaders: status report and a question

[Erik Huelsmann]

Hi all,

Sorry for the bumpy ride while I reorganized the auto loader system.

Thanks for taking the time to respond to my earlier questions on auto
loaders. This is the time to explain what I did with your answers (and what
I did in general).

To reiterate what this is all about: the autoloader is an important part of
the start up efficiency and internal file dependency resolution of ABCL: it
establishes function and macro bindings on symbols without loading the
actual code to bind to. Instead, loading the FASL is being delayed until
some part of the system (or a user) needs it. There are two variants of
autoloaders: Java class autoloaders and lisp (fasl) autoloaders. Their
purpose is the same and they operate roughly the same way, but one operates
by forcing a class to be loaded while the other causes a FASL to be loaded.

Most parts of the system don't know about autoloaders and simply evaluate
code. The autoloaders take special action when evaluated: their purpose is
not to execute the intended action, but instead, they load the fasl in
which the actual code is located. Then, when the actual code is loaded, the
function (or macro) call is forwarded to the loaded code. The FASL loader
process replaces the content of the function symbol slot with the true
function so the autoloader gets garbage collected after replacement.

This entire system works pretty well, that is: *if* you know which symbols
you have to install an autoloader on. Before my changes, we didn't know
which symbols we had to install autoloaders on. Instead, we maintained a
list of symbols by hand through trial and error. When a change caused total
havoc, you'd look for the cause and - hopefully - debugged the problem to a
missing autoloader. That's when you would add an autoloader to
autoloads.lisp. That works as long as our only purpose is to build ABCL.
However, we actually intend for ABCL to be used by our users.

Users may not be aware of what autoloaders do for them and choose to start
calling some random function in the AMOP definition. Since we hand-maintain
a list of functions to be decorated with autoloaders, that probably doesn't
work. So, in order to provide more stability for our users we need to
decorate all functions with autoloaders.

What's more: having all functions decorated with autoloaders -if done well-
should also help ABCL development quite a bit: the chances of a change
causing havoc because a non-decorated function is being used have been
reduced to next-to-none.

What then did I do?

The change involves a few steps:

1. Extraction of symbols to be decorated
=============================

The file compiler now generates (when compiling our own system, that is)
two extra output files: basename.funcs and basename.macs. These files
contain a single s-exp with all the symbols which have been DEFUNed or
DEFMACROed respectively. With this information, we can generate decoration
information.

Analysis of the extracted symbols leads to the conclusion we can't assign
an autoloader to each and every extracted symbol for 2 reasons:
 a. There is a built-in function associated with the symbol to help
bootstrapping
 b. There are multiple DEFUNs or DEFGENERIC/DEFMETHODs


2. Determination of what to decorate symbols with
====================================

Autoloaders - at this time - can only be used to load a single FASL. So,
when a symbol is in multiple files, we need to choose which file it load
the FASL for (and hence not the others).

Most symbols which occur in multiple files are DEFMETHODs which is
perfectly fine. Most symbols with double locations stem from the
EXTENSIBLE-SEQUENCES implementation. We want to exclude those FASLs, which
reduces the number of multi-homed symbols to only a few.

All single-homed symbols excluding those which have bootstrap functions
attached can be decorated with autoloaders.

3. Generate code to install the autoloaders
===============================

We just used to have autoloads.lisp (and Autoload.java). But that file will
contain only the manual bits which can't be auto-generated. So there's now
a second file autoload-gen.lisp. This file is being generated during the
build and populated with all symbols which need an autoloader. Today's
autoload.lisp has been stripped to the bare minimum of AUTOLOAD and
AUTOLOAD-MACRO forms.

With the stripped autoload.lisp and the fact that the autoloaders are
bootstrapping helpers, I had to take additional measures to help us
bootstrap again. The result is autoloads-gen.lisp in the source directory.
It is loaded at build time to help bootstrapping. It's an unmodified
version I took from the build directory:
build/classes/org/armedbear/lisp/autoloads-gen.lisp - the one which is
generated on each build.


4. Exporting the right symbols
======================

Not a problem related to autoloading per se, but definitely related to the
fact that we don't load all fasls at boot time: autoload.lisp - as the
canonical location being loaded at startup - had grown to be a listing of
EXPORT forms. Having EXPORT forms and actual code widely apart is
counter-intuitive: to the occasional developer it's inexplicable how some
symbols become exported and some don't. (We had files with additional
EXPORT forms.)

So, when the above was "solved", I created another facility which collects
all the symbols in the second argument of EXPORT throughout our source
tree. Those symbols are added to the autoload-gen file and the EXPORT forms
executed at start up as well. Even though the developer still doesn't know
what's going on, the system should be working transparently.


All of the above has happened over the last week(s) - last week, mostly.


Then there is the question:

As it turns out, the macroexpansion for DEFSTRUCT happens in a way such
that the file compiler can't detect the slot accessors as being defined as
functions. The advantage is that we're *not* seeing a whole slew of symbols
to be attached to autoloaders and hence limiting the number of autoloaders
(the contructor function *is* detected).

Now: what do others think: should we depend on the constructor being called
before any of the accessors to make sure the autoloaders have been run? Or
should we attach an autoloader to each accessor function just to be sure?
(This requires small changes to the file compiler and defstruct macro
expansion.)


Bye,

Erik.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/armedbear-devel/attachments/20120818/1b6bd5e3/attachment.html>