[Asdf-devel] startup times and initialize-source-registry

[Faré]

I have implemented support for .cl-source-registry.cache in the
faster-registry branch, and added a script
./bin/update-cl-source-registry.lisp to create or update cache files,
as in:
   ~/common-lisp/asdf/bin/update-cl-source-registry.lisp ~/common-lisp/

It does significantly speed up the scanning of the source-registry
indeed, and thus slash the startup time of cl-launch from "noticeably
slow" to "acceptable". Problem: it's only useful if you use the
install-asdf-as-module script to override your implementation's ASDF
with that from the branch, or else, the old asdf will still spend time
scanning the source-registry at startup... except the script seems to
be broken in master, unlike in the minimakefile branch. Sigh. So I
merged minimakefile into faster-registry. (Note to Robert: that's one
more thing broken in master that's working in minimakefile.)

There are still more things to speed up in starting cl-launch, though,
like the loading (and possibly upgrading) of ASDF, and (on SBCL)
priming the CLOS caches for all the ASDF classes and functions. But it
looks like they are out of my reach, and they account for most of the
.6s of startup left on SBCL.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Hi! I'm a signature virus. Copy me into your sig file and help me spread!


On Thu, Aug 28, 2014 at 11:17 AM, Faré <fare at tunes.org> wrote:
>>> Any insta-theories for where other half second comes from?
>>>
>> If you mean the second half of cl-launch's startup time, I fear it
>> might be a combination of shell and CL compilation overhead, but I
>> admit I haven't tried timing where the time is going. You're welcome
>> to investigate. At some point, I was thinking of translating cl-launch
>> to CL and having it output a bootstrap script to make itself. Then it
>> would have a "fast" mode that loads code onto the cl-launch image,
>> rather than start a new process (with maybe a different CL
>> implementation) to implement the requested software. At that point,
>> maybe Xach's buildapp is a better starting point. Or not.
>>
> The other half of cl-launch's startup time is spent in CL, and is
> definitely a performance bug:
> % time env SBCL=/bin/true cl -ip 1
> env SBCL=/bin/true cl -ip 1  0.03s user 0.02s system 102% cpu 0.045 total
> % time env CL_SOURCE_REGISTRY="/does/not/exist" cl -ip 1
> env CL_SOURCE_REGISTRY="/does/not/exist" cl -ip 1  0.43s user 0.06s
> system 98% cpu 0.494 total
>
> Inserting a few (time) forms and enabling verbose loading, I find that
> the cl:load'ing of cl-launch itself is taking most of the time, and
> the most time in that is loading then upgrading asdf, i.e. loading the
> fasl's for it. I'm not sure how to speed THAT up; ASDF is big, and not
> likely to shrink significantly while preserving backward
> compatibility. SBCL also seems to be spending some notable fraction of
> the time compiling the cl-launch code itself, though most of the time
> is the fasl loading. Conclusion: FASt Loading is only so "fast".
>
> PS: your topology.sh file has a ":" ; # ... comment as if it were a
> bilingual script, but it isn't bilingual.
> Also, the path /usr/local/bin/cl is not standard and neither is the -S argument.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> "The Army is a place where you get up early in the morning to be yelled
> at by people with short haircuts and tiny brains."
>                 — Dave Barry