[asdf-devel] Faster loading for deployed systems?

[Erik Pearson]

Hi Faré,

On Tue, Apr 24, 2012 at 8:02 AM, Faré <fahree at gmail.com> wrote:
> Dear Erik,
>
>> I thought I'd report back briefly on some results.
>>
> Thanks for trying. Can you share your code?

I'd be glad to -- perhaps after a bit more tinkering so that at least
the changes are "switchable" via a global flag.

>
>> First, I spent some time attempting to implement "perform with session
>> state". This is the idea that when working with ASDF, it would be useful to
>> be able to invoke or create specific behavior for my use case and also to be
>> able to capture work session state without relying on globals. This was
>> partially successful, but I broke something and ran out of patience. ASDF is
>> some wonderful code, but takes a bit to grok it all.
>>
> In the past, I've slightly modified ASDF
> to make it easier for POIU to do its job.
> If there's some interface I can expose
> so your hack can be an extension of ASDF
> rather than a wholesale replacement of its functionality, I'll be glad.
>
>> So I took the crude approach of just installing a global and using it to
>> accumulate loaded files from within a perform. I created an .asd file which
>> defined the dependencies I wanted the image to have. The loaded it with
>> load-system. Then at the end of load-system I inserted code to save a
>> structure containing a list of the *defined-systems* as well as a list of
>> the loaded files (in order of loading).
>>
> NB: POIU also has a mode to record and replay a list of accumulated files.
>
> Am I correct that these are the things you're trying to do?
> (1) load all the dependencies

yes

> (2) record what are the things loaded so far

yes

> (3) load your application
> (4) when you refresh your application, skip the recorded dependencies.

Basically, yes. I would state, though, that I want the ASDF to work
like the primary contract of REQUIRE. That is, this clause from CLHS:

require tests for the presence of the module-name in the list held by
*modules*. If it is present, require immediately returns.

I think that ASDF's default mode of grokking the filesystem for
changes is what I want in a development environment.

However, it is not what I want for systems that I'm just depending on
and (not necessarily) modifying. It should compile those if needed,
and load those just one time.

And it it not what I want in a system that is deployed, because the
source code for loaded modules will never change, and if I need to
work some patches in or load some runtime functions, that will be
under application control

I realize that I'm expecting ASDF to magically behave differently when
developing code, to discover changed files and recompile and reload
them. Thus the need for a global flag to tell ASDF its operating mode.

One culprit, it seems, is REQUIRE. Otherwise, ASDF wouldn't do
anything in a normal system without ASDF specific calls. REQUIRE is
not used all that much in Lisp code, not nearly as much as an import
in other languages which depend on it. And I would partly blame the
big confusion between PACKAGE, DEFSYSTEM, and REQUIRE -- it is kind of
a mess. Still, REQUIRE does pop up in library code, and it is a very
easy to use and easy to understand way of both expressing a dependency
and possibly loading your toplevel system when developing, deploying,
or running code ad-hoc. And it does sorta map from modules to systems,
albeit with no facility for dependencies.


Another goal, though, is to reduce the noise in the generated image.
If the image is created after ASDF has been doing its work, it is not
necessarily suitable for deployment. Maybe the target application
needs ASDF, maybe it doesn't. Plus I just get nervous with an image
that has just been worked on (Lisp has enough moving parts as it is.)
Having the load plan, or rather the plan implementation code,
recorded, saved, and loaded into a fresh image would allow the
omission of ASDF, its dependencies, or any other system-building code
and data structures. The target image might not even have a compiler
or code loading functionality. On the other hand, it might be
desirable to include ASDF and initialize the ASDF system database with
some or all of the loaded systems marked as not-reloadable.


>
> If so, then I propose you use the functionality I added to 2.20.21
> for the measly cost of 12 lines of code, registered-systems and :force-not:

Aww, you shouldn't have :)

>
> (asdf:load-systems '(dependency-a dependency-b)) ;(1)
> (defparameter *recorded-dependencies* (asdf:registered-systems)) ;(2)
> (asdf:load-system :my-application) ;(3)
> (asdf:load-system :my-application :force-not *recorded-dependencies*) ;(4)
>
> If you want more fine-grained control, you could provide an override
>   defmethod operation-done-p :around
>
> If you like my :force-not feature, I'd appreciate you write
> a test case in the asdf test suite.
>
>> It was about 2.5 seconds to traverse normally, and about 0.5
>> with this optimization for a require after the first one (of course for a
>> deployed system not development.)
>>
> Can you confirm it works well with my proposed solution?
>
>> In any case load time is not my only interest.  It is also a desire to be
>> able to use ASDF more effectively for more use cases.
>>
> Same here.

I really appreciate the responses and the new code to play with. That
really is exceptional. I'll send some feedback later, hopefully today,
but I have a message queue processing system to build and a web site
to get up :(

Erik.

>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> It is a miracle that curiosity survives formal education.
>        — Albert Einstein