[asdf-devel] Fwd: Faster loading for deployed systems?

Sun Apr 22 23:23:42 UTC 2012

HI Faré,
First, thanks for your excellent work on this and other projects.

On Sun, Apr 22, 2012 at 1:47 PM, Faré <fahree at gmail.com> wrote:

> Dear Erik,
>
> > I am trying to implement a method of using a "batteries included" lisp
> image
> > (ccl, linux, 64bit) which can be used for running a variety of lisp
> tasks in
> > a batch mode. I chose to pre-load the image because it was just too slow
> to
> > load the individual fasl files at run time, either via asdf or through a
> > generated load-list.
> >
> At ITA, we also deliver applications as dumped images with lots of systems.
> But I admit that during incremental development,
> we recompile files individually with SLIME
> rather than load entire systems with ASDF.
>

I have a bad habit of just requring or load-system'ing whatever I'm working
on. I sometimes have mutliple changes that i want to get loaded.

>
> > I'm using ASDF to load a small system at run-time, but finding that even
> > with small system (say, one file), performance is sluggish, over 2
> seconds.
> >
> First, how big are that small system's dependencies?
> Because they will all be scanned.
> Second, how much time is spent in initialize-source-registry?
> Because if you reset it when dumping an image
> (which you probably should
> unless the source is guaranteed to be at the same place at runtime),
> it will scan all the declared source trees.
>

Well, a dependency might include one or more of my own systems, and then it
goes into the big wonderful world of 3rd party depencies, like iterate,
ppcre, base64, sqlite, etc.

I haven't analysed initialize-source-registry, will do.

I have not used asdf at run-time before -- haveing used a dumped image to
run just a small hook file that starts pre-loaded functions, and then any
other functions loaded after this do not declare any dependencies (via
require or direct asdf invocation). This is a new thing for me.

>
> > It looks like the problem crops up when there are many dependencies.
> Poking
> > around the asdf code this appears to be because operate always traverses
> the
> > system dependencies, and pokes the asd files to find if they have
> changed.
> >
> > This makes sense for asdf in development mode, since it needs to discover
> > changed systems. Not so good for a deployed system.
> >
> Well, what do you propose ASDF should do?
> Have some option to operate and/or traverse
> to skip checking some dependencies if they are already loaded,
> kind of a :force-not complement to :force ?
> I suppose I'll accept a patch that does it.
>

In an abstract sense, it would be great if ASDF had a sense of target
environment or strategy, I had a clever word for it before my kids came
home and my brain went out the window. This would be development,
deployment, testing, etc. and could wrap up things like the source,
destintation directories as well as files storage types. E.g. it might be
good to be able to package up stuff in zip files for delivery and have ASDF
look inside them (with proper implementation of that file storage type.)

The target environment could be used to specialize the behavior of asdf,
for instance, how asdf deals with hunting down dependencies, dealing with
no source files.

It may be that the object system would already allow this by specializing
operations and components, but it seems that the concept is orthogonal, no?

This might even be a way at getting at the problem of searching the entire
depency tree for upwind changes.

>
> > ASDF assumes that since the system def file has not changed, the system
> > hasn't been changed either (I know there has been discussion of this
> > assumption.) But lets assume that at least part of the reason for that is
> > because it is very time consuming to check the entire set of files every
> > time the developer wants to recompile and load just a single file.
> >
> I don't think that's true, but my head hurts just thinking about it.
>

>
> > Why not assume in a deployment mode that NO system will have changed if
> it
> > is already loaded in memory, and unless the application has asked that
> it be
> > reloaded? This would be for performance as well as for safety -- we
> really
> > don't want dependency checking to result in any filesystem checks, etc.
> >
> Because that's wrong? I don't have as many dependencies as you do when
> I reload a system with asdf, but usually, I do want the dependencies
> to be reloaded.
>

Assume you have a couple dozen dependencies managed under quicklisp. These
files are really hardly every updated, other than quicklisp batch update.
There should be a way of ensuring that once loaded they are never consulted
again during that lisp session (or in that saved image.)

>
> > I feel that this problem is really pushing my grokking of asdf, but I'm
> > delving ahead anyway.
> >
> > I haven't tested it extensively, but I did just waste a perfect sunday
> > morning fiddling with this:
> >
> > extended module with a slot "up-to-date-p", but surely there is a better
> > name, this I just chose while in early play mode:
> >
> > added these methods:
> >
> > (defmethod operation-done-p ((o load-op) (c module))
> >   (module-up-to-date-p c))
> >
> > (defmethod mark-operation-done ((operation load-op) (c module))
> >   (setf (module-up-to-date-p c) t))
> >
> > tweaked the operate method so that it exits early if the module has been
> > marked as up-to-date
> >
> >  (if (operation-done-p op system)
> >  (return-from operate (values op nil)))
> >
> > and do-traverse to only collect "kids" if it is not done
> >
> > (unless (operation-done-p operation c)
> >  (while-collecting (internal-collect)
> >
> I think that's quite wrong.
> How are you to maintain the up-to-date-ness of your modules?
>

Yes, this is just to make "an asdf which works the way I want it to in this
situation". Remember, this is for a short-lived lisp session which just
needs to load a system (module) once, and to never reload an already loaded
system (because there are many preloaded systems in the image.) Also, for
long lived deployed sessions I'm not at all interested in reloading systems
through asdf. I do have functions that may need to be reloaded, but that is
through a much simpler process of loading and compiling a function and
storing it in a slot somewhere.

In this case I was trying to use the concept of "done-ness", since it is
already there in asdf. I consider a module loading to be done ... after it
is loaded.

>
> > Note that this code goes in the special section of operate for handling
> > modules.
> >
> > I think that is all that I did.
> >
> > I found that the time to handle the require for my current project
> dropped
> > from about 2 1/2 to about 1/2 second.
> >
> I'd rather you implement a :force-not. It sounds cleaner to me.
>

Ok, I'll explore that.

>
> > Thoughts?
> >
> I imagine a far future where an XCVB daemon watches changes to the
> filesystem
> to detect which registered builds require an update. Not quite there yet.
>

I'm very interested in the XCVB project, and this capability sounds great,
but at the moment I'm working with a ton of asdf enlightened modules,
including all of quicklisp, and am really interested in getting it to work
optimally.

Erik.

>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics•
> http://fare.tunes.org
> The older I grow, the more I distrust the familiar doctrine that age
> brings wisdom.  — H.L. Mencken
> (Plus je vieillis, moins je crois en l'idée commune que l'age apporte
> la sagesse.)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20120422/24d90670/attachment.html>