[asdf-devel] Faster loading for deployed systems?

[Erik Pearson]

Hi,
I am trying to implement a method of using a "batteries included" lisp
image (ccl, linux, 64bit) which can be used for running a variety of lisp
tasks in a batch mode. I chose to pre-load the image because it was just
too slow to load the individual fasl files at run time, either via asdf or
through a generated load-list.

I'm using ASDF to load a small system at run-time, but finding that even
with small system (say, one file), performance is sluggish, over 2 seconds.

It looks like the problem crops up when there are many dependencies. Poking
around the asdf code this appears to be because operate always traverses
the system dependencies, and pokes the asd files to find if they have
changed.

This makes sense for asdf in development mode, since it needs to discover
changed systems. Not so good for a deployed system.

ASDF assumes that since the system def file has not changed, the system
hasn't been changed either (I know there has been discussion of this
assumption.) But lets assume that at least part of the reason for that is
because it is very time consuming to check the entire set of files every
time the developer wants to recompile and load just a single file.

Why not assume in a deployment mode that NO system will have changed if it
is already loaded in memory, and unless the application has asked that it
be reloaded? This would be for performance as well as for safety -- we
really don't want dependency checking to result in any filesystem checks,
etc.

I feel that this problem is really pushing my grokking of asdf, but I'm
delving ahead anyway.

I haven't tested it extensively, but I did just waste a perfect sunday
morning fiddling with this:

extended module with a slot "up-to-date-p", but surely there is a better
name, this I just chose while in early play mode:

added these methods:

(defmethod operation-done-p ((o load-op) (c module))
  (module-up-to-date-p c))

(defmethod mark-operation-done ((operation load-op) (c module))
  (setf (module-up-to-date-p c) t))

tweaked the operate method so that it exits early if the module has been
marked as up-to-date

 (if (operation-done-p op system)
  (return-from operate (values op nil)))

and do-traverse to only collect "kids" if it is not done

(unless (operation-done-p operation c)
  (while-collecting (internal-collect)


Note that this code goes in the special section of operate for handling
modules.

I think that is all that I did.

I found that the time to handle the require for my current project dropped
from about 2 1/2 to about 1/2 second.

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20120422/1275d793/attachment.html>