[asdf-devel] prepare-op

[Faré]

I have finally fixed an essential "dependency problem" of ASDF by
introducing a new operation "parent-load-op" which I'm going to rename
"prepare-op". This in turns opens the way for many many improvements
to the internals, which will make ASDF easier to explain and extend,
so expect some work there — and I'm postponing any 2.27 release until
that's stable. However, any and all changes introduces potential
backwards-incompatibility, and I'll be searching through packages
available in quicklisp for any potential such incompatibility before I
go ahead with renaming internals.

Some details:

In 2.26.14, I introduced an operation called parent-load-op, which
ensures the dependencies of the parent are loaded before the child is
compiled. This ensures the graph produced by following
component-depends-on isn't missing crucial dependencies instead of
relying on the specifics of the TRAVERSE algorithm (and at long last I
really understand what rpgoldman was telling me while we were writing
the ASDF 2 paper). I had to tweak traverse, because parent-load-op
trickles upward, not downward, the component hierarchy: you depend-on
the parent-load-op of your parents, but not on that of your children
(that would create a circular dependency). So I had to add a check in
TRAVERSE (now — 2.26.20 — in its nice and short visit-children helper)
to not automatically propagate subclasses of parent-op dependencies
downward. I started thinking about a theory of parent-op and child-op
dependencies and how to rewrite the current hierarchy, and as I was
doing it found that actually, parent-load-op is not parent-specific:
it's actually a dependency-op or prepare-op, that ensures that the
image is prepared to compile or load the component, by having already
loaded all the dependencies, and it could do it for the "normal" case,
too: instead of having load-op or compile-op depend on the load-op of
all the dependencies, they could simply depend on prepare-op. Then, we
don't need to have traverse know anything about propagating
dependencies either downward or upward, it can all be done with two
methods on component-depends-on! [which is one of the many misnomers
in ASDF: components don't depend on anything, it's (operation .
component) pairs, which I'm calling "actions", like in Kent Pitman's
article), that are nodes in the dependency graph]. It was all quite a
revelation. And so I'll be working on drastically cutting down ASDF,
and making the basic dependency graph cleaner, which will make ASDF
much simpler and more reliable to extend.

Some change I'd like to make if no one uses these internals (or only
use them without relying on the current setup to define methods, in
which case I can define an alias]:

module ==> make it a subclass of both child-component and parent-component
system ==> is a parent but not a child, and thus not a module.
component-depends-on ==> rename it to action-depends-on
if-component-fails ==> I'd like to remove it altogether, it's a crock
that doesn't go well with the reified graph model. I suppose I could
preserve it the hard way, but would be ugly.
mark-operation-done ==> maybe rename to stamp-action ? Or at least
mark-action-done.
operation-description ==> action-description
component-visited-p ==> action-visited-p
component-depends-on ==> component-dependencies - likely not possible
for backwards compatibility.
component-load-dependencies ==> component-depends-on - same. Then
component-sibling-dependencies.
circular-dependencies-components ==> circular-dependencies-actions,
it's a list of (operation . component) pairs
hash-tables of symbol, component ==> hash-tables of operations and
component, systematically using make-sub-operation
make-sub-operation ==> find-operation, to work like find-component,
and ignore the first contextual argument if the second is already an
object. Canonicalizes named operations while still allowing explicitly
different operations of the same class.
module-components ==> component-children
module-components-by-name ==> component-children-by-name
load-fasl-op ==> load-system-fasl-op
load-op ==> split between load-op and load-fasl-op (see below).
Alternatively, for backwards compatibility, call the latter
load-compiled-op


My current temporary names would also be renamed:
parent-load-op ==> prepare-op (or dependency-op?)
visit-children ==> done away with -- replaced entirely by one
component-depends-on method.
parent-op ==> upward-operation
child-op ==> downward-operation

Chasing "horizontal" dependencies to siblings under the same parent
would be done entirely by prepare-op, which is a subclass of
upward-operation. load-op would remain a downward-operation, but would
have a separate strategy for deciding whether to delegate to
load-source-op, load-fasl-op or load-system-fasl-op or whatever (e.g.
some load-instrumented-op extension), based on a generalization of the
force / force-not mechanism that would decide the compilation
strategy. These ones would NOT be downward-operations, unlike the main
load-op which would be a downward-op and propagate everywhere as
appropriate. Since parents vs children vs siblings are no longer a
fixture of the TRAVERSE algorithm but something reified in the object
graph through the normal use of simple methods, you can easily define
arbitrary graphs of your choice, instead of ASDF only being applicable
to one particular kind of graph shapes. etc. Then the base object
hierarchy becomes robust and stable, and it makes sense and is easy to
extend, with a nice well-define dependency graph, that can be reified
thanks to POIU's current make-parallel-plan.

A lot of that (and previous) work was prompted by the desire to keep
POIU working as I refactored ASDF, and then the desire to fix ASDF
based on the things understood while adapting POIU. POIU interestingly
builds an actual complete graph of dependencies from the object model,
or at least did it within each system, because, not being able to fix
ASDF, Andreas Fuchs had intercepted perform, not operate or
perform-plan (which didn't exist then). One ugly inefficient detail
Andreas had in his graph-building function was an
additional-dependencies argument he used to copy the parent's
dependencies on each and every child. That's really ugly. I wanted to
get rid of it, but discovered I couldn't... until I invented
parent-load-op, which makes it unnecessary, and also promises to fix
compute-action-stamp. In any case, Andreas Fuchs certainly deserves a
lot of credit for his POIU hack, of which I didn't fully understand
the elaborate until I had to fully reimplement it bit by bit, to make
it cleaner and fit the new ASDF, and admire each detail he got just
right. And he did it all as an add-on, without changing a single line
of ASDF itself (though overriding a few definitions — which I promptly
made unnecessary once I became ASDF maintainer). That's impressive.
And the language is also impressive for allowing such things, at all.
But the result was absolute ugly. Being the unexpected co-maintainer
of the two sometimes conflicting pieces of software has led me to make
both of them ugly no more.

PS: In the spirit of people who credit John McCarthy with having
discovered Lisp rather than created it, I like think ASDF was a great
*discovery* made by Dan Barlow, rather than some badly unfinished
program he created; I'm only just reaching what I believe is the
bottom of the matter, and it's much nicer than I thought it was,
or than Dan probably understood, either, at the time.

PPS: I was going to sent a mail to this list, then it grew very long
into what would be a blog post, then two, etc. Scrap it. Back to a
notice to the list. Still grew up to be too long. Oh well.

I hope at least one of you read that and wasn't bored.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Politics is the only profession that does without learning, probably because
those who suffer from mistakes are not the same as those who make them.
        — Achille Tournier, Pensées d'automne