[asdf-devel] prepare-op

[Faré]

OK, this is now committed as of ASDF 2.26.21 and POIU 1.29.3.
It's slightly smaller, it's much cleaner, and it all works.
I hope I have not broken users — please test.

Casualties of the cleanup were :feature and :if-component-dep-fails.
They were just horrible things.
Another, disabled, :feature feature could replace it:
it was never working in ASDF 1, but
I fixed it some time ago and it's there behind a cerror
waiting to be used.

On the other hand, there are many new features
for those who extend ASDF or inspect its results.
ASDF and POIU are now officially equivalent,
using the exact same traversal functions,
just with different production functions:
ASDF drops information and creates a linear list,
POIU keeps all the information and builds a graph representation.
If you want to examine the graph, you can.

component-depends-on is now more powerful:
it faithfully describes the graph without any implicit inherited dependency,
it allows you to express operations that do not flow down the component
hierarchy, but instead go up or sideways or don't flow (like test-op),
or skip merrily around whichever way you prefer.
It may return component objects as well as designators relative to
the argument component's parent, and the resolve-dependency-spec
is made available to you for custom operations.

component-self-dependencies and the default input-files method
were fixed to handle more cases.

force is implemented via a simple method on prepare-op,
and the mechanism it and force-not use has been factored in a way
that it could be reused for other interesting future uses.

A whole slew of special cases were done away with.

For the first time, ever, I can say that ASDF has a design that makes sense.
It's not perfect, it's not ideal, but it does make sense.

I may write for Dan Barlow his ASDF apology.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
There are two kinds of pacifists: those who try to disarm the criminals, and
those who try to disarm the victims.

On Fri, Dec 14, 2012 at 2:48 AM, Faré <fahree at gmail.com> wrote:
> I have finally fixed an essential "dependency problem" of ASDF by
> introducing a new operation "parent-load-op" which I'm going to rename
> "prepare-op". This in turns opens the way for many many improvements
> to the internals, which will make ASDF easier to explain and extend,
> so expect some work there — and I'm postponing any 2.27 release until
> that's stable. However, any and all changes introduces potential
> backwards-incompatibility, and I'll be searching through packages
> available in quicklisp for any potential such incompatibility before I
> go ahead with renaming internals.
>
> Some details:
>
> In 2.26.14, I introduced an operation called parent-load-op, which
> ensures the dependencies of the parent are loaded before the child is
> compiled. This ensures the graph produced by following
> component-depends-on isn't missing crucial dependencies instead of
> relying on the specifics of the TRAVERSE algorithm (and at long last I
> really understand what rpgoldman was telling me while we were writing
> the ASDF 2 paper). I had to tweak traverse, because parent-load-op
> trickles upward, not downward, the component hierarchy: you depend-on
> the parent-load-op of your parents, but not on that of your children
> (that would create a circular dependency). So I had to add a check in
> TRAVERSE (now — 2.26.20 — in its nice and short visit-children helper)
> to not automatically propagate subclasses of parent-op dependencies
> downward. I started thinking about a theory of parent-op and child-op
> dependencies and how to rewrite the current hierarchy, and as I was
> doing it found that actually, parent-load-op is not parent-specific:
> it's actually a dependency-op or prepare-op, that ensures that the
> image is prepared to compile or load the component, by having already
> loaded all the dependencies, and it could do it for the "normal" case,
> too: instead of having load-op or compile-op depend on the load-op of
> all the dependencies, they could simply depend on prepare-op. Then, we
> don't need to have traverse know anything about propagating
> dependencies either downward or upward, it can all be done with two
> methods on component-depends-on! [which is one of the many misnomers
> in ASDF: components don't depend on anything, it's (operation .
> component) pairs, which I'm calling "actions", like in Kent Pitman's
> article), that are nodes in the dependency graph]. It was all quite a
> revelation. And so I'll be working on drastically cutting down ASDF,
> and making the basic dependency graph cleaner, which will make ASDF
> much simpler and more reliable to extend.
>
> Some change I'd like to make if no one uses these internals (or only
> use them without relying on the current setup to define methods, in
> which case I can define an alias]:
>
> module ==> make it a subclass of both child-component and parent-component
> system ==> is a parent but not a child, and thus not a module.
> component-depends-on ==> rename it to action-depends-on
> if-component-fails ==> I'd like to remove it altogether, it's a crock
> that doesn't go well with the reified graph model. I suppose I could
> preserve it the hard way, but would be ugly.
> mark-operation-done ==> maybe rename to stamp-action ? Or at least
> mark-action-done.
> operation-description ==> action-description
> component-visited-p ==> action-visited-p
> component-depends-on ==> component-dependencies - likely not possible
> for backwards compatibility.
> component-load-dependencies ==> component-depends-on - same. Then
> component-sibling-dependencies.
> circular-dependencies-components ==> circular-dependencies-actions,
> it's a list of (operation . component) pairs
> hash-tables of symbol, component ==> hash-tables of operations and
> component, systematically using make-sub-operation
> make-sub-operation ==> find-operation, to work like find-component,
> and ignore the first contextual argument if the second is already an
> object. Canonicalizes named operations while still allowing explicitly
> different operations of the same class.
> module-components ==> component-children
> module-components-by-name ==> component-children-by-name
> load-fasl-op ==> load-system-fasl-op
> load-op ==> split between load-op and load-fasl-op (see below).
> Alternatively, for backwards compatibility, call the latter
> load-compiled-op
>
>
> My current temporary names would also be renamed:
> parent-load-op ==> prepare-op (or dependency-op?)
> visit-children ==> done away with -- replaced entirely by one
> component-depends-on method.
> parent-op ==> upward-operation
> child-op ==> downward-operation
>
> Chasing "horizontal" dependencies to siblings under the same parent
> would be done entirely by prepare-op, which is a subclass of
> upward-operation. load-op would remain a downward-operation, but would
> have a separate strategy for deciding whether to delegate to
> load-source-op, load-fasl-op or load-system-fasl-op or whatever (e.g.
> some load-instrumented-op extension), based on a generalization of the
> force / force-not mechanism that would decide the compilation
> strategy. These ones would NOT be downward-operations, unlike the main
> load-op which would be a downward-op and propagate everywhere as
> appropriate. Since parents vs children vs siblings are no longer a
> fixture of the TRAVERSE algorithm but something reified in the object
> graph through the normal use of simple methods, you can easily define
> arbitrary graphs of your choice, instead of ASDF only being applicable
> to one particular kind of graph shapes. etc. Then the base object
> hierarchy becomes robust and stable, and it makes sense and is easy to
> extend, with a nice well-define dependency graph, that can be reified
> thanks to POIU's current make-parallel-plan.
>
> A lot of that (and previous) work was prompted by the desire to keep
> POIU working as I refactored ASDF, and then the desire to fix ASDF
> based on the things understood while adapting POIU. POIU interestingly
> builds an actual complete graph of dependencies from the object model,
> or at least did it within each system, because, not being able to fix
> ASDF, Andreas Fuchs had intercepted perform, not operate or
> perform-plan (which didn't exist then). One ugly inefficient detail
> Andreas had in his graph-building function was an
> additional-dependencies argument he used to copy the parent's
> dependencies on each and every child. That's really ugly. I wanted to
> get rid of it, but discovered I couldn't... until I invented
> parent-load-op, which makes it unnecessary, and also promises to fix
> compute-action-stamp. In any case, Andreas Fuchs certainly deserves a
> lot of credit for his POIU hack, of which I didn't fully understand
> the elaborate until I had to fully reimplement it bit by bit, to make
> it cleaner and fit the new ASDF, and admire each detail he got just
> right. And he did it all as an add-on, without changing a single line
> of ASDF itself (though overriding a few definitions — which I promptly
> made unnecessary once I became ASDF maintainer). That's impressive.
> And the language is also impressive for allowing such things, at all.
> But the result was absolute ugly. Being the unexpected co-maintainer
> of the two sometimes conflicting pieces of software has led me to make
> both of them ugly no more.
>
> PS: In the spirit of people who credit John McCarthy with having
> discovered Lisp rather than created it, I like think ASDF was a great
> *discovery* made by Dan Barlow, rather than some badly unfinished
> program he created; I'm only just reaching what I believe is the
> bottom of the matter, and it's much nicer than I thought it was,
> or than Dan probably understood, either, at the time.
>
> PPS: I was going to sent a mail to this list, then it grew very long
> into what would be a blog post, then two, etc. Scrap it. Back to a
> notice to the list. Still grew up to be too long. Oh well.
>
> I hope at least one of you read that and wasn't bored.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> Politics is the only profession that does without learning, probably because
> those who suffer from mistakes are not the same as those who make them.
>         — Achille Tournier, Pensées d'automne