[asdf-devel] Re: Some questions re new ASDF structure

[Faré]

Copying this conversation to asdf-devel, since these are topics of
interest to any hacker who would like to take over development of
ASDF.

On Mon, Jan 6, 2014 at 11:07 AM, Robert Goldman <rpgoldman at sift.net> wrote:
> I'm dribbing and drabbing these out, I'm afraid....
>
> FIND-OPERATION vs. MAKE-OPERATION:  I'm a bit confused by this.  I see
> that operations are all indexed in the *OPERATIONS* hash table, but
> although the names suggest that MAKE-OPERATION would make a fresh
> OPERATION object, and FIND-OPERATION would require that there be a
> previously existing one, it seems like they *both* address the hash
> table.  Nevertheless, a careful distinction is made between calls,
> suggesting that I am missing an important distinction.  It looks like
> there's a future interest in recovering initargs by using the CONTEXT
> argument, but for now that seems to be ignored (i.e., the CONTEXT
> argument never provides anything).
>
> PERFORM:  I believe that the ASDF protocol expects that PERFORM will not
> return anything of interest, correct?  I don't see anyone catching
> return values.  I figure I should probably document this as part of any
> documentation overhaul.
>
I'm glad you ask.

This is another unresolved design issues tracing back to ASDF 1, and
one that I wasn't fully aware of until I wrote ASDF 3,
at which point I fumbled around and ended up settling for a compromise.

The problem at heart is that
* on the one hand, ASDF seems to allow operations to be parametrized
by user-defined slots
* on the other hand, when computing the plan, ASDF drops any such parameters
 as it walks the graph and discards any action such that an action of same type
 on same component has already been visited.

When rewriting ASDF, I tried to reconcile the two, by memoizing
operations created with the same initargs, but eventually failed by
fear of breaking backward compatibility:
* users can and often do call make-instance 'some-operation, at which
point you don't have the unique operation object.
* I tried to have a per-traversal cache, by making the initial
operation given to traverse be the "parent" of all other operations in
a traversal; but this required incompatible changes in the
component-operation-time / mark-operation-done protocol (not a big
deal, since no one should be looking inside), this most importantly
fails to work when one tries to load-system the second time around,
and since no operation object is the same that second time, every
action looks like it hasn't been done yet, and asdf loads everything
the second time around.
* I eventually settled for a global cache of *operations*, but got
discouraged before to check with every user inside (and outside?!)
quicklisp that I could indeed make the incompatible change in
component-operation-time. Also, every single function or method that
can be called with a user-provided operation object must now
"canonicalize" the object before to use it in any of these comparisons
(or more likely, the comparison function should be abstracted, and do
the canonicalization, and for decent speed, cache the canonicalized
operation in a slot of each compared operation object).
* Finally, there was the issue of what it meant for two initargs to be
equivalent — that totally depends on the operation class, and there's
no protocol to help determine when that is or isn't the case. And this
is further complicated by the fact that operate historically passes
its keyword arguments to the operation, which includes things such as
force, which are actually arguments to the plan, and shouldn't affect
the operation — except that swank and gbbopen somehow used that slot
(though they have both been weened off it recently, so we could leave
a compatibility nil method in backward-interface and be done with it).

So there, I did an incomplete job here, and I apologize for it. It
will be your call to either fully allow operation initargs, or
definitely deprecate them. Obviously, no one is relying on the feature
right now, since it has never worked (not uncommon a thing for ASDF —
:depends-on (:feature featr component), :force (sys1 sys2), and
probably more are ASDF 1 features that never worked until I fixed
them, that plus the module dependency that you fixed).


PERFORM returns nothing of interest indeed — it either succeeds or
signals an error.

Now, my plans for an ASDF 4 (or at least 3.2) were to move ASDF
towards a cross-compilation model, which would make life easier for
MOCL, but would also bring to the ASDF world most of the goodies of
XCVB: off-image compilation, deterministic build, parallel build, etc.
This would mean deprecating PERFORM, instead having a method
PERFORM-FORM or such that returns some source code to be run on the
target slave (as opposed to the planning master — the slave doesn't
run ASDF, only UIOP, and doesn't compute the entire plan, so can be
started quickly on N machines or cpus) — and maybe a list of
dependencies to be loaded on the target. When no method is defined for
PERFORM-FORM, ASDF4 would check for a PERFORM method, in which case it
would load ASDF on the target as well as some set of dependencies to
be determined (e.g. the defsystem-depends-on of the current system),
and then would either import the entire set of system definitions
being used or create a fake trivial system with one entry, and then
call perform on the chosen operation and component.

Depending on how serious you are about it, you may or may not also
adopt XCVB's model of :compile-depends-on, :load-depends-on
:run-depends-on, with :depends-on foo implying a :compile-depends-on
foo.cfasl (assuming you also support cfasls), and :load-depends-on
foo.fasl (must be loaded before) and :run-depends-on foo.fasl (can be
loaded after with circular dependencies), whereas
:defsystem-depends-on foo implies :compile-depends-on foo.fasl and
:load-depends-on foo.fasl. Or something more complex;
cross-compilation is hard.

That's also advanced hacking, and a somewhat incompatible change
(modulo that fallback strategy) that requires a lot of social work for
a smooth transition. But hopefully, you can convince the MOCL guy to
do the social work, since it's in his interest to promote a
standardized cross-compilation solution, and good marketing for him
(Cc'ed).


Regards,

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Live as if you were living already for the second time and
as if you had acted the first time as wrongly as you are about to act now!
        — Viktor Frankl, "Man's Search for Meaning"