[asdf-devel] An interesting ASDF conundrum

[Faré]

On Sun, Jan 6, 2013 at 3:26 PM, Robert Goldman <rpgoldman at sift.info> wrote:
> This one I think is relevant to the proposed reorganization of ASDF to
> have changes in upstream systems trigger recompilation in systems that
> depend on them.
>
> I have made a crude component which is a MADE-FILE.  The made-file is
> built by invoking make to produce some output file.
>
> MADE-FILE's compile-op OPERATION-DONE-P method always returns NIL,
> because ASDF does not understand what files the made file depends on --
> that information is buried in the makefile.  We don't want to duplicate
> that logic in ASDF.  If we always invoke make, it will do the Right
> Thing and only rebuild the file when necessary.
>
> *Unfortunately*, any components in a system that have a :depends-on
> relationship to a MADE-FILE, will always be recompiled, whether
> necessary or not.  If I remember correctly, the logic in traversal is
> that you do the child if its input files are fresher than the output
> file *or* if one of its dependencies was run.
>
> Now, with current ASDF, I can work around this for MADE-FILEs with
> stable APIs: I just shove them out into systems of their own, so they
> don't trigger constant recompilations of systems that depend on them.
>
> But this behavior will be changing with the new ASDF.
>
> I don't *believe* the existing controls on :force will solve this
> problem, either.  We can't control this by simply saying "recompile
> everything if any upstream system changes" or "don't recompile anything
> based on upstream system changes" (although the latter will restore the
> current buggy behavior).
>
> I can think of two solutions:
>
> 1. KLUDGY -- allow individual system definitions (like my MADE-FILE
> wrapper systems) to specify that they do not propagate recompilation
> forcing onto their dependents.
>
> 2. BETTER, but bigger effect on the protocol:  Separate the question of
> whether an operation is performed on a dependency from whether that
> dependency propagates change downstream.  We could do this by adding a
> function that is run to test whether an <operation,component> pair
> propagates forcing onto its dependents.  By default it would be the same
> as "did the operation get performed?" as now, but it could be overridden
> by component classes like MADE-FILE where, e.g., I could say "did the
> file-write-date of the OUTPUT-FILES change?"
>
> Does this make sense?
>
> cheers,
> r
>
OK, so since it's already working, I assume it's respecting
the constraints of ASDF, which is that planning happens
before any Lisp computation.

I see two cases:
1- The computation that creates your MADE-FILE doesn't depend on
 any Lisp computation. Then you could either
 1a- systematically perform this Make computation BEFORE the Lisp build,
     and have OPERATION-DONE-P return T, or
 1b- (that's ugly) have this computation does during OPERATION-DONE-P itself,
     and return T or NIL depending on whether things have changed.
     Actually, I don't recommend that at all: since OPERATION-DONE-P
     is called *after* OUTPUT-FILES timestamps have been checked,
     this would be bad. But maybe that's a bug in ASDF,
     and we should run OPERATION-DONE-P first (unless we are JUST-DONE).
2- The computation that creates your MADE-FILE *does* depend on
 some previous Lisp computation. Then the only solution is
 to have this computation be a deterministic product of files
 that you can all declare as COMPONENTs of your system
 as STATIC-FILEs if need be, and/or have its INPUT-FILES return
 (a superset of) the set files from which it will be computed.
 Then OPERATION-DONE-P can just return T.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Any sufficiently advanced misquotation is indistinguishable
from an original statement. — John McCarthy, misquoted