[asdf-devel] Module dependencies still arguably wrong

[Robert Goldman]

On 2/10/10 Feb 10 -10:08 PM, dherring at tentpost.com wrote:
> [I don't expect the following to appear in ASDF.  :)]
> 
> Many build systems consider dependencies to be a solved problem. 
> Unfortunately, I don't know of any suitable for CL.
> 
> There are two proven approaches to detecting changes
> - timestamps
> - file hashes
> 
> The beauty of timestamps is that, on most filesystems, they don't require
> saving any extra data.  If any of an output's dependencies are newer than
> it is, then it needs to be rebuilt.  However, some filesystems (e.g.
> network shares) exhibit significant time jitter or coarse time resolution,
> thus breaking this approach.  Moving or untarring old files into the build
> area can also be painful.  I suspect the unix "touch" command was
> initially created to force updates in timestamp-based systems.

....

I'm going to ignore file-hashes.  If someone else is really gung-ho for
them, great.
> 
> System descriptions like ASDF tend to be overly simplistic; indeed that's
> a good design goal.  They specify the minimal information required for a
> clean first compile -- library dependencies that must be met, obvious
> internal dependencies, etc.  A good compiler can output more information;
> as each file is compiled, a separate output lists the files which
> contained definitions used by this one.  Thus the next time this system is
> compiled, the build tools have a better picture of what dependencies to
> look at.
> 
> Automated dependency tracking is fairly straightforward in C/C++; the
> compiler must simply track the paths of each #include file.  Java was
> designed to make it a non-issue (e.g. runtime resolution of constants
> across class files).
> 
> In CL, dependency semantics are more complicated; but I think they boil
> down to tracking the source files for each nonstandard (not in the impl)
> function used by the reader (e.g. reader macros and #.), all macros, all
> constants, and all inline functions.  CL semantics resolve everything else
> at load/run time.  Faré may have a better defined set due to his work on
> XCVB.

A real issue is that ASDF has provided (insufficiently documented) means
for extending the protocol for updating things /without/ clearly boiling
this back to something like timestamps.  Which means that the notion of
"dependency" is not operationalized.  A subsidiary hindrance is the
notion of operations on composites (modules and systems) have odd
semantics because of the postorder traversal.

WRT macros, BTW, I believe that the Allegro defsystem has a special
dependency relationship that corresponds to a macro definition,
:definitions, which imposes more substantial recompilation obligations
than a non-macro component.  See
http://www.franz.com/support/documentation/current/doc/defsystem.htm for
possibly helpful discussion.

Another issue is that CL has a notion of RUNTIME state, which make does
not have to deal with.  That is, in a running lisp image, a CL
make-alike must provide the means to go from one good state to another
through a combination of compiling and loading.  Make need only worry
about creating a good state from zero.
> 
> Unfortunately, I don't think any CL implementations can output this
> information.  Then again, ISTR XCVB making progress on this front.

It's not clear to me that this information is effectively computable, in
which case we have to fall back on the defsystem author providing it.

best,
r