[asdf-devel] Module dependencies still arguably wrong

[dherring at tentpost.com]

[I don't expect the following to appear in ASDF.  :)]

Many build systems consider dependencies to be a solved problem. 
Unfortunately, I don't know of any suitable for CL.

There are two proven approaches to detecting changes
- timestamps
- file hashes

The beauty of timestamps is that, on most filesystems, they don't require
saving any extra data.  If any of an output's dependencies are newer than
it is, then it needs to be rebuilt.  However, some filesystems (e.g.
network shares) exhibit significant time jitter or coarse time resolution,
thus breaking this approach.  Moving or untarring old files into the build
area can also be painful.  I suspect the unix "touch" command was
initially created to force updates in timestamp-based systems.

File hashes require extra computation and disk storage.  Existing systems
show that their computation can be a reasonably small fraction of the
compile time.  Rechecks can be optimized by saving the timestamp and a
cheap hash for a quick scan, and a longer hash to compare files flagged by
the first scan.  Time spent determining that one file doesn't need to
recompile may be regained by not recompiling files that depend on it. 
Storage can be shared with a related issue: compile-time dependency
tracking.

System descriptions like ASDF tend to be overly simplistic; indeed that's
a good design goal.  They specify the minimal information required for a
clean first compile -- library dependencies that must be met, obvious
internal dependencies, etc.  A good compiler can output more information;
as each file is compiled, a separate output lists the files which
contained definitions used by this one.  Thus the next time this system is
compiled, the build tools have a better picture of what dependencies to
look at.

Automated dependency tracking is fairly straightforward in C/C++; the
compiler must simply track the paths of each #include file.  Java was
designed to make it a non-issue (e.g. runtime resolution of constants
across class files).

In CL, dependency semantics are more complicated; but I think they boil
down to tracking the source files for each nonstandard (not in the impl)
function used by the reader (e.g. reader macros and #.), all macros, all
constants, and all inline functions.  CL semantics resolve everything else
at load/run time.  Faré may have a better defined set due to his work on
XCVB.

Unfortunately, I don't think any CL implementations can output this
information.  Then again, ISTR XCVB making progress on this front.


Later,
Daniel