[asdf-devel] proposal to split ASDF in half

[Daniel Herring]

Hi all,

This has bugged me for years, and recent traffic brings it back to the 
forefront.  I apologize that this email is more a teaser than a complete 
solution, but that's what time allows for right now.

In my opinion, a lot of the complexity in ASDF is caused by its conflation 
of two separate goals: a "build system" and a "loader".

There are a wide variety of build tools available for C/C++ and many other 
languages:  make, imake, cmake, (s)cons, (b)jam, ninja, redo, etc.  While 
the added overhead of installing a build system before compiling a project 
has drawbacks, many of these tools have compatibility with make, and most 
users don't compile most code.  The compatibility of files produced by 
these diverse tools allows a lot of experiments to be run, and successful 
features often get absorbed into tools like GNU make.

This compatibility is allowed because the loader (dynamic linker in C 
speak, class loader in Java) has a simple, well-defined protocol for 
telling it where to find things, independent of the build system.  Look at 
the linux manpages for ld.so and ldconfig...

If CL is to grow from a dev-centric ecosystem to have a large base of 
"normal users", I think we need to start finding ways to enable better 
long-term stability.  Pre-tested, "binary" releases are exactly what an 
end user wants.  Applications need to "just work" and not break whenever a 
dependency changes its API.


What does this mean for ASDF?

Since many CL developers want features like load-time recompilation, the 
simple static decoupling in C between link and load is not possible.  A 
richer interface is required.

While deprecated, the CL PROVIDE and REQUIRE API is very close to ideal 
for a loader.  Obvious extensions include protocols for adding versioning 
checks/negotiation, search paths, re-build hooks, something better than 
*MODULES*, and implementation independent behavior.  This API should be as 
small as possible, truly abstracting the different activities, and 
allowing multiple implementations to coexist.

Then tools like ASDF could produce "installs" that fit into this 
framework.  The core loader would still be integrated into each CL distro, 
but the build tool, with its dependency tracking, source file searching, 
and other features could be distributed separately, and multiple tools 
could coexist in a single image.

Wherever possible, the simplest loader solution should be chosen, leaving 
sysadmins and/or build tools to handle any "difficult" configuration (such 
as hidden directories).  For example, the loader could require that all 
libraries be installed using some UUID or hash-based scheme in a common 
location, and hooks inside these installs are used to activate fancy 
builder-specific features.



I apologize, what I just wrote looks way too complicated.  For example, it 
is possible that versioning, searching, rebuilding, and loading could be a 
builder-specific behavior inside a single try-load function.  However it 
works out, I do believe that a simple decoupled API that breaks ASDF into 
a simple, stable loader core and a more featureful, faster changing piece, 
would be good for everyone in the long term.

Loaders have a simpler required feature set than build systems.  Loaders 
just need to allow for a few types of search scoping (per system, per 
user, and per program).  Build systems require configuration, searching, 
incremental rebuilds, installation, unit testing, etc.

Later,
Daniel