[asdf-devel] proposal to split ASDF in half

[Faré]

>: dherring

> In my opinion, a lot of the complexity in ASDF is caused by its conflation
> of two separate goals: a "build system" and a "loader".
>
Actually, in my asdf3 article, which I invite you to review,
I actually list that as an asset that ASDF has compared to C systems:
by having the same component path at compile-time and runtime,
you minimize a lot of the complexity that C software distribution systems have.
C systems need to ensure coherency between .h files and .so files
via a complex mix of technical and social rules, and umpteen different
configuration managers (autoconf, gconf, kdeconfig, etc., etc.).
C code needs to be installed twice, once as source, another time as libraries.

[Source] https://github.com/fare/asdf3-2013/blob/master/asdf3-2014.scrbl
[HTML] http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html
[PDF] http://fare.tunes.org/files/tmp/asdf/asdf3-2014.pdf

> This compatibility is allowed because the loader (dynamic linker in C speak,
> class loader in Java) has a simple, well-defined protocol for telling it
> where to find things, independent of the build system.  Look at the linux
> manpages for ld.so and ldconfig...
>
The CL_SOURCE_REGISTRY was used by XCVB as well as ASDF, so there.
But admittedly there is only one codebase: I
originally wrote the code for XCVB,
ebzzry wrote an initial port to ASDF,
I hacked the code a lot as part of ASDF,
and XCVB is reusing most of the code directly from ASDF.
cl-launch also uses the registry via ASDF,
and can be passed options to override it.

XCVB usually produces self-contained executables;
cl-launch can produce either self-contained executables
or scripts that rely on the source-registry.

> If CL is to grow from a dev-centric ecosystem to have a large base of
> "normal users", I think we need to start finding ways to enable better
> long-term stability.  Pre-tested, "binary" releases are exactly what an end
> user wants.  Applications need to "just work" and not break whenever a
> dependency changes its API.
>
If a user installs libraries in ~/common-lisp/ or wherever
that override system libraries, he probably means it.
If a programmer really wants to not have his libraries overridden by users,
he can either fixate the source-registry in his script,
or dump a self-contained executable.

I think ASDF is already good enough in this regard.

> What does this mean for ASDF?
>
> Since many CL developers want features like load-time recompilation, the
> simple static decoupling in C between link and load is not possible.  A
> richer interface is required.
>
> While deprecated, the CL PROVIDE and REQUIRE API is very close to ideal for
> a loader.  Obvious extensions include protocols for adding versioning
> checks/negotiation, search paths, re-build hooks, something better than
> *MODULES*, and implementation independent behavior.  This API should be as
> small as possible, truly abstracting the different activities, and allowing
> multiple implementations to coexist.
>
They are not ideal at all; indeed, they work completely differently
on each implementation (or sometimes not at all, e.g. in GCL).
And there is no way to portably extend this API...
and/or the closest thing there is to it is ASDF itself,
which manages to hook itself into require on all seven maintained
free software CL implementations.

> Then tools like ASDF could produce "installs" that fit into this framework.
> The core loader would still be integrated into each CL distro, but the build
> tool, with its dependency tracking, source file searching, and other
> features could be distributed separately, and multiple tools could coexist
> in a single image.
>
I'm not sure what you mean that cl-launch doesn't do.

Are you suggesting a shared "big image" with everything pre-loaded,
and what more one that maintains a buildapp-like and extensible
list of dispatched-entry main functions depending on
what the image is invoked under?

> Wherever possible, the simplest loader solution should be chosen, leaving
> sysadmins and/or build tools to handle any "difficult" configuration (such
> as hidden directories).  For example, the loader could require that all
> libraries be installed using some UUID or hash-based scheme in a common
> location, and hooks inside these installs are used to activate fancy
> builder-specific features.
>
ASDF has an extensible *system-definition-search-functions*,
courtesy of the far-seeing Dan Barlow. You can certainly adapt it some way.
Or you can preload systems and use the new *immutable-systems* feature
to prevent reloading.

> I apologize, what I just wrote looks way too complicated.  For example, it
> is possible that versioning, searching, rebuilding, and loading could be a
> builder-specific behavior inside a single try-load function.  However it
> works out, I do believe that a simple decoupled API that breaks ASDF into a
> simple, stable loader core and a more featureful, faster changing piece,
> would be good for everyone in the long term.
>
> Loaders have a simpler required feature set than build systems.  Loaders
> just need to allow for a few types of search scoping (per system, per user,
> and per program).  Build systems require configuration, searching,
> incremental rebuilds, installation, unit testing, etc.
>
I'm not sure at all what you mean, and what that would look like.
ASDF is pretty minimal within constraints — certainly, the mission creep
has certainly led to a tenfold size increase in size since the Dan Barlow days
(only fivefold and a half since I took over), but I'll argue that
extending the mission was justified. Being able to write a #!/usr/bin/cl
script that doesn't have to be modified specially depending on the
implementation used and local installation paths — that's worth it.
Oh, and not having any of the many bugs in ASDF1.

That said, if you want to release a "load-only" version of ASDF that
just hooks into require, only does the loading part assuming fasl-op output,
go ahead. That might be interesting.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
God is Dead
        — Nietzsche
Nietzsche is Dead
        — God
Nietzsche is God
        — The Dead