Is it possible to load multiple versions of the same system?

[Faré]

On Thu, Apr 28, 2016 at 8:05 PM, Ian Tegebo <ian.tegebo at gmail.com> wrote:
> I've been reading the manual, papers, and slides on ASDF and XCVB.  As a
> frame of reference, I prefer the qualities exhibited by Racket's
> implementation of modules.  I was disappointed both to see XCVB bitrot, and
> then to see that a major ASDF overhaul would be necessary.
>
Well, unhappily CL is stuck in the early 1990s and unlikely to grow
any Racket-like features.
As for XCVB, it had many good ideas, but some of them in retrospect
were adding too much complexity for not enough benefits.
If you want to build a Lisp-specific build system today, ASDF3 might
be a better place to start hacking than XCVB:
a few of XCVB's good ideas were added to ASDF2 and ASDF3, and a few of
them are in the asdf/TODO in case anyone wants to write an ASDF4.

One issue with XCVB is that since it was not backward-compatible with
ASDF, it had to be ten times better than ASDF on every implementation
on every OS for every system before people would bother switching. Or
you'd have to convert every system to the new build.
Bazel now supports CL, and is also incompatible and requires users to
convert every system they use. Maybe it's better enough, but I don't
think so, and certainly not for all Lisp implementations at this
point, only SBCL on Linux (and soon MacOS X?).
See https://github.com/qitab/bazelisp

As to me, if I had to design a build system today... see
http://ngnghm.github.io/blog/2016/04/26/chapter-9-build-systems/


> That said, ignoring Racket/ASDF/XCVB, I'm curious about how loading multiple
> versions of the same system could possibly be implemented in Common Lisp
> (CL).  I'm sure people have thought about it, but I'm having a hard time
> finding references.  I'm afraid I've been skimming some of the material
> linked off of the ASDF and XCVB pages, so forgive me if I missed something.
>
What is your use case? Do you want a fully automated general answer,
or just a manual hack that works for one or a few very specific
systems on a specific implementation? In the former case, you're
facing an uphill battle. In the latter case, many hacks can help you.

Various hacks:

* unregistering a system with clear-system then calling
initialize-source-registry with a different value so you see one
system at one time then the other system at another time.
* playing games with ASDF plan generation to load all the dependencies
of a system without loading the system itself.
* detecting packages present before and after loading the system
itself, so you can rename them away.
* using my system package-renaming to cleverly rename relevant
packages around the compilation, loading and/or use of one system.
* fixing your systems to not refer to packages by name at run-time,
and if possible not at load-time either (or make sure to wrapper
load-time in proper package-renaming).

> On the JVM, there's the notion of "classloader" that can be used to load
> multiple versions of the same class.  Unfortunately, the only approach I can
> imagine is to use some form of package renaming.  There appear to be several
> variations floating around [1], and the trade-offs aren't clear to me.
>
> If I think about how it might be done in ASDF, it seems like it would
> require a series of things:
>
> * intercept package definition to rename with version
> * :around-compile to handle some kind of per-system aliasing so code doesn't
> have to change
> * teaching dependency resolution how to find the renamed, versioned
> packages/systems
>
> But that's a wild guess.  And the more I think about it, I wonder what to do
> about some code that uses strings to put together a symbol - I don't think
> symbol-macrolet can help there.
>
> [1] implementation-specific package-local-nicknames, :around-compile in
> ASDF, "pseudonyms"
>

If what you want is a general way to have multiple versions of any
system in a same image, you're out of luck, and/or you need to use a
CL-in-CL implementation (or CL-in-foo) that shields the various
instances of user code from each other.

If what you want is a build system that is full-featured like XCVB
with plenty of dependencies yet compiles things in-image like ASDF but
using user-provided versions of the same systems that your build
system depends on, then yes, some magic package renaming could help,
and could be a one-time thing if your build system never calls a
function that assumes unrenamed packages at runtime. Or then again,
you could use the XCVB farmer model, where the build image with lots
of dependencies farms the compilation to workers that only need to
load a small stub, and no package renaming is necessary. If your
workers also support forking, this can be very efficient, too.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Computer Science is no more about computers than astronomy is about telescopes.
                — E. W. Dijkstra