[asdf-devel] Proposal for improved support for versions of systems

[Faré]

Dear Vsevolod,

> Approximately after the release of ASDF 2 I've got interested in creating
> yet another system distribution management system on top of ASDF, and that
> made me investigate ASDF internals.  Through that investigation, I've
> understood 2 things:
> * ASDF has support for much more, than is well known and in ordinary use by
> most Common Lisp programmers :)
> * still it's support for versioning (which is essential to distribution
> management) is lacking due to: [...]
>
Yes, ASDF is lacking in terms of support for versioning. And I admit I
am not convinced that extending that support is a good thing. Unless
you're going to reimplement and maintain something that's better than
existing toolchains, then there's no point at all in the exercise:
Lisp users are much better off using existing tools to manage versions
than roll their own half-assed mock up. Dpkg or RPM or Nix are very
well suited for managing the mix-and-match of many different versions
of packages. Git or SVN are very well suited for managing the exact
match of package combinations.

I'm not saying that distribution management is useless, or that you
shouldn't be coding anything, but rather that there are a lot of
interesting problems in the world, and a lot of things that could help
the Lisp community, and that it's not obvious that yet another
distribution management system is the one most worth of the *huge*
cost of doing it right. — And I speak from the experience of trying to
turn a simple one-sentence idea of an improvement upon ASDF into
software (XCVB).

If I may suggest a simpler, cheaper and more useful strategy for
helping with the version management of Lisp software, it would be to
automatically produce Debian, RPM or Nix packages from ASDF
specifications, and let those tools do the rest of the integration.

> Attached is the proposed patch to ASDF, that, in my view, solves the
> aforementioned versioning problems.  It does the following:
> * Unifies internal version representation format in the form of a list of
> integers: (for example, ASDF 2.106 will have an internal version of '(2 106
> 0), while ASDF 2.005 — '(2 5 0)).
Why only three integers? While you're at it, why not accept the whole
range of Debian or RPM version comparisons?

> A final note about versioning: I needed to use pre-reading of ASD files in
> order to know their versions without arising version conflicts.  This works,
> except for the fact, that READ-EVAL is turned off in the process, so such
> version specifiers as:
> * :version #.*hunchentoot-version*
> * :version #.(with-open-file (vers (merge-pathnames "version.lisp-expr"
> *load-truename*)) (read vers))
> do not produce expected results and are treated as empty versions.  There
> will be a need to make a note about that to library developers.
>
If we ever go this way, it might be better to extend ASDF to accept
reading versions from file, either by READing an object, or by
READ-LINEing it.

> Besides, an incompatible change is introduced to FIND-SYSTEM.  The ERROR-P
> optional argument is removed, so plain NIL is unconditionally returned, when
> system is not found, and VERSION and VERSION-P optional arguments are added
> instead.
This sounds very bad to me. What is the use case? How does that affect
the semantics of find-system? Can there now be many instances of a
system in the source-registry? Does find-system now have to grovel for
*all* these instances, load the .asd files to determine the version
and pick the most recent one by default? That's C R A Z Y.

Note that the current interface of find-system seems to be copied from
that of find-class.

The way I see things is that whoever sets up the source-registry
thereby specifies which versions of each piece of software is going to
be used, at which point version management can be taken completely out
of ASDF and into the hands of whoever specifies said source-registry.

If you want to check that what is configured is correct, you can check
versions. And e.g. POIU, XCVB or ASDF-DEPENDENCY-GROVEL do have such
consistency checks. But that's as much as you need.

> I have supplied a number of unit tests to the functions,
> that I have added.  This is done with a very simple unit testing library of
> my own (a microframework so to say) MUTEST, that can be found at:
> http://github.com/vseloved/mutest/.
>
I'd rather ASDF has only one test framework, if possible one with few
dependencies if at all. The current one kind of sucks but works so
far. I'm not against using a different one, but someone should then do
the conversion of existing tests.

> I have also done functional testing of the changes with the systems, that I
> have got installed already (around 70 libraries), and have verified, that
> there are no problems.  At the same time I would not claim, that there will
> not be any problems with some system definitions, that use obscure ASDF
> features (although, I have acquainted myself with ASDF to the point, that I
> quite well understand most of the features as well as internals now).  So I
> ask anyone interested in this set of changes to thoroughly test them.
>
I think that's a good strategy, and one many people have been
following lately. Maybe there's a way to automate and share these
things further.

> Finally, the patch can be applied by simply loading ASDF-VERSION.LISP on top
> of the loaded ASDF 2.106.
>
> I hope, this work proves useful and is integrated into ASDF.  Anyway I'm
> willing to answer any questions and/or improve the implementation, if
> needed.
>
I admit this is not on top of my priority list. I'll try to review
your code in detail later this week. Ideally, a new ASDF maintainer
would step forward and replace me.

Thanks for your efforts,

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Most economic fallacies derive... from the tendency to assume that there
is a fixed pie, that one party can gain only at the expense of another.
	— Milton Friedman