[asdf-devel] Proposal for improved support for versions of systems

[Vsevolod Dyomkin]

Dear Faré,

thanks for the detailed reply.  I wanted to clarify some points below.

On Mon, Jun 28, 2010 at 4:14 PM, Faré <fahree at gmail.com> wrote:

> Dear Vsevolod,
>
> > Approximately after the release of ASDF 2 I've got interested in creating
> > yet another system distribution management system on top of ASDF, and
> that
> > made me investigate ASDF internals.  Through that investigation, I've
> > understood 2 things:
> > * ASDF has support for much more, than is well known and in ordinary use
> by
> > most Common Lisp programmers :)
> > * still it's support for versioning (which is essential to distribution
> > management) is lacking due to: [...]
> >
> Yes, ASDF is lacking in terms of support for versioning. And I admit I
> am not convinced that extending that support is a good thing. Unless
> you're going to reimplement and maintain something that's better than
> existing toolchains, then there's no point at all in the exercise:
> Lisp users are much better off using existing tools to manage versions
> than roll their own half-assed mock up. Dpkg or RPM or Nix are very
> well suited for managing the mix-and-match of many different versions
> of packages. Git or SVN are very well suited for managing the exact
> match of package combinations.
>
> I'm not saying that distribution management is useless, or that you
> shouldn't be coding anything, but rather that there are a lot of
> interesting problems in the world, and a lot of things that could help
> the Lisp community, and that it's not obvious that yet another
> distribution management system is the one most worth of the *huge*
> cost of doing it right. — And I speak from the experience of trying to
> turn a simple one-sentence idea of an improvement upon ASDF into
> software (XCVB).
>

I understand that and I'm not pushing the particular issue of a distribution
management tool here.  I'm more in the mood of trying to build something on
top of ASDF with the smallest effort.  Moreover this is not directly
connected to versioning issue, or let's say, proper versioning may have
other application besides what I have in mind.



>
> If I may suggest a simpler, cheaper and more useful strategy for
> helping with the version management of Lisp software, it would be to
> automatically produce Debian, RPM or Nix packages from ASDF
> specifications, and let those tools do the rest of the integration.
>
> > Attached is the proposed patch to ASDF, that, in my view, solves the
> > aforementioned versioning problems.  It does the following:
> > * Unifies internal version representation format in the form of a list of
> > integers: (for example, ASDF 2.106 will have an internal version of '(2
> 106
> > 0), while ASDF 2.005 — '(2 5 0)).
> Why only three integers? While you're at it, why not accept the whole
> range of Debian or RPM version comparisons?
>

Actually, the number of integers is not limited, it's just that this is the
default practice and currently available predicates only take into account
the first two.  So the internal representation is a list of integers.  But
the user can specify versions in many different formats, like "1.2.3" or
even 1.2 (float number).  Moreover it's possible to build adapters for more
representations by defining methods for the generic function PROPER-VERSION.


>
> > A final note about versioning: I needed to use pre-reading of ASD files
> in
> > order to know their versions without arising version conflicts.  This
> works,
> > except for the fact, that READ-EVAL is turned off in the process, so such
> > version specifiers as:
> > * :version #.*hunchentoot-version*
> > * :version #.(with-open-file (vers (merge-pathnames "version.lisp-expr"
> > *load-truename*)) (read vers))
> > do not produce expected results and are treated as empty versions.  There
> > will be a need to make a note about that to library developers.
>

To clarify my point: there are cases of libraries, that use the read-eval
syntax in version definitions, most notable of which are Edi Weitz's and
Cyrus Harmon's.  I agree here with Robert Goldman, that this is a little
contrary to the declarative nature of DEFSYSTEM form.  Still it is not a
problem with neither the existing, nor the proposed variants.  It's just
that in the FIND-SYSTEM pipeline such version declarations (or rather
evaluations) won't be counted (i.e. considered equal to wildcard version).
But during the evaluation of DEFSYSTEM form itself the version would be
properly evaluated.  So this peculiarity will effect the new behavior, but
not the old, that is why I don't consider it a problem: if someone would
like to use the new behavior, he is free to adapt his code, otherwise he can
leave it as is and no problems would arise.


> >
> If we ever go this way, it might be better to extend ASDF to accept
> reading versions from file, either by READing an object, or by
> READ-LINEing it.
>
> > Besides, an incompatible change is introduced to FIND-SYSTEM.  The
> ERROR-P
> > optional argument is removed, so plain NIL is unconditionally returned,
> when
> > system is not found, and VERSION and VERSION-P optional arguments are
> added
> > instead.
> This sounds very bad to me. What is the use case?


The use case is that we can now (oos 'load 'system :version "1.2.3") or
write in defsystem :depends-on (system "1.2.3" :above).  As all those
user-facing facilities use FIND-SYSTEM as the underlying tool.
Independently from adding VERSION and VERSION-P I would still argue to
remove ERROR-P argument, as it adds unnecessary complexity.  There's only
one place in the original ASDF code (OPERATE), where it's expected and it is
easily handled in the caller.


> How does that affect
> the semantics of find-system? Can there now be many instances of a
> system in the source-registry?


The semantics is the following: only zero or one instance of a system with a
particular name can be in memory (in *defined-systems*) — just like before.
The current behavior is that if some system is loaded, there's no
possibility to load the new version.  The new behavior is that, if
FIND-SYSTEM is forced to load the different version of the system, a
continuable error is signaled, that informs the user of the situation and
it's possible either to abort or continue.  If the user continues, the new
definition is loaded instead of the old.


> Does find-system now have to grovel for
> *all* these instances, load the .asd files to determine the version
> and pick the most recent one by default? That's C R A Z Y.
>

Yeah, now SYSDEF-CENTRAL-REGISTRY-SEARCH et al. return a list of found ASD
files instead of the first one.  Then the most suitable is chosen by these
criteria:
* if no version is specified it will be either the one already in memory, or
the first one in the list
* if version is specified it will be the one in memory, if
VERSION-SATISFies, or the first from a list, for which version satisfies


>
> Note that the current interface of find-system seems to be copied from
> that of find-class.
>

Hmm, I see.  I still don't support such decision, but maybe I just not
understand something...


> The way I see things is that whoever sets up the source-registry
> thereby specifies which versions of each piece of software is going to
> be used, at which point version management can be taken completely out
> of ASDF and into the hands of whoever specifies said source-registry.
>

Yeah, that's also a possible approach.  It has it's advantages for sure.


>
> If you want to check that what is configured is correct, you can check
> versions. And e.g. POIU, XCVB or ASDF-DEPENDENCY-GROVEL do have such
> consistency checks. But that's as much as you need.
>
> > I have supplied a number of unit tests to the functions,
> > that I have added.  This is done with a very simple unit testing library
> of
> > my own (a microframework so to say) MUTEST, that can be found at:
> > http://github.com/vseloved/mutest/.
> >
> I'd rather ASDF has only one test framework, if possible one with few
> dependencies if at all.


I agree.  If there's one already chosen, I'll stick to it.  as I've said,
when I've started this work, I didn't find any tests, so I used my own
approach.  By the way MUTEST is dependency-free.


> The current one kind of sucks but works so
> far. I'm not against using a different one, but someone should then do
> the conversion of existing tests.
>
> > I have also done functional testing of the changes with the systems, that
> I
> > have got installed already (around 70 libraries), and have verified, that
> > there are no problems.  At the same time I would not claim, that there
> will
> > not be any problems with some system definitions, that use obscure ASDF
> > features (although, I have acquainted myself with ASDF to the point, that
> I
> > quite well understand most of the features as well as internals now).  So
> I
> > ask anyone interested in this set of changes to thoroughly test them.
> >
> I think that's a good strategy, and one many people have been
> following lately. Maybe there's a way to automate and share these
> things further.
>
> > Finally, the patch can be applied by simply loading ASDF-VERSION.LISP on
> top
> > of the loaded ASDF 2.106.
> >
> > I hope, this work proves useful and is integrated into ASDF.  Anyway I'm
> > willing to answer any questions and/or improve the implementation, if
> > needed.
> >
> I admit this is not on top of my priority list. I'll try to review
> your code in detail later this week. Ideally, a new ASDF maintainer
> would step forward and replace me.
>
> Thanks for your efforts,
>
> [ François-René ÐVB Rideau | Reflection&Cybernethics |
> http://fare.tunes.org ]
> Most economic fallacies derive... from the tendency to assume that there
> is a fixed pie, that one party can gain only at the expense of another.
>        — Milton Friedman
>

Thanks,
Vsevolod
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/asdf-devel/attachments/20100629/47094881/attachment.html>