[asdf-devel] In defense of ASDF & Semantic versioning

[Daniel Herring]

On Tue, 19 Nov 2013, Faré wrote:

> Thanks to #+foo, #.(if (find-symbol "FOO" :bar) ...)
> and (eval-when ...), CL can deal with source-level compatibility
> in ways that C cannot.

Moving a bit off-topic, GNU Autoconf takes probing for function 
availability and behavior further towards a science than perhaps any other 
system in use.  [Its a shame that M4 and portable shell scripting were 
required for pragmatic reasons.  That said, many lispers would learn some 
interesting macro lessons by studying Autoconf's use of M4.]

Also, runtime probing for functionality actually are often used in C/C++ 
programs, though usually not understood by most of the dev team.  These 
probes frequently take the form of dlopen/dlsym and their Win32 
equivalents.  Also, many APIs like NPAPI, OpenGL, CUDA, and CPUID have 
their own extension registry systems.

FWIW, my little "read-macros" package demonstrated some functionality to 
simplify writing read-time conditional code without pushing everything on 
*features*.

http://git.tentpost.com/?p=lisp/read-macros.git


Back on-topic, semantic versioning systems such as advocated by GNU 
libtool try to provide a conservative estimate on portability.  In such 
environments, it is not uncommon for a project to make a new release 
simply because a dependency bumped a version number.  This is a primary 
motivation for the autoconf-style detection of behavior rather than 
trusting in names and version numbers.  If behavior is detected, then a 
new version may simply require recompilation instead of a new source 
release.


One approach to solving "DLL-hell" is to essentially avoid shared 
libraries altogether.  Significant progress can be made by having an 
application fully specify the revisions of all dependencies.  Then these 
can be delivered atomically with the application, and a build system can 
assemble them.  This in the path used by most commercial MSWin 
applications, several Java and Javascript frameworks, etc.  Problems arise 
when two dependencies require different versions of a common third 
dependency, especially when that third dependency needs to have 
"singleton" behavior for consistency.


On Tue, 19 Nov 2013, Anton Vodonosov wrote:

> But my point - it's not enough to just bump major
> version number, as semantic versioning suggests.
>
> If author of "somelib" library wants to make an API incompatible
> change, it is better to release new ASDF system "somelib2"
> and put code into new package somelib2.

This concept resonates with me.  The existence of a new API version should 
not preclude further releases of older API versions.  Branching, and thus 
API versions, should not be restricted to a simple linear progression. 
Most semantic versioning systems partially acknowledge this through the 
use of "minor" and "micro" version numbers.  (A git-like tree would be 
better...)

Towards this end, I had started investigating a set of macros and/or 
features to simplify the process of embedding version information directly 
into the CL package names themselves.  Unfortunately, I didn't find any 
clean solution that met my goals.  Here are a couple emails on the 
subject.

http://article.gmane.org/gmane.lisp.libcl.devel/110
http://article.gmane.org/gmane.lisp.libcl.devel/123
[Note: LibCL and its mailing lists are now defunct.]


Another possible source of ideas is the FreeBSD's new "pkg-ng" system. 
Apparently they found a nice solution to integrate binary packages with 
locally-compiled source ports.


All that said, semantic versioning is tried and true, easy to implement, 
and a useful improvement on the current ASDF status quo.  Other approaches 
such as behavior testing and nonlinear are harder to implement and should 
play nicely with a semantic versioning system.  Thus I am all in favor 
with ASDF adopting a reasonable semantic system today.


Whatever we do, please implement an escape hatch for the end user to 
override the versioning system's idea of compatibility.  These things 
often have obscure failure modes and/or prevent nuanced usage.  Just like 
CL::internal symbols, it is nice to have a straightforward way to void the 
warranty and bypass the normal safety mechanisms.

- Daniel