ASDF backward compatibility track record

[Faré]

[Starting a new thread with different topic, as suggested by Robert.]

On Wed, May 3, 2017 at 3:30 PM, Daniel Kochmański <daniel at turtleware.eu> wrote:
> Is there any chance for specification of the "core ASDF protocol" -
> subset guaranteed to not change every few years, so one may write today
> systems which will be loadadable in the future?
>
As far as I can tell, the (API for the) core of ASDF has been
remarkably stable. ASDF still supports many systems written for ASDF 1
in the early 2000s. While the manual is by no means great, especially
as regards documenting the API, it does introduce the main concepts;
and then every function and every class has a docstring, except for a
few internals you're supposed never to use, that only have a comment.
I made a pass over the entire source code last year specifically to
ensure no function was left behind. (If you find a place I failed to
document, please report a bug.)

If there is any part of ASDF that you are struggling to understand,
please discuss what it is, and I can try to document it better.

As to writing systems that will be loadable in the future, I recently
wrote a best practices document to help authors:
https://gitlab.common-lisp.net/asdf/asdf/blob/master/doc/best_practices.md
It can certainly be improved, but probably will be updated on a need
basis, so please ask questions.


That said, a lot of ugly workarounds were necessary in the bad old
days of ASDF 1 (and to a lesser degree ASDF 2). Many of these
workarounds have been deprecated, and many of them have ceased to
work. Instead, they were replaced by simple portable alternatives that
do not require poking at internals or relying on exponentially bad
algorithms.

A few features were removed (if-component-dep-fails, replaced by
if-feature) or are still being removed (run-shell-command, being
replaced by run-program).

Now, there have been cases where someone did something that relied on
undocumented behavior, and never contacted ASDF maintainers to either
get it documented or replaced by a stable API. And then sometimes,
that code breaks when I refactor and fix ASDF internals. In those
cases, tough nuggies, such code *deserves* to lose. Yet, if it's in
Quicklisp, I will find out the breakage, contact the authors, and
submit a patch or a suggestion on what could be done instead. I've
notably fixed code (such as asdf-system-connections) and .asd files
for countless systems.

The proximate issue that prompted this discussion is that I dropped
the heroic support that ASDF had to properly upgrade from 2.x on some
implementations that were less broken than others (including SBCL).
That support was becoming increasingly costly to maintain and test for
dwindling benefit, when all implementations have distributed ASDF 3
for years now, and ASDF 2.x has long been unsupported (except that
Xach still distributes it in Quicklisp, which is a shame). These
heroics included redefined gf sets, symbols to unintern, u-o-f-r-c
methods, classes for which to break transient inheritance cycles
during upgrade, special variables to filter, etc. (Some of it
explained in Robert's and my 2010 paper.) It is hard enough to
maintain that back to ASDF 3, but back to ASDF 2 after a while proved
too much trouble. Most implementations already resorted to renaming
away the entire ASDF package from ASDF 1 or 2, so as to let ASDF 3
create fresh packages. I recently made that "punting" the rule on all
implementations, even SBCL. This was a great relief for me. If it
breaks someone who somehow tries to upgrade from ASDF 2.26, I say they
deserve to lose.


As to what I guess may have triggered your question, one part of ASDF
that has changed a lot is the bundle support (2012-2017), née asdf-ecl
(2005). It was a pain to transform this cool implementation-specific
hack into a portable infrastructure that truly supports all
implementations, doesn't have quadratic or cubic performance
degradation, and generally well with the ASDF build model rather than
cheats and defeats it. Paradoxically, what started as an ECL-only
extension has had plenty of issues on ECL up until the recent 3.2.1
which I hope solves them all (notably thanks to your input). The final
interface while broadly similar to that of asdf-ecl, also has notable
differences. Mostly, asdf-ecl did cut a lot of corners and made a few
questionable choices. I believe most of these choices were probably
well justified at the time of ASDF 1, when fighting to fix ASDF could
have been deemed a pointless endeavor. But when making asdf/bundle out
of asdf-ecl, I made some deep changes to fix the ASDF build model, at
which point these asdf-ecl choices did not make sense anymore. From
your point of view as ECL maintainer, you might see the result as
breaking compatibility with the old asdf-ecl. From my point of view as
ASDF developer, what I did was restoring compatibility with a newly
fixed ASDF 3 (2013).


Best regards,

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Il n'y a absolument rien à dire de ce qui échappe complètement aux mots.
The utterly unspeakable is utterly irrelevant.
                — Faré