Upgrade failures for asdf-3.1.7 and later

[Faré]

>> The problem with central-registry is a bootstrap problem.
>> It requires everyone to be a system administrator and edit configuration files.
>
> I don't follow this claim.  All central registry requires is that you
> run some lisp code, generally in your lisp-init file.
>
Yes, and that's what I mean by "system administrator": you need to
write code in configuration files. Something that end-users shouldn't
have to do.

With ASDF 3.1's source-registry, end-users don't have to edit
anything, and it will just work: just put your code under
~/common-lisp/ — or they can add one line worth of configuration per
non-default source location they want to use.

I wanted to make ASDF suitable for writing code deployed to end-users.
Now it is.

> We have one piece of lisp code that does a
> simple depth-first search and jams things into ASDF:*CENTRAL-REGISTRY*.
> People stuff that into their lisp-init files and that's it.
>
Writing that code is nothing for an expert like you, but a huge hurdle
for novices. Copy/pasting it without understanding it is also full of
dangers.

> You are right, of course, that we can do that with
> INITIALIZE-SOURCE-REGISTRY, too, but I don't understand your claims
> about needing root privilege or needing to understand the bootstrap
> environment any more than just understanding how to write a lisp-init file.
>
It's not about root privilege, it's about being a power user who has
deep understanding of configuration and your Lisp application's
bootstrap process.

>> You may indeed need to re-run (asdf:initialize-source-registry) after
>> you install or remove .asd files. That is documented, and a small
>> price to pay for scaling the registry much beyond what the
>> *central-registry* will crumble with.
>
> It's a small price to pay IF YOU NEED TO SCALE THE REGISTRY BEYOND WHAT
> *CENTRAL-REGISTRY* WILL CRUMBLE WITH.  Otherwise, it's an annoyance
> created by an optimization you don't need.
>
> The largest project I'm working on now has 212 entries in
> ASDF:*CENTRAL-REGISTRY*.  Could be zippier, but that certainly doesn't
> cause the list representation to crumble.
>
> I get it that the new structure scales better, but it is a tradeoff, and
> some users will find that it's not a tradeoff that works for them.
>
Having 212 entries is one to two order of magnitude more than most people need.

I consider my home installation advanced, and it has all of 6 lines of
configuration, counting lines for each of the :source-registry tag and
the :inherit-configuration directive.

For QRes, I had the makefile export CL_SOURCE_REGISTRY with a single
entry, and that was it.

>> A novice does not need to understand it. If you follow the
>> instructions, such as "install your software under ~/common-lisp/",
>> then it Just Works™.
>
> That doesn't work if you work on multiple different systems that have
> different source repositories.  I typically work on at least three
> different CL projects at once, which means a single configuration is a
> non-starter for me.
>
The easiest way is to export CL_SOURCE_REGISTRY from the shell, and/or
equivalently have each project's Lisp init script use
asdf:initialize-source-registry with a parameter.
Unlike loading asdf from .sbclrc, that works without polluting a
shared file for other environments.


>> Experts can dump tables. I hesitated about replicating
>> alexandria:hash-table-alist in uiop, to make it easy to dump some
>> tables. Decided against it in the end, but if you believe novices
>> should be able to dump those tables, that's something to consider.
>
> Yes, I do believe that novices should be able to dump those tables,
> because I have seen novices make mistakes (e.g., having both lisp code
> configuration and a .conf file configuration -- the former being added
> after the latter was forgotten), and just be completely confused and lost.
>
I recently helped Raymond Toy debug an issue he had with ASDF, and for
diagnosing his problem, I asked him to tell me what that form
returned:

(list asdf:*central-registry*
 (uiop:getenv "CL_SOURCE_REGISTRY")
 asdf::*source-registry-parameter*
 (mapcar (lambda (x) (list x (uiop:read-file-string x)))
  (mapcan 'uiop:directory-files
  (list
   (uiop:subpathname (user-homedir-pathname)
    ".config/common-lisp/source-registry.conf")
   (uiop:merge-pathnames*
    ".config/common-lisp/source-registry.conf.d/"
    (user-homedir-pathname)))))
 (loop :for a :in (asdf/source-registry::flatten-source-registry)
   :when (and (getf (cdr a) :recurse) (uiop:probe-file* (car a)))
   :append
   (uiop:while-collecting (c)
     (uiop:collect-sub*directories
      (car a)
      (lambda (d)
       (uiop:if-let
        ((f (uiop:probe-file*
          (uiop:subpathname d ".cl-source-registry.cache"))))
        (c f))
        t)
      (constantly t)
      (constantly t)))))


[It turns out his problem was that he was trying to manually load .asd
files that were not in any of his registries, and expecting it to work
even though they themselves called other unregistered .asd files via
:defsystem-depends-on. Don't do that.]

> But in addition to dumping the tables, everyone must be able to figure
> out where the table entries came from.  That's why something like
> tracing is so important.  Again, if anyone, including a novice, gets
> their configuration messed up -- and, indeed, novices are very likely
> to, because they often just cargo-cult configs from others -- they need
> to be able to see where that configuration comes from and how to fix it.
>
I agree that the source-registry is less easy to debug than could be.
Yet, I have never been contacted for misuse of it, but I have been contacted
a lot for misuse of *central-registry* (though it has gotten much better
since the missing trailing / became detected by asdf itself).

> The current "there's more than one way to do it" configuration process
> means that this is especially important, because we can no longer simply
> look in our lisp initialization to figure out why things have gone up
> the spout.  Instead, we have to check hidden configuration directories,
> environment variables, lisp code, etc.
>
That is true. And it gets only more complicated when Quicklisp
introduces its own extensions. Yet in the end, that was the true power
of ASDF's extensible system search design (thanks, Dan Barlow!): let
people define their own system search facilities, and see which grows
and survives.

>> That code is not the most readable, but what it does it well
>> documented, well debugged, and it's easy to look at the result, change
>> something and look at the result again.
>
> Only if you know where the configuration was that needs to be changed.
> I have definitely seen people who have configurations slopped all over
> the place, some forgotten, and had to try to help them debug.
>
We probably can provide better diagnostic tools.
At the same time, if you grow complex configurations, you're probably
doing it wrong.


>>> Finally, if you configure with configuration files in your XDG config
>>> directories, there's not even a chance to trace, because by the time you
>>> could set up a trace, the configuration's been loaded.
>>>
>> It doesn't actually make tracing harder than usual. Tracing asdf
>> configuration is a normally hard bootstrap issue.
>
> Not if the configuration is in your lisp code it isn't.  Only when we
> added the configuration files.
>
OK, so it makes things slightly harder for the expert who uses SLIME,
but much easier for the novice who doesn't use it yet.
I believe this is a vast net win.


>> On the other hand, the source-registry often "just works" out of the
>> box, or requires a one-liner to configure. The *central-registry* is
>> cognitively much heavier, especially so on novices.
>
> My philosophy is to assume that code will always break and then consider
> how people can recover from that.  So yes, I agree, ASDF now typically
> just works, but we haven't left users with much support when it doesn't.
>
I agree we can offer better debugging support.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Truth comes as conqueror only to those who have lost the art of receiving it
as friend. — Tagore