Upgrade failures for asdf-3.1.7 and later

[Robert Goldman]

On 5/4/17 May 4 -4:23 PM, Faré wrote:
> On Thu, May 4, 2017 at 4:52 PM, Robert Goldman <rpgoldman at sift.net> wrote:
>> On 5/4/17 May 4 -2:34 PM, Faré wrote:
>>> In any case, friends don't let novices use the central-registry.
>>
>> The counter-argument to this is that *central-registry* is much more
>> conceptually simple and easier to debug than the new configuration
>> framework.
>>
> The problem with central-registry is a bootstrap problem.
> It requires everyone to be a system administrator and edit configuration files.

I don't follow this claim.  All central registry requires is that you
run some lisp code, generally in your lisp-init file.
> 
>> If I write code that populates my *central-registry*, I can trace that
>> code and I can look at the *central-registry*.  It's as easy as (PPRINT
>> asdf:*central-registry*)
>>
> But to write that code, you must be an expert who can intercept CL
> configuration and be productive in that bootstrap environment.

Again, I don't get this.  We have one piece of lisp code that does a
simple depth-first search and jams things into ASDF:*CENTRAL-REGISTRY*.
People stuff that into their lisp-init files and that's it.

You are right, of course, that we can do that with
INITIALIZE-SOURCE-REGISTRY, too, but I don't understand your claims
about needing root privilege or needing to understand the bootstrap
environment any more than just understanding how to write a lisp-init file.
> 
> But then you can just as well call (asdf:initialize-source-registry)
> and inspect the contents of asdf::*source-registry* or what is
> returned by (asdf/source-registry:flatten-source-registry), or trace
> functions in the source-registry. But mostly you don't need to,
> because it just works out of the box without configuration.
> 
>> At least in the past, if you added a new asd file to a directory in the
>> *central-registry*, then ASDF would find it.  If you add it to somewhere
>> covered by the new configuration framework, it won't, will it? because
>> the new configuration framework eagerly searches for asd files.
>>
>> (Admittedly, *central-registry* scales poorly for very large programs).
>>
> You may indeed need to re-run (asdf:initialize-source-registry) after
> you install or remove .asd files. That is documented, and a small
> price to pay for scaling the registry much beyond what the
> *central-registry* will crumble with.

It's a small price to pay IF YOU NEED TO SCALE THE REGISTRY BEYOND WHAT
*CENTRAL-REGISTRY* WILL CRUMBLE WITH.  Otherwise, it's an annoyance
created by an optimization you don't need.

The largest project I'm working on now has 212 entries in
ASDF:*CENTRAL-REGISTRY*.  Could be zippier, but that certainly doesn't
cause the list representation to crumble.

I get it that the new structure scales better, but it is a tradeoff, and
some users will find that it's not a tradeoff that works for them.
> 
>> For the new configuration arrangements, things are much more difficult
>> for a novice to understand.  The actual configuration is held in tables
>> that are opaque (yes, they can be dumped, but it's non-trivial and not
>> discussed in the manual).
>>
> A novice does not need to understand it. If you follow the
> instructions, such as "install your software under ~/common-lisp/",
> then it Just Works™.

That doesn't work if you work on multiple different systems that have
different source repositories.  I typically work on at least three
different CL projects at once, which means a single configuration is a
non-starter for me.

> 
> Experts can dump tables. I hesitated about replicating
> alexandria:hash-table-alist in uiop, to make it easy to dump some
> tables. Decided against it in the end, but if you believe novices
> should be able to dump those tables, that's something to consider.

Yes, I do believe that novices should be able to dump those tables,
because I have seen novices make mistakes (e.g., having both lisp code
configuration and a .conf file configuration -- the former being added
after the latter was forgotten), and just be completely confused and lost.

But in addition to dumping the tables, everyone must be able to figure
out where the table entries came from.  That's why something like
tracing is so important.  Again, if anyone, including a novice, gets
their configuration messed up -- and, indeed, novices are very likely
to, because they often just cargo-cult configs from others -- they need
to be able to see where that configuration comes from and how to fix it.

The current "there's more than one way to do it" configuration process
means that this is especially important, because we can no longer simply
look in our lisp initialization to figure out why things have gone up
the spout.  Instead, we have to check hidden configuration directories,
environment variables, lisp code, etc.
> 
>> Much of the configuration code cannot be traced because key operations
>> take place in side-effecting anonymous lambdas.
>>
> That code is not the most readable, but what it does it well
> documented, well debugged, and it's easy to look at the result, change
> something and look at the result again.

Only if you know where the configuration was that needs to be changed.
I have definitely seen people who have configurations slopped all over
the place, some forgotten, and had to try to help them debug.
> 
>> Finally, if you configure with configuration files in your XDG config
>> directories, there's not even a chance to trace, because by the time you
>> could set up a trace, the configuration's been loaded.
>>
> It doesn't actually make tracing harder than usual. Tracing asdf
> configuration is a normally hard bootstrap issue.

Not if the configuration is in your lisp code it isn't.  Only when we
added the configuration files.
> 
>> In short, while the new configuration structure is better in many ways,
>> that's only if it works.  If something odd happens, it leaves you pretty
>> flat.  If you have a small set of libraries in use, *central-registry*
>> can be a lot easier to deal with.
>>
> On the other hand, the source-registry often "just works" out of the
> box, or requires a one-liner to configure. The *central-registry* is
> cognitively much heavier, especially so on novices.

My philosophy is to assume that code will always break and then consider
how people can recover from that.  So yes, I agree, ASDF now typically
just works, but we haven't left users with much support when it doesn't.
> 
>> I've been meaning to provide more debugging to the configuration
>> process, but just keeping up with everyone else's mods to ASDF has
>> eaten all the ASDF time I have for now.
>>
> That would be a good thing.
> 
> When you have time, a post hoc review of the changes from 3.2.0 to
> 3.2.1 would be nice. Also a review of the plan branch, and some
> opinions on what to do about syntax-control and when.

Will do.  I'm just nudging up from half time, though, so I'm moving
slowly.  And maybe I need to prioritize something I want to do (the
debugging utils) over chores, or I'm just going to end up hating to work
on ASDF....

Please note that I don't think what you did was bad, I just think that
it can on occasion leave people flat, and I would like to work to help
them when that happens.

best,
r