[asdf-devel] Make the CL syntax predictable

[Faré]

>>: Faré
>: Robert

>> Here is my theory: safe code needs to respect hygiene, because you
>> never know which system are loaded before or after: it depends on the
>> overall plan of the toplevel target system. Therefore is it never
>> correct to rely on other systems having set a particular readtable,
>> and it is never correct to side-effect the current readtable into
>> something that modifies standard characters — or any character that
>> any other program might want to use, for the matter. If you
>> side-effect the readtable, and the readtable escapes, you are going to
>> hurt someone, somewhere.
>>
>> In the current setup, it is therefore a bug if you side-effect the
>> readtable, and those programs that do are buggy. Using
>> named-readtables is fine, but only if you reset the readtable to the
>> standard one at the end of every file, or have a method that binds
>> *readtable* around your compilation.
>>
>> I see two ways forward: (1) recognize that it is a bug, and help
>> programmers by being strict, modulo some compatibility mode for older
>> systems until they are fixed, or (2) slightly change the semantics so
>> that it's not a bug anymore, by making the readtable either per-file
>> or per-system; then you can modify the readtable to your heart's
>> content, because it's private.
>>
>> My initial reaction was (1), and that's more or less the patch I sent
>> to anton, who tells us it breaks a lot of things. At Anton's
>> suggestion, I considered (2) and it looks promising — but we need to
>> modify the protocol and/or use a new method-combination to achieve the
>> effect in a backward compatible way.
>
> No, we cannot do either (1) or (2).
>
> CL does not say it is a bug to rely on an upstream-loaded file to set
> the readtable to a value that you expect.  So breaking code that makes
> that assumption by forcing file-scoped readtables and pprint dispatch
> tables on the programmer is not an OK thing for us to do.
>
> It's not ASDF's job to break code that we consider to be bad code.
>
> The CL spec went out of its way to *permit* the code that you are
> arguing to be buggy.  They *could* have demanded a readtable argument,
> but they did not.  They could have imposed a scope on readtables, but
> they did not.
>
> Furthermore, we are not omniscient: the programmer could knowingly count
> on a non-file-scoped READTABLE in a way that s/he knows is safe, because
> s/he can prove to him/herself that *READTABLE* will be properly bound at
> all locations in the code.  That information could be perfectly correct,
> but invisible to ASDF.  We should not break such code.
>
> Making constructs file-scoped willy-nilly is not something that ASDF
> should do.  In my opinion, it's overreach.
>
> To analogize: that's a job for lint, not a job for make.
>
I disagree. You cannot ever rely on a readtable from a different
system, because you don't control which system was loaded immediately
before you. That depends on the user's toplevel system and its
dependencies. Therefore, it's always a bug to rely on it. You might
want to depend on changes made by the file before you if it's in the
same system, and ASDF could either bind *readtable* to a per-system
object, or, if you want to try harder, update the *readtable* object
after each file that is compiled or loaded, initializing it in
*prepare-op* to a copy-readtable nil. Similarly, it's always a bug to
pollute the next system. It really can't reasonably expect any useful
changes, so every system should get its own table. If we try hard,
there's a whole lot of syntax variables that would have to be
preserved from file to file. And of course, even within a system, you
better only rely on changes made by files you depend on, and not rely
on changes not being made by files that don't depend on you — the
latter being trickier.

Without ASDF enforcement, any syntax change is a landmine, *even at
the REPL*. I remember using fare-quasiquote at the REPL, then doing a
load-system after having changed asdf or such — now every file in asdf
using ` was compiled into something that depended on fare-quasiquote
at runtime. Big bug. Really, safe syntax change requires ASDF support.
Once again, see
http://fare.tunes.org/files/tmp/asdf/asdf3-2014.html#%28part._.Safety_before_.Ubiquity%29
(which I just edited some more).

This is all probably too much work to be a 3.1.1 issue, but that's
definitely something you'll need if you want to support syntax
modification rather than let it be a big danger.

PS: you wanted to somehow have things happen around the loading of a
system. We now have that, kind of: the prepare-op first, the load-op
last. Except it might be only a compile-op or a load-fasl-op. Also,
while this require special user magic, ASDF does not prevent a
non-system to depend on a system, yet kind of makes the assumption
that this will not happen. I suppose this requires documentation
and/or painful runtime checking.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Pacifism is a shifty doctrine under which a man accepts the benefits of the
social group without being willing to pay — and claims a halo for his
dishonesty. — Robert Heinlein