[asdf-devel] Make the CL syntax predictable

[Faré]

On Sat, Mar 15, 2014 at 4:52 PM, Robert P. Goldman <rpgoldman at sift.info> wrote:
> Here is an example of a kind of system that will be broken by this
> proposed change:
>
> * We have a distributed system that is lisp-based.
>
> * There is a small set of ASDF systems that constitutes the common base
> of the modules of this distributed system.
>
> * For each module, there is an ASDF system, which incorporates the
> common base, the module proper, and possibly some additional systems.
>
> * The common base has a readtable (and it may have a PPRINT-DISPATCH-TABLE).
>
> * The distributed modules systems assume, safely, that the readtable
> will be set by the common base.
>
> * Code in the common base uses readtable-setting functions with the
> default values for the readtable.
>
The clean thing to do would be to use named-readtables and/or
cl-syntax, and have each file evaluate (in-readtable :foo) or have a
perform :around method or around-compile hook that does it for you.

We could also support your doing (in-readtable :foo) only once for the
entire system, though it's trickier.

(Question, either way: how does the readtable get re-initialized (or
not) when you recompile a modified system? Should you start from a
fresh readtable every time, or reuse the readtable of the end of
compilation while at the start of compilation?)

But polluting arbitrary systems loaded after you is definitely a bad
idea: if someone else write a system that depends on part of yours,
that also depends on unrelated systems, these unrelated systems will
end up being affected by your readtable changes, and, if they are just
as "dirty" as yours, they might in turn affect your system.

Such idiom is guaranteed to break, badly, in *some* circumstances, and
we shouldn't encourage or support it. The readtable interferences will
only get worse at the REPL, when a modified system is re-compiled with
readtable modifications from a system that it doesn't depend on, and
the next time you try to load it from clean, you find it mysteriously
depends on the other system.

> As I understand it, the proposed change will break this kind of system.
>  Indeed, I think it's likely to break the parent common base system as
> well as breaking the system composition (the use of the readtable is not
> confined to a single file in the common base, and I believe it straddles
> multiple subsystems, which are represented as individual ASDF systems).
>
Yes, it's a feature to break this kind of system, so as to enable
instead systems where the programmer can safely rely on his system not
being either victim or author of interference with other systems, and
may both rely on a sane readtable and be able to modify his readtable
safely.

But that there are systems like that suggests that the change should
not be made without a lot of heads up, and so definitely not for
3.1.1.

> I'm not going to defend the structure of this code, but I strongly feel
> that it is not the job of ASDF to impose a personal opinion that this
> coding pattern is bad, and should be *forced to break* when previously
> it worked perfectly adequately.
>
It did not work "perfectly". It happened to work because you control
the build process to make sure you don't interfere with other systems,
and ASDF de facto imposes on general purpose libraries that they can't
safely do anything fancy with readtables, which sucks.

To allow libraries at large to safely enjoy localized syntax
modification at the cost of requiring a little bit of discipline from
a handful of dirty systems sounds like a pretty good deal to me, and
totally the job of ASDF. Without ASDF, there's no way the libraries
will be safe by default. Once again, the example of a random
load-system at the REPL causing havoc because I had syntax
modifications at the REPL is pretty dreadful.

> To the best of my knowledge, the readtable manipulation functions in CL
> don't provide any easy replacement for the pattern of coding applied here.
>
named-readtables and cl-syntax are there to help.
ASDF could and in my opinion should make things safe by default
instead of unsafe by default.

> Yes, one *can* set a global to the special readtable, and then one can
> write a special PERFORM method (one? or many?) that will bind
> *READTABLE* to the value of this global.
>
Yes, and many systems do that, including ironclad.
Actually, inspecting the ironclad failure in cl-test-grid, I find that
the problem was one with my code, whereby my :around method was run
inside ironclad's :around method rather than outside, causing the
breakage. I will implement a better protocol, and see how much
breakage that causes.

> This is already a monumental PITA, since the PERFORM method will have to
> refer to a variable named in a package that doesn't exist at the time
> the ASDF system is defined.  It also forces our poor programmer to know
> more than s/he ever wanted to know about ASDF, and IMO inappropriately
> bleeds details of the code of the system into the system's build file.
>
The referring to the variable can be done through the now usual "read
the string late" method as long used to name component classes and
functions and recently operation classes.

I don't see how this "bleeds details about ASDF". Hacking readtables
was always tricky. Up until now, it is unsafe to modify the global
readtable, and for that reason, library authors are strongly
recommended to not do it, while application authors do it at their own
risk. With some help from ASDF, it can be made safe for everyone, and
those application authors who used to do it the ugly way can still do
it at the expense of one call to in-readtable or similar.

> I believe that the proposed hygiene practice *also* leaves the
> programmer in an uncomfortable position if s/he incrementally compiles
> functions inside an editor buffer, since the readtable will not be
> properly set.  I.e., AFAICT, although careful binding of the readtable,
> rather than treating it as a global variable, feels like the Right
> Thing, it has undesirable consequences because we don't have good ways
> to bind the readtable around the editor.  If a -*- READTABLE: foo; -*-
> file header worked, this would be less of a concern, but using a header
> like that requires named readtables.
>
No, it's the CURRENT practice that makes it unsafe for everyone to
compile files,
because that will be done with the *current* readtable,
which will be completely inappropriate for files in a different system.
I've been bitten, hard, by that.

With my proposed changes (see what I pushed in branch standard-syntax),
programmers can recompile a file and that will happen automatically in
the correct readtable.
(Though they will need a properly updated swank-asdf, if using SLIME.)

> My conclusion is that enforcing some sort of compartmentalization of
> readtable is not only an inappropriate imposition of a particular style,
> but *also* means we are hanging the poor programmer out to dry.  CL
> simply doesn't provide good tools for manipulating the readtable in the
> way we are proposing to enforce.
>
> IMO, named readtables are the way to go, but they are not a common
> feature of the language, and the portable library is bit-rotted.*
>
> I will not be pushing this proposed modification into the next release,
> and it will take a lot of convincing for me ever to incorporate it into
> ASDF.
>
Yes, CL does provide the tools, we were just not using them.
and *that* was hanging the poor programmer out to dry.
By implementing per-system syntax, we're providing them the hygiene
they sorely need.
People who were previously leaking readtable state will have it
private to their system;
if they were leaking readtable state from one system to the next, that
won't work anymore,
but that was already a hazard to systems they were not writing,
so we're trading a "that's not working for another poor sod
who can't know and can't do anything about it"
to a "that's not working for someone doing something ugly
who should know better and can do something about it".
And that was the guiding principle of ASDF2:
"he who knows is who specifies the information, he who doesn't know
doesn't have to do anything".

> *This is partly my fault, but the current disarray of cl.net, the
> library's position as part of an abortive EDITOR-HINTS master system,
> and the previous maintainer's putting it into a revision control system
> (darcs) that I don't use and don't understand are additional
> contributing factors.
>
If you're the new maintainer, I suggest converting the darcs
repository to git and maybe putting the library on github.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
The aim of public education is not to spread enlightenment at all; it is simply
to reduce as many individuals as possible to the same safe level, to breed and
train a standardized citizenry, to put down dissent and originality.
        ― H.L. Mencken