[Asdf-devel] syntax control

[Faré]

On Wed, Apr 2, 2014 at 9:06 PM, Robert P. Goldman <rpgoldman at sift.info> wrote:
> Faré wrote:
>> compile-file and load already bind *readtable*, which means that for
>> asdf itself to bind *readtable* should be a no-op in the common case,
>> and a BIG save for those who want to switch the readtable at the REPL.
>
> I'm puzzled by this claim.  As actually implemented, this seems to me to
> tend to *force* procedural interaction with the *READTABLE*, in ways
> that you deplore.  But perhaps I misunderstand your proposed revision.
>
Well, yes, and that's what you said I had to preserve backward
compatibility with, isn't it?

This branch
1- *documents* the previously implicit manual hygiene requirements for
shared readtable usage
2- *enforces* that the same readtable supporting CL + shared
extensions is used for every build,
  independently from what the user uses at the REPL, which now can be
completely different.

The hygiene requirements can't be enforced without breaking backward
compatibility. But we already know that in practice, people already
make do, and now there is a documented guideline.To enable a read-only
readtable (on SBCL, CCL and other supported implementations), use in
the syntax-control branch:
(setf asdf:*cl-reading-hook* 'uiop:call-with-standard-io-syntax)


> If the programmer wants his/her changes to the readtable to persist over
> a session, in the absence of an easy way to switch readtables (e.g., by
> named-readtables and a READTABLE: head-of-file directive as there is in
> ACL's ELI, and I believe there was in Zmacs), there isn't a pleasant
> alternative to destructively modifying the value of *READTABLE* that
> will be carried over multiple calls to COMPILE-FILE and LOAD.
>
Yes, and this will work with my branch. Indeed, since *READTABLE* is
bound by COMPILE-FILE and LOAD, any such escaping change is a
side-effect to the existing *readtable* datastructure as supplied by
ASDF, and not due to a new datastructure being bound to *READTABLE*.

Therefore, all ASDF has to do is remember the readtable that was used
while loading ASDF itself, and make sure it is always reused while
compiling CL code. Unless you switch the *READTABLE*, which you can't
do from within an ASDF system (at least not without advanced tricks),
the whole thing is thus a nop. And from those who *do* switch the
*READTABLE* at the REPL, which supposes incompatibility, we really do
not want to use such incompatible readtable, and do want the readtable
bound back to normal when re-compiling.

Those who *really* want to trick ASDF into using an incompatible
readtable will have to reconfigure ASDF to store output files
somewhere else, anyway, and binding the asdf:*shared-readtable* will
be a minor detail along the way — modifying output-translations would
probably be important there.

> But my understanding of your proposal is that you will somehow bind the
> readtable to a *clean* readtable, rather than the current readtable, so
> the behavior you propose will be radically different from the behavior
> specified by the standard.  Is that not so?
>
Not so much *clean* than *shared*. Or if "clean", staying so by every
programmer respecting the now documented hygiene discipline.

> Consider the case where there's no ASDF (as at the time of the CL
> standardization), and no named-readtables.  What's the programmer to do
> aside from picking a readtable and mangling it?  I suppose the
> super-wizard could switch back and forth, but nothing in the spec seems
> to make this at all pleasant.
Well, I've been there, with Scribble, Exscribe, and fare-quasiquote.
In the beginning, indeed, I modified the global readtable.
Then I learned good practices, and instead stored my readtable in a variable,
and did an
  (eval-when (:compile-toplevel :execute) (setf *readtable* *my-readtable*))
in every file that needed it.
These days, I just use named-readtables for the same general effect.
It's not super-pleasant indeed, but it works...
except that if I do it at the REPL then use load-system,
it can pollute my output translation cache, and then I have to remove it.

> In practice, my experience is that if,
> e.g., you were working in Nisp, you just switched to the Nisp readtable
> for your whole session, and that was it.  So the Nisp system definition
> would copy the readtable, change *READTABLE* to point to the copy, and
> then following systems (which would be written in Nisp, rather than
> vanilla CL), would get the readtable in a smooth way.
>
No, you can't change the binding of *READTABLE* in a LOAD or COMPILE-FILE,
because CL binds it around these two functions. You must either side-effect
the shared readable, or use eval-when to select another one. Or with ASDF,
you can have a perform around wrapper select a different readtable
before you load.

With my patch, there are better entry points for such wrappers,
and a default wrapper that ensures all builds get the same shared CL readtable
that is not any of the incompatible readtable,
in particular not the fare-quasiquote readtable.

> Also, COMPILE-FILE binds the readtable to the *current value* of the
> readtable.  This lets us copy it if we want, but given that the behavior
> of SET-MACRO-CHARACTER is to destructively modify its readtable argument
> (which may be *READTABLE*), I don't see that this gives any precedent to
> the proposed syntax hygiene.
>
> If COMPILE-FILE bound *READTABLE* to a *COPY* of the current readtable,
> then I would see the precedent, but as it is, I'm still not convinced.
>
The "hygiene" for the shared readtable is a matter of following these
conventions,
which are the current docstring for uiop:*shared-readtable*:

  This shared readtable allows legacy applications to keep modifying a
global shared readtable
  while maintaining some hygiene for those who want to use their own readtable.
  It is subject to the following restrictions, which always existed
but were previously implicit:
  A- no modifying any standard character,
  B- no two dependencies assigning different meaning to the same
non-standard character.
    Using any non-standard character while expecting the
implementation to treat it some way
    counts as such an assignment of meaning.
  C- libraries need to document these assignments of meaning to
non-standard characters.
  D- free software libraries will register these changes on:
        http://www.cliki.net/Macro%20Characters

> Indeed, I am only more convinced that the proposed behavior is a
> departure from the current practice, and so should be signaled by some
> expressed desire for strictness, rather than being the default.
>
Since no one can possibly modify the shared readtable binding from a
loaded or compiled file,
it is by construction compatible with current practice to force this
binding at the beginning of the build.
The big win is that it becomes safe to bind the *readtable* at the REPL,
when it is today completely unsafe.

The only incompatible case is when you'd modify the *readtable* to
something incompatible,
e.g. that has a CL-in-CL implementation,
then point the output translations to a different place,
then run ASDF again in this crazy setup and expect it to work.
To achieve the same effect, you'll now have to also bind *shared-readtable*
to your magic CL-in-CL readtable.
I don't believe anyone is doing this thing right now,
and if they are, having to bind *shared-readtable* will be a small
additional cost.


> Note that I'm not arguing that syntax hygiene is a Bad Thing, only that
> it's not vanilla CL, so should not be the behavior you get from vanilla
> CL.  By analogy, hygienic macros are arguably a Good Thing, but we keep
> CL macros unhygienic and make you GENSYM your way to hygiene if you want
> it.  I could imagine a library that exports a hygienic variant of
> DEFMACRO, but I couldn't imagine making CL:DEFMACRO hygienic by default.
>
The thing with DEFMACRO is that it only affects people who use it.
A bad *READTABLE* also pollutes people who don't use it.
Binding *readtable* to *shared-readtable* saves users who bind it at the REPL,
while being 100% backward compatible with those who modify the
readtable structure.
I think this should be the default, because it has great positive
impact on those
who have a non-standard *readtable* at the REPL, while having no
imaginable impact
on anyone else.

My patch also makes it trivial to make the readtable read-only instead,
or a private copy for each system, or whatever you want.
But I agree none of these can be the default, because of backward compatibility.

And whereas I'd like to bind syntax variables beside the *readtable*,
I accept that this is not going to happen now or in the forseeable future,
but I also make it trivial to implement, for those who want to try.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
As the Chinese say, 1001 words is worth more than a picture.  — John McCarthy