[slime-devel] [investigation] Font lock, reader macros and SLIME repl

[Terje Norderhaug]

On Aug 31, 2010, at 2:18 AM, Tobias C Rittweiler wrote:
> In article <87pqx0kmga.fsf at desktop-ng.home.sw4me.com>,
> Anton Kovalenko <anton at sw4me.com> wrote:
> 
>> Hello SLIME developers,
>> 
>> Both in the lisp-mode and in slime REPL, there is currently one major
>> thing that Emacs isn't getting right: constructs using reader macros
>> like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead
>> of one. E.g. command for wrapping a sexp may turn #P"/home/" into
>> (#P)"/home/", etc. It gets even more awful with paredit-mode (that I
>> prefer to use), but the problem is noticeable even without it. 
>> 
>> I have investigated one possible solution to it, and here are the
>> results.

I recall we've discussed this issue in the past.

Rather than making emacs (as swank client) recognize the bounds of the lisp expression, swank could cover this functionality on the server. That is, emacs sends the expression to the swank server and gets back information about the boundaries of the expression. 

For example, in the case of a reader macro:

1. Emacs streams the reader macro as characters to the swank server, continuing beyond the end of the expression.
2. The swank server parses from the stream until the reader reaches the end of the expression.
3. The server notifies emacs via swank how many characters were read when reaching the end of the expression.
4. Emacs uses the information about how many characters were read to determine when the expression ended.

Using the inferior lisp to determine expression bounds has several benefits, including:

1.  Emacs doesn't have to be configured to know the specific lisp syntax.
2. Clojure, Scheme and other lisp dialects can be supported with no fuss.
3. SLIME can handle custom reader macros with ease.
4. No additional configuration required to cover custom reader macros.

-- Terje

>> First, I decided to limit the scope of solution to some predefined
>> "shape" of reader macro: for starters, when #\# is used as dispatch
>> macro character, and its reader function calls READ to get the sexp
>> following #[numarg]<char>. This is the case with all standard
>> reader-macros and many implementation-specific and user-defined ones.
>> Furthermore, I discarded the possibility to consult SWANK for the
>> buffer's *readtable* (SWANK doesn't *know* it, anyway).
>> 
>> To get right behavior for #P, #2A and the like, the most appropriate
>> syntax for all macro-characters seems to be `expression prefix': we want
>> Emacs to regard the whole #P thing as it would regard a quote, backquote
>> or comma.
>> 
>> Let's use the feature of font-lock mode in modern emacsen: syntactic
>> keywords (in a nutshell, it's when a code used for text-coloring is
>> [ab]used to add syntax-table properties to some text fragments).
>> `parse-sexp-lookup-properties' set to true causes the sexp machinery
>> to look into syntax-table properties of individual characters.
>> 
>> Let's add a syntactic keyword to lisp-mode:
>> 
>> (defun aak:add-lisp-reader-macros-syntactic-keyword ()
>>  "Register # numarg macro-character as a font lock syntactic
>> keyword, turning it into expression prefix."
>>  (set (make-local-variable 'parse-sexp-lookup-properties) t)
>>  (set (make-local-variable 'font-lock-syntactic-keywords)
>>       '(("\\(\\W\\|$\\)\\#\\([0-9]*[A-Za-z]\\)" (2 "'")))))
>> 
>> (add-hook 'lisp-mode-hook 'aak:add-lisp-reader-macros-syntactic-keyword)
> 
> Yes, I've been using
> 
> (defun enable-cl-syntax ()
>  (make-local-variable 'font-lock-syntactic-keywords)
>  (set (make-local-variable 'parse-sexp-lookup-properties) t)
>  (let ((regexp (regexp-opt '("#*" "#." "#=" "#A" "#C" "#P" "#S"))))
>    (pushnew `(,regexp 0 "'") font-lock-syntactic-keywords :test 
> #'equal)))
> 
> 
> (add-hook 'lisp-mode-hook #'enable-cl-syntax)
> 
> for a long time now.
> 
>> With the hook above, any lisp-mode buffer with font-lock mode enabled
>> gets good enough navigation/wrapping/unwrapping for previously
>> misinterpreted items.
>> 
>> I decided to leave macro-characters handled by this regexp to [A-Za-z],
>> because normal behavior of reading the following sexp can't really be
>> expected from other characters: readtable additions along the lines
>> of #"something" and #{something} are also popular, and we can't get
>> _them_ right without knowing a concrete syntaxt they expect. Another
>> decision that has its merits is to limit it further to _standard_
>> macro-characters only.   
> 
> There could be a SWANK:DECLAIM-SYNTAX along SWANK:DECLAIM-INDENTATION
> which people could put into their CL source files.
> 
>  -T.
> 
> 
> _______________________________________________
> slime-devel site list
> slime-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/slime-devel