[slime-devel] [investigation] Font lock, reader macros and SLIME repl

[Anton Kovalenko]

Hello SLIME developers,

Both in the lisp-mode and in slime REPL, there is currently one major
thing that Emacs isn't getting right: constructs using reader macros
like #P"/home/" and #2A((1 0) (0 1)) are regarded as two sexps instead
of one. E.g. command for wrapping a sexp may turn #P"/home/" into
(#P)"/home/", etc. It gets even more awful with paredit-mode (that I
prefer to use), but the problem is noticeable even without it. 

I have investigated one possible solution to it, and here are the
results.

First, I decided to limit the scope of solution to some predefined
"shape" of reader macro: for starters, when #\# is used as dispatch
macro character, and its reader function calls READ to get the sexp
following #[numarg]<char>. This is the case with all standard
reader-macros and many implementation-specific and user-defined ones.
Furthermore, I discarded the possibility to consult SWANK for the
buffer's *readtable* (SWANK doesn't *know* it, anyway).

To get right behavior for #P, #2A and the like, the most appropriate
syntax for all macro-characters seems to be `expression prefix': we want
Emacs to regard the whole #P thing as it would regard a quote, backquote
or comma.

Let's use the feature of font-lock mode in modern emacsen: syntactic
keywords (in a nutshell, it's when a code used for text-coloring is
[ab]used to add syntax-table properties to some text fragments).
`parse-sexp-lookup-properties' set to true causes the sexp machinery
to look into syntax-table properties of individual characters.

Let's add a syntactic keyword to lisp-mode:

(defun aak:add-lisp-reader-macros-syntactic-keyword ()
  "Register # numarg macro-character as a font lock syntactic
keyword, turning it into expression prefix."
  (set (make-local-variable 'parse-sexp-lookup-properties) t)
  (set (make-local-variable 'font-lock-syntactic-keywords)
       '(("\\(\\W\\|$\\)\\#\\([0-9]*[A-Za-z]\\)" (2 "'")))))

(add-hook 'lisp-mode-hook 'aak:add-lisp-reader-macros-syntactic-keyword)

With the hook above, any lisp-mode buffer with font-lock mode enabled
gets good enough navigation/wrapping/unwrapping for previously
misinterpreted items.

I decided to leave macro-characters handled by this regexp to [A-Za-z],
because normal behavior of reading the following sexp can't really be
expected from other characters: readtable additions along the lines
of #"something" and #{something} are also popular, and we can't get
_them_ right without knowing a concrete syntaxt they expect. Another
decision that has its merits is to limit it further to _standard_
macro-characters only.   

Now I wanted the same thing to be active for slime-repl mode. The only
obstacle was that slime-repl mode is currently font-lock-unfriendly:
when some faces are set programmatically and some left to font-lock, the
programmatically-set faces should use font-lock-face property instead of
face property (the latter gets overridden by font-lock results).

A workaround that I use currently is the following advice to
add-text-properties:

(defadvice add-text-properties
  (before aak:slime-propertize-with-font-lock-face activate)
  (when (eq 'slime-repl-mode major-mode)
    (let* ((props (ad-get-arg 2))
	   (face (getf props 'face)))
      (when face
	(push face props)
	(push 'font-lock-face props)
	(when (memq 'face (getf props 'rear-nonsticky))
	  (push 'font-lock-face (getf props 'rear-nonsticky)))
	(ad-set-arg 2 props)))))
        
When in slime-repl mode, it copies face property to font-lock-face
property, and also makes it rear-nonsticky when face is requested to
be. It's the only part that really touches SLIME, and I think it would
better insert font-lock-face property itself, together with face property
(don't forget rear-nonstickyness). This duplication (face =>
font-lock-face) does no harm when font-lock mode is disabled, but
enables slime-repl buffer to be used with font-lock enabled as well. 

Another hook to enable font-lock mode in slime buffers:

(defun aak:enable-font-lock-for-slime-repl ()
  "Enable font lock mode in slime REPL buffer."
  (font-lock-fontify-buffer) ;; don't know why it's needed..
  (font-lock-mode 1))

(add-hook 'slime-repl-mode-hook
          'aak:add-lisp-reader-macros-syntactic-keyword)
(add-hook 'slime-repl-mode-hook
	  'aak:enable-font-lock-for-slime-repl)

NB. The problem that I'm trying to solve is even more important for
slime-repl buffers, where incorrect sexp wrapping tends to break
_presentations_ apart. 

-- 
Regards, Anton Kovalenko
+7(916)345-34-02 | Elektrostal' MO, Russia