[climacs-devel] Number recognition in the Lisp syntax module

[Troels Henriksen]

Climacs' Lisp syntax module does not recognize numbers such as "2" or
"3.17", though it does recognize them when written as "#2r1" or other
complex forms. Instead, the `lex-token' function returns
`complete-token-form's, where it should return `number-lexeme'. I
think this slightly modified definition of `lex-token' will work:

(defun lex-token (scan)
  ;; May need more work. Can recognize symbols and numbers.
  (flet ((fo () (forward-object scan)))
    (let ((could-be-number t)
          sign-seen dot-seen slash-seen)
      (flet ((return-token-or-number-lexeme ()
               (return-from lex-token
                 (if could-be-number
                     (make-instance 'number-lexeme)
                     (make-instance 'complete-token-lexeme))))
             (this-object ()
               (object-after scan)))
        (tagbody
         START
           (when (end-of-buffer-p scan)
             (return-token-or-number-lexeme))
           (when (constituentp (object-after scan))
             (cond ((or (eql (this-object) #\+)
                        (eql (this-object) #\-))
                    (when sign-seen
                      (setf could-be-number nil))
                    (setf sign-seen t))
                   ((eql (this-object) #\.)
                    (when dot-seen
                      (setf could-be-number nil))
                    (setf dot-seen t))
                   ((eql (this-object) #\/)
                    (when slash-seen
                      (setf could-be-number nil))
                    (setf slash-seen t))
                   ((not (digit-char-p (this-object)))
                    (setf could-be-number nil)))
             (fo)
             (go START))
           (when (eql (object-after scan) #\\)
             (fo)
             (when (end-of-buffer-p scan)
               (return-from lex-token (make-instance 'incomplete-lexeme)))
             (fo)
             (go START))
           (when (eql (object-after scan) #\|)
             (fo)
             (return-from lex-token (make-instance 'multiple-escape-start-lexeme)))
           (return-token-or-number-lexeme))))))

I am not very familiar with writing lexers, so I'm going to put it
here before committing. It seems to work properly for me, so unless
someone points out issues, or I discover any problems, I'll commit it
in a day or two.
-- 
\  Troels "Athas" Henriksen
/\ - Insert witty signature