[climacs-cvs] CVS update: climacs/Doc/climacs-internals.texi

[Robert Strandh]

Update of /project/climacs/cvsroot/climacs/Doc
In directory common-lisp.net:/tmp/cvs-serv9534

Modified Files:
	climacs-internals.texi 
Log Message:
Documented the incremental lexer protocol.

Date: Tue Mar 15 07:19:23 2005
Author: rstrandh

Index: climacs/Doc/climacs-internals.texi
diff -u climacs/Doc/climacs-internals.texi:1.16 climacs/Doc/climacs-internals.texi:1.17
--- climacs/Doc/climacs-internals.texi:1.16	Sat Mar  5 12:53:52 2005
+++ climacs/Doc/climacs-internals.texi	Tue Mar 15 07:19:21 2005
@@ -718,6 +718,117 @@
 
 @section Incremental parsing framework
 
+ at deftp {Protocol Class} parse-tree
+
+The base class for all parse trees. 
+ at end deftp
+
+We use the term parse tree in a wider sense than what is common in the
+parsing literature, in that a lexeme is a (trivial) parse tree.  The
+parser does not distinguish between lexemes and other parse trees, and
+a grammar rule can produce a lexeme if that should be desired. 
+
+ at deffn {Generic Function} {start-offset} parse-tree
+
+The offset in the buffer of the first character of a parse tree.  
+ at end deffn
+
+ at deffn {Generic Function} {end-offset} parse-tree
+
+The offset in the buffer of the character following the last one of a
+parse tree.
+ at end deffn
+
+The length of a parse-tree is thus the difference of its end offset
+and its start offset.
+
+The start offset and the end offset may be NIL which is typically the
+case when a parse tree is derived from the empty sequence of lexemes.
+
+ at subsection Lexical analysis
+
+ at deftp {Protocol Class} lexer
+
+The base class for all lexers.
+ at end deftp
+
+ at deftp {initarg} :buffer
+
+Associate a buffer with a lexer
+ at end deftp
+
+ at deffn {Generic Function} {buffer} lexer
+
+Return the buffer associated with the lexer
+ at end deffn
+
+ at deftp {Class} incremental-lexer
+
+A subclass of lexer which maintains the buffer in the form of a
+sequence of lexemes that is updated incrementally. 
+ at end deftp
+
+In the sequence of lexemes maintained by the incremental lexer, the
+lexemes are indexed by a position starting from zero. 
+
+ at deffn {Generic Function} {nb-lexemes} lexer
+
+Return the number of lexemes in the lexer. 
+ at end deffn
+
+ at deffn {Generic Function} {lexeme} lexer pos
+
+Given a lexer and a position, return the lexeme in that position in
+the lexer. 
+ at end deffn
+
+ at deffn {Generic Function} {insert-lexeme} lexer pos lexeme
+
+Insert a lexeme at the position in the lexer.  All lexemes following
+POS are moved to one position higher. 
+ at end deffn
+
+ at deffn {Generic Function} {delete-invalid-lexemes} lexer from to
+
+Invalidate all lexemes that could have changed as a result of
+modifications to the buffer 
+ at end deffn
+
+ at deffn {Generic Function} {inter-lexeme-object-p} lexer object
+
+This generic function is called by the incremental lexer to determine
+whether a buffer object is an inter-lexeme object, typically
+whitespace. Client code must supply a method for this generic
+function. 
+ at end deffn
+
+ at deffn {Generic Function} {skip-inter-lexeme-objects} lexer scan
+
+This generic function is called by the incremental lexer to skip
+inter-lexeme buffer objects.  The default method for this generic
+function increments the scan mark until the object after the mark is
+not an inter-lexeme object, or until the end of the buffer has been
+reached. 
+ at end deffn
+
+ at deffn {Generic Function} {update-lex} lexer start-pos end
+
+This function is called by client code as part of the buffer-update
+protocol to inform the lexer that it needs to analyze the contents of
+the buffer at least up to the END mark of the buffer.  START-POS is
+the position in the lexeme sequence at which new lexemes should be
+inserted. 
+ at end deffn
+
+ at deffn {Generic Function} {next-lexeme} lexer scan
+This generic function is called by the incremental lexer to get a new
+lexeme from the buffer.  Client code must supply a method for this
+function that specializes on the lexer class.  It is guaranteed that
+scan is not at the end of the buffer, and that the first object after
+scan is not an inter-lexeme object.  Thus, a lexeme should always be
+returned by this function. 
+ at end deffn
+
 @subsection Earley parser
 
 Climacs contains an incremental parser that uses the Earley