[cdr-discuss] [RfC] WITH-READTABLE-ITERATOR

[Tobias C. Rittweiler]

Request for comments. Including grammatical and stylistic corrections.

  -T.


Issue: WITH-READTABLE-ITERATOR

Forum: Common Lisp Document Repository (CDR)

Status: Draft.

References

   * CLHS 2.1.1 Readtables

   * X3J13 Issue #188

   * `with-package-iterator' (macro)

   * `with-hash-table-iterator' (macro)

Category: Addition.

Edit History: 02-Oct-2008 by Rittweiler (Draft)

Problem Description:

  Even though the ANSI Common Lisp standard provides simple getters for
  readtables (`get-macro-character', `get-dispatch-macro-character'), the
  standard does not provide any means to efficiently get at all the macro
  characters and dispatch macro characters defined in a readtable.

  The omission of any iteration facility for readtables makes readtables
  unneccessarily opaque, and keep users from writing libraries that try to deal
  with readtables in a general way. For example, the author discovered that this
  lack of an iteration form is the main obstacle to writing an otherwise
  portable library that establishes an organizational namespace for readtables
  akin to the namespace that is provided for packages.

Proposal (WITH-READTABLE-ITERATOR:ADD-GENERATOR)

  Add a macro `with-readtable-iterator' that establishes a generator in its
  scope; each invocation of this generator returns a macro character or a
  dispatch macro character from the readtable the generator was established
  for--along side some additional information.

  A detailed specification of `with-readtable-iterator' can be found in the
  appendix of this document.

Rationale

  The proposed macro `with-readtable-iterator' represents a general iteration
  facility for readtables that can be used to implement a variety of iteration
  forms.

  The proposal is closely modelled on the macros `with-hash-table-iterator' and
  `with-package-iterator'.  The semantics of `with-readtable-iterator' should
  thus be intuitive to any experienced Common Lisp programmer.

Notes

  The proposal does deliberately say nothing about the home-package of the
  symbol `with-readtable-iterator'. However, implementors are encouraged to
  export this symbol from their extensions package (often called "`EXT'") or
  another appropriate package--unless a later CDR document specifies a more
  explicit location.

Current Practice

  No implementation the author is aware of provides a way to iterate through a
  readtable.

  The author implemented the proposal for SBCL (http://www.sbcl.org/), and sent
  the relevant patches upstream; the patches are currently waiting to be
  integrated into mainline. Ariel Badichi implemented the proposal for CLISP
  (http://clisp.cons.org/), and is going to send his work upstream shortly.
  Stephen Compall did an implementation for Clozure CL (http://ccl.clozure.com/)
  which needs to be somewhat revised to fully conform to the proposal as
  presented in this document.

Cost to implementators

  The macro `with-readtable-iterator' should be straightforwardly
  implementable. Extrapolating from actual experience, people--who were
  previously not acquainted with the relevant code sections--were able to
  implement it in a couple of hours.

Discussion

  Ariel Badichi proposed coalescing a generator's fourth return value
  (indicating if the returned character is a dispatch macro character) with its
  first return value (indicating if the generator is exhausted.)

  There is technically no reason that speaks against doing so; in fact, a
  generator would return one value less this way--which may lead to positive
  performance characteristics on register-anemic processor architectures.

  Stephen Compall and the author opposed such a change mostly for idiomatic
  reasons, as both `with-hash-table-iterator' and `with-package-iterator', the
  generator-establishing macros specified by the ANSI standard, return a purely
  boolean exhaustion flag as first value. In particular, `with-package-iterator'
  does _not_ coalesce the accessibility type (third return value) with the
  exhaustion flag (first return value.)

  The author notes that allowing `:terminating', and `:non-terminating' as valid
  MACRO-CHAR-TYPES was considered, but rejected for reasons of simplicity. It is
  not apparent that there is a real necessity for supporting these out of the
  box.

Acknowledgements

  The author wants to specially credit Ariel Badichi and Stephen Compall which
  in spirit of true hackerism promptly agreed to hack an early version of the
  proposal into the implementations of their choice.


Appendix

   -- Macro: with-readtable-iterator 
                (name readtable &rest macro-char-types) declaration* form*                                  
              => results

  Arguments and Values
  ....................

  NAME            - A symbol.

  READTABLE       - A form, evaluated once to produce a readtable.

  MACRO-CHAR-TYPE - One of the symbols `:macro-char', or `:dispatch-macro-char'.

  DECLARATION     -  A `declare' expression; not evaluated.

  FORMS           - An implicit progn.

  RESULTS         - The values of the FORMS.


  Description
  ...........

  Within the lexical scope of the body FORMS, the NAME is defined via `macrolet'
  such that successive invocations of `(name)' will return the macro characters,
  one by one, from the READTABLE. The order of the macro characters returned is
  implementation-dependent.

  An invocation of `(name)' returns the following five values:

    1. A generalized boolean that is true if a macro character is returned.

    2. A macro character that is defined in READTABLE.

    3. A reader macro function of the macro character returned.

    4. A generalized boolean that is true if the macro character is a dispatch
       macro character.

    5. An association list between the "sub-characters" of the dispatch macro
       character and their reader macro functions.

  After all macro characters have been returned by successive invocations of
  `(name)', only one value is returned, namely `nil'.

  It is unspecified what happens if any of the implicit interior state of an
  iteration is returned outside the dynamic extent of the
  `with-readtable-iterator' form such as by returning some closure over the
  invocation form

  In spirit of CLHS 3.6, consequences are undefined if READTABLE is modified
  except for modification of the current macro character under traversal.

  Exceptional Situations
  ......................

  Signals an error of type `program-error' if a MACRO-CHAR-TYPE is supplied that
  is not recognized by the implementation.

  See Also
  ........

  Traversal Rules and Side Effects (CLHS 3.6), `with-package-iterator'