[pro] Compiler APIs in Common Lisp

[Pascal Costanza]

On 12 Apr 2011, at 10:46, Mark Cox wrote:

> On 12/04/2011, at 7:45 AM, Matthew Swank wrote:
> 
>> I am current reading an interesting paper on Racket's macro system (referenced
>> here: http://lambda-the-ultimate.org/node/4196
>> ), and some of the objects they expose quack like CLTL-2 Environments:
>> http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html. I have read that
>> having write access to environments makes it hard for the compiler to optimize
>> code. However, even portable read only access would be helpful. Does anyone
>> besides me and Cyrus Harmon (http://cyrusharmon.org/blog/display?id=113)
>> miss Environments?
> 
> Alas, I wish I had been exposed to them before. Until Cyrus' post I had no knowledge of them. I thank Cyrus for making that post. Using the function CLTL2 VARIABLE-INFORMATION function, I produced the following higher level functionality as a proof of concept for an idea I have for a mathematical optimisation library.

[...]

> Given the possibilities it offers, I am interested to know why it isn't part of the standard. 

There are several possible answers to this. Ideally, you would want a data structure that is based either on structs or on CLOS classes that gives you information about the lexical environment. However, the compiler must exist already at a very early phase of bootstrapping a Lisp system, and the more advanced types are probably not ready yet at that stage. Secondly, there are quite a few namespaces (not just 2) that all need to handled in some way in lexical environments. Ensuring a good API to inspect all of them was apparently not trivial. Thirdly, different compilers have different ways how to organize their compilation phases - making them all agree on how to provide the information is maybe also not trivial.

And then, from a different angle, Common Lisp already provides environment objects and local macro definitions, and they already allow you to go a very long way. For example, I have found the discussion at http://www.lispworks.com/documentation/HyperSpec/Issues/iss066_w.htm very illuminating. (See the two local-type-declare examples - I particularly like the version using symbol macros.)

I have implemented an experimental hygienic-compatible macro system for Common Lisp using just these ingredients. See http://dx.doi.org/10.3217/jucs-016-02-0271 for a description.

I would strongly recommend considering using such techniques, instead of holding your breath and waiting for wide adoption of fully specified lexical environments. ;) (Just to avoid misunderstandings: such environments would definitely be more expressive than just opaque environment objects + local macros, it's just that environment objects + local macros are already a lot more powerful than some may expect!)


Pascal

--
Pascal Costanza
The views expressed in this email are my own, and not those of my employer.