[closer-devel] ContextL News (incl. first-class dynamic environments!!!)

Sat Feb 7 14:47:23 UTC 2009

Hi,

I have had some working code for first-class dynamic-environments for  
a few weeks by now, but have only come around to releasing it to the  
ContextL darcs repository today. The reason is that this is a  
relatively major change in the implementation of ContextL, so I wanted  
to be extra careful (also with regard to coming up with an interface  
to dynamic environments that I think is ready for prime time).

I am not going to make this an official release yet, because there may  
still be feedback from people who want to try the new features out. So  
I would like to leave some time for integrating such feedback before  
finalizing things.

Apart from the first-class dynamic environments, there are two more  
important changes that may or may not affect ContextL users:

- I have changed the implementation for ensuring internal unique names  
so that it doesn't use a "stop the world" synchronization lock  
anymore, to make it work better on multicore architectures. However,  
this means that ContextL now depends on trivial-garbage (which is asdf- 
installable).

- I have removed the asdf definitions for tweaking the layer garbage  
collector, and also changed the other ways of removing optional  
features to a more uniform model. See contextl.asd for more details.  
 From now on, ContextL doesn't by itself add anything to *features*  
anymore, but you can add keywords to *features* to remove optional  
features from ContextL before compiling it. (You can remove the  
support for first-class dynamic environments, the extra checks when  
rebinding special places, and the layer garbage collector.)

There is now also a testsuite specifically for Clozure Common Lisp.

Now to the really interesting stuff, the first-class dynamic  
environments:

The core of the extension are the operators dynamic-wind, capture- 
dynamic-environment, with-dynamic-mark, and with-dynamic-environment.  
Dynamic-wind can be used to add a certain action to the current  
dynamic environment, delimited by a nested proceed form. Here is an  
example:

(dynamic-wind
   (handler-case
        (proceed (do some stuff))
     (error () ...)
     (warning () ...)))

This example sets up exception handlers in the dynamic environment for  
the code enclosed in the proceed form. If somewhere in its dynamic  
extent some code calls capture-dynamic-environment (that therefore  
dynamically closes over this particular piece of code), then all the  
actions except what's in the proceed form are captured in the  
respective representation of the dynamic environment. Whenever such a  
representation of the dynamic environment is reestablished (using with- 
dynamic-environment), the exception handlers are set up again for the  
new code. So for example:

(defun foo () (capture-dynamic-environment))

(setq *env* (dynamic-wind
               (handler-case (proceed (foo))
                 (error () (format t "I caught an error.")))))

(with-dynamic-environment (*env*)
   (error "boo!"))

This will catch the error "boo!" with the exception handler that  
prints "I caught an error.", because the with-dynamic-environment form  
reestablished the exception handler from the previous form.

You can also understand this as if the with-dynamic-environment form  
is replaced by the following code:

(handler-case (error "boo!")
   (error () (format t "I caught an error.")))

...that is, everything from the dynamic environment is called, except  
the proceed forms which are replaced by the body of with-dynamic- 
environment. However, the code that is executed by a with-dynamic- 
environment is not determined statically but depends on the dynamic  
value of the respective environment object, which in general can be an  
arbitrary long chain of dynamic winds all nested inside each other.

The portion of the dynamic environment that is captured can be  
delimited by a dynamic mark. For example:

(defvar *mark*)

(defun foo ()
   (dynamic-wind (print 'foo) (proceed (bar))))

(defun bar ()
   (with-dynamic-mark (*mark*) (baz)))

(defun baz ()
   (dynamic-wind (print 'baz) (proceed (bam))))

(defun bam ()
   (capture-dynamic-environment *mark*))

(setq *env* (foo))

(with-dynamic-environment (*env*)
   (print 'duh))

This prints BAZ and DUH, but not FOO because it is not in the part  
delimited by *mark*. Such delimited dynamic environments should be  
especially useful in conjunction with partial continuations: You mark  
the code that is the top of the part of the computation that will be  
captured in a partial continuation, and when you actually capture the  
continuation, you also capture the dynamic environment up until that  
mark. You then only need your own way of 'bundling' both the  
continuation and the dynamic environment. [Since ContextL doesn't  
provide support for continuations itself, ContextL doesn't do this for  
you.] with-dynamic-mark can bind both lexical and special variables,  
so it can be used both for lexically and dynamically scoped partial  
continuations. (This part of the design is highly inspired by http://okmij.org/ftp/Computation/dynamic-binding.html#DDBinding 
  )

More often than not, dynamic-wind is too low-level. To abstract over  
dynamic-wind, ContextL provides a reimplementation of special  
variables and special places in terms of dynamic-wind so that they are  
automatically captured with capture-dynamic-environment. The same is  
true for layer activations and deactivations.

For special variables, I have followed the design of dynamic-variables  
in ISLISP. So instead of using defvar, defparameter, let and let*, you  
have to use defdynamic, the 'dynamic' operator for looking up values,  
and dynamic-let and dynamic-let* (as well as dlet and dlet* as  
abbreviations). I'm not doing this to annoy you ;), but providing new  
binding forms is actually necessary to be able to hide the  
implementation details when expressing these things in terms of  
dynamic-wind (but it is true that I like ISLISP's design for dynamic  
variables a lot).

Here is a very simple example:

(defdynamic value 42)

(defun foo () (print (dynamic value)))

(defun bar () (dynamic-let ((value 4711)) (foo)))

(foo) => 42
(bar) => 4711

In conjunction with dynamic environments, it is important to be able  
to make a particular distinction in dynamic binding forms: whether you  
want a binding to be recomputed every time a dynamic environment is  
reestablished, or whether you want exactly the same binding.  
Therefore, the binding forms exist in two different versions: dynamic- 
relet and dynamic-relet* (and their abbreviations drelet and drelet*)  
will recompute bindings every turn, and dynamic-let and dynamic-let*  
(and their abbreviations dlet and dlet*) will use exactly the same  
bindings.

Here are two examples to illustrate the different semantics. First for  
recomputation semantics:

(defdynamic value '(d e f))

(setq *env* (dynamic-relet ((value (append '(a b c) (dynamic value))))
               (capture-dynamic-environment)))

(with-dynamic-environment (*env*) (dynamic value)) => '(a b c d e f)

(setf (dynamic value) '(4 5 6))

(with-dynamic-environment (*env*) (dynamic value)) => '(a b c 4 5 6)

The reason for the different results is that for each reestablishment  
of *env*, the append of the respective lists will recomputed over and  
over again for determining the binding of (dynamic value).

Now an illustration for reusing the same binding:

(defdynamic value 0)

(setq *env* (dynamic-let ((value (incf (dynamic value))))
               (capture-dynamic-environment)))

(dynamic value) => 1

(with-dynamic-environment (*env*) (dynamic value)) => 1

(dynamic value) => 1

(with-dynamic-environment (*env*) (incf (dynamic value))) => 2

(dynamic value) => 1 ; !!! it's a different binding !!!

(with-dynamic-environment (*env*) (dynamic value)) => 2 ; !!! old  
binding reestablished !!!

So it's indeed the case that with dynamic-let and dynamic-let*, the  
bindings will be captured in the dynamic environments, not the values.  
This also means that if you pass a dynamic environment to a different  
thread in a multithreaded system and reestablish it there, you may  
have shared state between different threads whose accesses you may  
have to synchronize accordingly (by using locks, or so).

[A design question I am not 100% sure about at the moment: In the  
current version, defdynamic maps to defparameter internally, and there  
is no equivalent form for defvar. I don't know whether both are  
necessary, and/or which one should be the default one. I'm interested  
in opinions here.]

ContextL provides a first-class representation of such dynamic  
variables called dynamic symbols, which you can use like regular  
Common Lisp symbols. However, since they are _not_ regular Common Lisp  
symbols, you have to use dynamic-symbol-value, dynamic-symbol-boundp,  
dynamic-symbol-makunbound, etc., instead of symbol-value, boundp,  
makunbound, etc. You can check whether something is a dynamic symbol  
with dynamic-symbol-p. The rebinding forms are dynamic-progv (for  
binding-reuse semantics) and dynamic-reprogv (for binding-recompute  
semantics). The special symbols to express special places are now  
essentially also such dynamic symbols.

with-active-layers and with-inactive-layers are now also expressed in  
terms of dynamic-wind, and they use the binding-recompute semantics.  
So whenever you say (with-active-layers (l1) ...) and capture a  
dynamic environment in the dynamic extent of this layer activation,  
layer l1 will be reactivated on top of any other active layers  
whenever the respective dynamic environment is reestablished using  
with-dynamic-environment. Likewise for (with-inactive-layers (l1) ...)  
- which ensures that l1 will be inactive whenever the respective  
dynamic environment is reestablished.

You can take a look at all these higher-level operators (dynamic-let,  
dynamic-relet, dynamic-progv, dynamic-reprogv, with-activer-layers and  
with-inactive-layers) to see how you can implement such macros  
yourself on top of dynamic-wind. Special attention should be payed to  
giving the local proceed operators generated names [thanks a lot to  
Tobias, this proved to be an essential ingredient in the  
implementation!!!]. Here is template how to do this:

(dynamic-wind :proceed my-proceed
   ... (my-proceed ...) ...)

Here, my-proceed is a name for the proceed form given by the user,  
which can be a gensymmed name in macro definitions. If you define your  
own proceed name, ContextL's 'proceed cannot be used for that  
particular dynamic-wind anymore (to be able to have nested dynamic- 
winds).

There is a test suite for dynamic environments called dynenv.lisp  
which may throw more light on the semantics of the new features.

If for some reason, you don't want to use first-class dynamic- 
environments, you can push :cx-disable-dynamic-environments to  
*features* before compiling ContextL. You can then still use dynamic- 
wind and with-dynamic-environment (which are then essentially noops),  
but not capture-dynamic-environment or with-dynamic-environment.  
Dynamic symbols will then be plain, uninterned Common Lisp symbols,  
and dynamic-let and dynamic-relet will be shallow wrappers on top of  
Common Lisp's special variable binding forms (and not have different  
semantics anymore, of course ;). If it is indeed the case that you  
want to disable first-class dynamic environments, I would like to know  
about it, so please contact me in that case.

Ah, a final note, to avoid confusions: ContextL's dynamic-wind is  
inspired by Scheme's dynamic-wind, but apart from the fact that  
ContextL's dynamic-wind is not thrown together with call/cc, there is  
also the important distinction that ContextL's dynamic-wind doesn't  
perform an unwind-protect automatically. If you want that, you have to  
say something like this:

(dynamic-wind
   (unwind-protect
        (proceed ...)
     (clean-up)))

[I strongly believe in not overloading language constructs with too  
many semantics at the same time, so I prefer to keep the unwind  
protection separate from the rest of the dynamic environment semantics.]

OK, if you have any questions, please shoot!

Pascal

-- 
ELS'09: http://www.european-lisp-symposium.org/

Pascal Costanza, mailto:pc at p-cos.net, http://p-cos.net
Vrije Universiteit Brussel
Programming Technology Lab
Artificial Intelligence Lab
Pleinlaan 2, B-1050 Brussel, Belgium