From pc at p-cos.net Sat Feb 7 14:47:23 2009 From: pc at p-cos.net (Pascal Costanza) Date: Sat, 7 Feb 2009 15:47:23 +0100 Subject: [closer-devel] ContextL News (incl. first-class dynamic environments!!!) Message-ID: Hi, I have had some working code for first-class dynamic-environments for a few weeks by now, but have only come around to releasing it to the ContextL darcs repository today. The reason is that this is a relatively major change in the implementation of ContextL, so I wanted to be extra careful (also with regard to coming up with an interface to dynamic environments that I think is ready for prime time). I am not going to make this an official release yet, because there may still be feedback from people who want to try the new features out. So I would like to leave some time for integrating such feedback before finalizing things. Apart from the first-class dynamic environments, there are two more important changes that may or may not affect ContextL users: - I have changed the implementation for ensuring internal unique names so that it doesn't use a "stop the world" synchronization lock anymore, to make it work better on multicore architectures. However, this means that ContextL now depends on trivial-garbage (which is asdf- installable). - I have removed the asdf definitions for tweaking the layer garbage collector, and also changed the other ways of removing optional features to a more uniform model. See contextl.asd for more details. From now on, ContextL doesn't by itself add anything to *features* anymore, but you can add keywords to *features* to remove optional features from ContextL before compiling it. (You can remove the support for first-class dynamic environments, the extra checks when rebinding special places, and the layer garbage collector.) There is now also a testsuite specifically for Clozure Common Lisp. Now to the really interesting stuff, the first-class dynamic environments: The core of the extension are the operators dynamic-wind, capture- dynamic-environment, with-dynamic-mark, and with-dynamic-environment. Dynamic-wind can be used to add a certain action to the current dynamic environment, delimited by a nested proceed form. Here is an example: (dynamic-wind (handler-case (proceed (do some stuff)) (error () ...) (warning () ...))) This example sets up exception handlers in the dynamic environment for the code enclosed in the proceed form. If somewhere in its dynamic extent some code calls capture-dynamic-environment (that therefore dynamically closes over this particular piece of code), then all the actions except what's in the proceed form are captured in the respective representation of the dynamic environment. Whenever such a representation of the dynamic environment is reestablished (using with- dynamic-environment), the exception handlers are set up again for the new code. So for example: (defun foo () (capture-dynamic-environment)) (setq *env* (dynamic-wind (handler-case (proceed (foo)) (error () (format t "I caught an error."))))) (with-dynamic-environment (*env*) (error "boo!")) This will catch the error "boo!" with the exception handler that prints "I caught an error.", because the with-dynamic-environment form reestablished the exception handler from the previous form. You can also understand this as if the with-dynamic-environment form is replaced by the following code: (handler-case (error "boo!") (error () (format t "I caught an error."))) ...that is, everything from the dynamic environment is called, except the proceed forms which are replaced by the body of with-dynamic- environment. However, the code that is executed by a with-dynamic- environment is not determined statically but depends on the dynamic value of the respective environment object, which in general can be an arbitrary long chain of dynamic winds all nested inside each other. The portion of the dynamic environment that is captured can be delimited by a dynamic mark. For example: (defvar *mark*) (defun foo () (dynamic-wind (print 'foo) (proceed (bar)))) (defun bar () (with-dynamic-mark (*mark*) (baz))) (defun baz () (dynamic-wind (print 'baz) (proceed (bam)))) (defun bam () (capture-dynamic-environment *mark*)) (setq *env* (foo)) (with-dynamic-environment (*env*) (print 'duh)) This prints BAZ and DUH, but not FOO because it is not in the part delimited by *mark*. Such delimited dynamic environments should be especially useful in conjunction with partial continuations: You mark the code that is the top of the part of the computation that will be captured in a partial continuation, and when you actually capture the continuation, you also capture the dynamic environment up until that mark. You then only need your own way of 'bundling' both the continuation and the dynamic environment. [Since ContextL doesn't provide support for continuations itself, ContextL doesn't do this for you.] with-dynamic-mark can bind both lexical and special variables, so it can be used both for lexically and dynamically scoped partial continuations. (This part of the design is highly inspired by http://okmij.org/ftp/Computation/dynamic-binding.html#DDBinding ) More often than not, dynamic-wind is too low-level. To abstract over dynamic-wind, ContextL provides a reimplementation of special variables and special places in terms of dynamic-wind so that they are automatically captured with capture-dynamic-environment. The same is true for layer activations and deactivations. For special variables, I have followed the design of dynamic-variables in ISLISP. So instead of using defvar, defparameter, let and let*, you have to use defdynamic, the 'dynamic' operator for looking up values, and dynamic-let and dynamic-let* (as well as dlet and dlet* as abbreviations). I'm not doing this to annoy you ;), but providing new binding forms is actually necessary to be able to hide the implementation details when expressing these things in terms of dynamic-wind (but it is true that I like ISLISP's design for dynamic variables a lot). Here is a very simple example: (defdynamic value 42) (defun foo () (print (dynamic value))) (defun bar () (dynamic-let ((value 4711)) (foo))) (foo) => 42 (bar) => 4711 In conjunction with dynamic environments, it is important to be able to make a particular distinction in dynamic binding forms: whether you want a binding to be recomputed every time a dynamic environment is reestablished, or whether you want exactly the same binding. Therefore, the binding forms exist in two different versions: dynamic- relet and dynamic-relet* (and their abbreviations drelet and drelet*) will recompute bindings every turn, and dynamic-let and dynamic-let* (and their abbreviations dlet and dlet*) will use exactly the same bindings. Here are two examples to illustrate the different semantics. First for recomputation semantics: (defdynamic value '(d e f)) (setq *env* (dynamic-relet ((value (append '(a b c) (dynamic value)))) (capture-dynamic-environment))) (with-dynamic-environment (*env*) (dynamic value)) => '(a b c d e f) (setf (dynamic value) '(4 5 6)) (with-dynamic-environment (*env*) (dynamic value)) => '(a b c 4 5 6) The reason for the different results is that for each reestablishment of *env*, the append of the respective lists will recomputed over and over again for determining the binding of (dynamic value). Now an illustration for reusing the same binding: (defdynamic value 0) (setq *env* (dynamic-let ((value (incf (dynamic value)))) (capture-dynamic-environment))) (dynamic value) => 1 (with-dynamic-environment (*env*) (dynamic value)) => 1 (dynamic value) => 1 (with-dynamic-environment (*env*) (incf (dynamic value))) => 2 (dynamic value) => 1 ; !!! it's a different binding !!! (with-dynamic-environment (*env*) (dynamic value)) => 2 ; !!! old binding reestablished !!! So it's indeed the case that with dynamic-let and dynamic-let*, the bindings will be captured in the dynamic environments, not the values. This also means that if you pass a dynamic environment to a different thread in a multithreaded system and reestablish it there, you may have shared state between different threads whose accesses you may have to synchronize accordingly (by using locks, or so). [A design question I am not 100% sure about at the moment: In the current version, defdynamic maps to defparameter internally, and there is no equivalent form for defvar. I don't know whether both are necessary, and/or which one should be the default one. I'm interested in opinions here.] ContextL provides a first-class representation of such dynamic variables called dynamic symbols, which you can use like regular Common Lisp symbols. However, since they are _not_ regular Common Lisp symbols, you have to use dynamic-symbol-value, dynamic-symbol-boundp, dynamic-symbol-makunbound, etc., instead of symbol-value, boundp, makunbound, etc. You can check whether something is a dynamic symbol with dynamic-symbol-p. The rebinding forms are dynamic-progv (for binding-reuse semantics) and dynamic-reprogv (for binding-recompute semantics). The special symbols to express special places are now essentially also such dynamic symbols. with-active-layers and with-inactive-layers are now also expressed in terms of dynamic-wind, and they use the binding-recompute semantics. So whenever you say (with-active-layers (l1) ...) and capture a dynamic environment in the dynamic extent of this layer activation, layer l1 will be reactivated on top of any other active layers whenever the respective dynamic environment is reestablished using with-dynamic-environment. Likewise for (with-inactive-layers (l1) ...) - which ensures that l1 will be inactive whenever the respective dynamic environment is reestablished. You can take a look at all these higher-level operators (dynamic-let, dynamic-relet, dynamic-progv, dynamic-reprogv, with-activer-layers and with-inactive-layers) to see how you can implement such macros yourself on top of dynamic-wind. Special attention should be payed to giving the local proceed operators generated names [thanks a lot to Tobias, this proved to be an essential ingredient in the implementation!!!]. Here is template how to do this: (dynamic-wind :proceed my-proceed ... (my-proceed ...) ...) Here, my-proceed is a name for the proceed form given by the user, which can be a gensymmed name in macro definitions. If you define your own proceed name, ContextL's 'proceed cannot be used for that particular dynamic-wind anymore (to be able to have nested dynamic- winds). There is a test suite for dynamic environments called dynenv.lisp which may throw more light on the semantics of the new features. If for some reason, you don't want to use first-class dynamic- environments, you can push :cx-disable-dynamic-environments to *features* before compiling ContextL. You can then still use dynamic- wind and with-dynamic-environment (which are then essentially noops), but not capture-dynamic-environment or with-dynamic-environment. Dynamic symbols will then be plain, uninterned Common Lisp symbols, and dynamic-let and dynamic-relet will be shallow wrappers on top of Common Lisp's special variable binding forms (and not have different semantics anymore, of course ;). If it is indeed the case that you want to disable first-class dynamic environments, I would like to know about it, so please contact me in that case. Ah, a final note, to avoid confusions: ContextL's dynamic-wind is inspired by Scheme's dynamic-wind, but apart from the fact that ContextL's dynamic-wind is not thrown together with call/cc, there is also the important distinction that ContextL's dynamic-wind doesn't perform an unwind-protect automatically. If you want that, you have to say something like this: (dynamic-wind (unwind-protect (proceed ...) (clean-up))) [I strongly believe in not overloading language constructs with too many semantics at the same time, so I prefer to keep the unwind protection separate from the rest of the dynamic environment semantics.] OK, if you have any questions, please shoot! Pascal -- ELS'09: http://www.european-lisp-symposium.org/ Pascal Costanza, mailto:pc at p-cos.net, http://p-cos.net Vrije Universiteit Brussel Programming Technology Lab Artificial Intelligence Lab Pleinlaan 2, B-1050 Brussel, Belgium