[iterate-devel] Macroexpansion, bindings and information flow (Iterate package)

[Andreas Fuchs]

Today, Joerg-Cyril Hoehle <Joerg-Cyril.Hoehle at t-systems.com> wrote:
> Hi,
>
> you're invited to look up comp.lang.lisp on this subject where I
> wrote:
>
> "I believe in its over 10 years of existence, it's never been able
> to correctly handle this form: (macrolet ((over(x) `(collect ,x)))
> (iterate
> 	(for n in '(1 2 3))
> 	(flet ((over(x)(declare (ignore x)) (collect 2)))
> 	  (over n)))) ; would yield (2 2 2) if correct
> One reason is that it does not contain a code walker aware of lexical
> bindings. ..."
> [BTW, I added that to iterate-test.lisp :)]

Great! (-:

> I believe it's possible to add restricted support for macrolet. The
> restriction is that the macro expanders neither contain nor expand
> into iterate clauses (the body may contain clauses).  This would
> mean that Iterate would play nice with uses of macrolet which are
> independent on it, e.g. hashtable iterators.

Right. Restricted macrolet support for stuff like
e.g. with-hashtable-iterator (which doesn't expand into iterate
clauses) would be a sensible valuable thing to have.

I did the non-sensible thing instead and hacked together a more or
less macrolet/flet/labels-aware code walker that uses custom lexical
environments. It gives the right answer with your MACROLET/FLET
example, and the right answer if you swap the FLET/MACROLET
forms. with-hashtable-iterator seems to work with sbcl, cmucl, clisp
and ACL (personal edition 6.x), as do a few other examples I tried.

The whole thing is a bit ugly, but for the macrolets I tried it with,
it seems to work. Here's a diff (against 1.0.9):

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ,macrolets.diff
Type: application/octet-stream
Size: 10510 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/iterate-devel/attachments/20041115/2585c874/attachment.obj>
-------------- next part --------------

(The code really isn't beautiful. Most of it could stand being
improved in some places. But as a proof of concept, it seems to work
pretty well. I thought it would be much harder to get this to work)
(-:

> Together with support for locally (and fixing some more subtle bugs
> I did not yet talk about), I believe Iterate would be more valuable
> if
>
> a) it accepted macrolet instead of giving an error
> b) there would be some restrictions
> b2) Iterate would issue (warn ...) upon macrolet
> c) the above shows that it is hard to have Iterate generate correct
>    code in all cases, but users can hopefully live with that.

I'm not entirely sure that mine is the way to go. But yours (as the
more conservative one) sounds good, from a maintainer's point of view.

> My fear is that the Iterate user may not realize that some
> situations are deadly. E.g. I've often thought about an
> implementation style as follows: (defmacro loop-finish () (error
> "Use of LOOP-FINISH outside LOOP")
> ;; or `(error ...))
> (defmacro loop (&body)
> `(macrolet (loop-finish () `(go loop-end))
> ...))
> The idea is to provide both global and local macro definitions. The global one is solely to ease debugging.
>
> Iterate would generate incorrect code when this loop definition
> would be used somewhere inside the iterate body. A real problem is
> that the user may not even notice that this situation arises.
> 
> What do you think?

With working custom environments, this doesn't seem like a big
problem. The problem lies in the OAO20TM nature of the whole custom
environment thing. I guess we should give your 

> Notice that correct macrolet handling requires correct lexical
> parsing. In my posting to comp.lang.lisp I'm asking how/whether this
> can be portably done.

A hacky solution isn't hard. One that is guaranteed to work with all
macrolet/flet combinations (recognizing only macro lambda lists, for
example) will be quite a bit harder. But that's the problem with code
walkers /-:

-- 
Andreas Fuchs, <asf at boinkor.net>, asf at jabber.at, antifuchs