[iterate-devel] Iterate in-hashtable expansion, with-hashtable-iterator vs. mapha sh

[Hoehle, Joerg-Cyril]

Hi,

first, some good news: I think I found an easy way to make Iterate support nested in-hashtable, together with with-hashtable-iterator:
Just map with-hashtable-iterator to walk-cddr-with-declarations in *special-form-alist*
That should basically work good enough for a start, but it does not walk the hashtable parameter.

Next: a possible change -- what do you think?
Some packages like SCREAMER have explicitly chosen to support Iterate instead of loop because they walk code and suffer from similar limitations about macrolet as Iterate does.
In the past, Iterate never expanded to forms that Screamer was unable to analyse.
Starting with Andreas Fuchs' change, he made IN-HASHTABLE expand to with-package-iterator, which is based on MACROLET. Similarly for in-package.
As a result, Screamer would error out should it be updated to use one of ours iterate.lisp.

Therefore, I'm thinking about introducing a choice, either as a *settable-variable*, or as new iteration keywords, for how to Iterate through hashtables (and packages).
I first thought about
in-hashtable: use with-hashtable-iterator
on-hashtable: use maphash to build up list upfront.

Then I realized that it should rather be the other way round, or somebody would have to modify the Screamer examples, and all other unknown packages built using Screamer (and possibly Series, I don't know).

An alternative may be to get Screamer support macrolet, but that's IMHO even farther away than having Iterate do that.

So what do you prefer?
a) in-hashtable -> maphash, on-hashtable -> with-*-iterator
   and same for packages
  "reintroduce old behaviour"
b) on-hashtable -> maphash, in-hashtable -> with-*-iterator
   and same for packages
  "affirm changed behaviour, need to change random packages"
c) defvar iterate::*use-with-iterators*
   Screamer could set such a variable, or at least the examples should do so (since only the examples depend on Iterate, not Screamer itself)

I'm biased towards b). Most users will benefit from the iterator, and I'd need to convince the Screamer authors anyway to package a more recent version of Iterate with it, so the examples could be updated at that time, or to at least have the debian cl-screamer package not include but depend on cl-iterate (and similarly for cl-pdf etc.). Still, the users would have to change their Screamer code.

(and no, I haven't yet tested Screamer with my version of Iterate, this is just code review and thinking loud, yet I expect no problems beyond that).

BTW, having a maphash-based iterator allows to modify any hashtable item while iterating it, instead of just the current one (forbidden by ANSI-CL and many other language specifications). That can come handy sometime.

While we are at it, there's a difference between the old in-package and the new (since Andreas Fuchs in 2004): the new (with-package-iterator based) one does not evaluate the :external-only argument any more -- How I hate it when documentations does not precisely state what is evaluated and what not! it always makes simple examples work and complex ones fail.
With-package-iterator does not provide for evaluation of this, while the old in-hashtable dispatched to either do-symbols or do-external-symbols.
So far, I plan to stick to that change, or does anybody wave hands?

Regards,
	Jörg Höhle.