[pro] Lisp and DSLs

[Alessio Stalla]

On Wed, Jul 20, 2011 at 3:32 PM, Didier Verna <didier at lrde.epita.fr> wrote:
>
>  Dear friends,
>
> I'm starting to write a chapter for an upcoming book on domain specific
> languages. The chapter is called (tentatively):
>
> Extensible languages -- blurring the distinction between DSLs and GPLs
>
> GPL meaning General Purpose Language in this context ;-)
>
>
> My intention is to demonstrate how the task of implementing a DSL is
> made easier when it boils down to an extension or subset of your
> original GPL (hence reusing its infrastructure), instead of being a
> totally different language, only written on top of the other.
>
> Obviously, I'm going to illustrate this with Common Lisp, and I intend
> to speak of dynamicity (not only dynamic typing, but in general all
> things that can be deferred to the run-time), introspection,
> intersession, structural or procedural reflexivity, meta-object
> protocols (not sure about this one), macro systems and JIT-compilation.
> Also, more specifically to Lisp, reader macros (compiler macros maybe?),
> the condition system (and its ability to *not* unwind) and restarts.
>
>
> Right now, I would like to know if any of you have DSL "pearls", nice
> examples of DSLs that you have written in Lisp by using some of its
> features in a clever or elegant way. I would also gladly accept any
> point of view or comment on what's important to mention, in terms of
> design principle or anything else, things that I may have missed in the
> list above.

Not really "pearls", but lately I found myself using more and more of
Lisp at macroexpansion time, which seems to be what you're looking
for. Two examples:

 - for my DO+ (doplus) iteration macro - a somewhat iterate-like
construct with a simpler implementation that does not use a code
walker - I used structures quite heavily. do+ controls iteration via a
series of clauses that are macros (either built-in or written by
users) that have a peculiar aspect: they don't expand to proper Lisp
code, but return structures or lists of structures [1]. do+
macroexpands its clauses (recursively when needed) and uses the
returned structures to drive the building of its own expansion.
Basically I explicitly used macros as compile-time functions, defining
my own limited "AST" (not really a tree but a flat list of structures)
which is not Lisp's but is later translated to Lisp, so it's one extra
indirection than what is typically found in macros. I think this shows
nicely how the distinction between code and data is blurry in Lisp,
and as a consequence the distinction between DSL and GPL is blurry as
well. This allows the implementation to remain simple: do+ is a little
less than 350 lines of code, but only roughly half of them are the
actual macro and its supporting code, the rest are built-in clause
macros and other minor stuff.

 - I also have sketched a HTML generation library - tentatively called
tag-pool - where I used CLOS generic functions at macroexpansion time
to drive the expansion of tag macros to HTML-outputting code. Contrary
to the popular approach, used by CL-WHO among others, to represent
HTML as a limited sexp-based DSL where tag names are keywords and a
single macro does all the translation work, I have one macro per tag.
For CLOS dispatch purposes, to each tag is also associated a class,
but it's used only at expansion time, the output code does not
instantiate any CLOS object. Tag macros follow a CLOS-based protocol
to emit the various bits of HTML like attributes, body, and so on.
Users can specialize methods that are part of the protocol to finely
control how tag macros are translated, for example providing defaults
for certain attributes, or post-processing some attribute's value,
etc. As an example, the SCRIPT macro translates its Lisp body to
JavaScript using Parenscript, by specializing a single generic
function. The body code can itself contain HTML-generating macros and
those are translated to HTML-emitting JavaScript as well. Again, a key
feature of Lisp (CLOS and generic function dispatch) is used at
macroexpansion time to make the implementation of the DSL simpler and
more easily extensible.

You can find both examples here: <http://code.google.com/p/tapulli/>
although only do+ is documented and reasonably complete.

[1] ok, technically a structure *is* proper Lisp code since it is a
self-evaluating object and it can be externalized...

Best,
Alessio