[Ecls-list] Announce: A new C++ based implementation of Common Lisp based on the ECL CL code

Tue Mar 5 18:15:06 UTC 2013

Juanjo,

To summarize - I needed a Common Lisp implementation that interfaced well with C++ so I wrote one rather than use an existing CL implementation.  
By "interface well" I mean interface functions are generated automatically and much of the horribleness that is C++ is handled by template programming. 
I don't want to argue about whether implementing my own CL was a good or bad choice because I did it and I learned an awful lot.

I borrow a lot of Common Lisp code from the ECL distribution and I think I have something to give back to ECL in the interests of cooperation (C++ integration and LLVM-JIT compilation).  The LLVM-JIT compilation builds on the C++ integration.

I'm going to keep working on it and I'm going to release it because there are several people on #lisp who have expressed an interest in it and I'd love to get more advice and input from the CL community on how to improve it.

If you would like to integrate it into ECL (the C++ integration and LLVM-JIT compilations) I'm happy to help out.

On Mar 5, 2013, at 7:59 AM, Juan Jose Garcia-Ripoll <juanjose.garciaripoll at gmail.com> wrote:

> On Tue, Mar 5, 2013 at 1:33 AM, Christian Schafmeister <chris.schaf at verizon.net> wrote:
> I needed a CL interpreter which would host the compiler (which I wrote in Common Lisp) which calls the LLVM-IR library to generate llvm::Module objects which contains llvm::Function objects that are Just-In-Time compiled into machine code and called via C function pointers from the Common Lisp or C++ code. 
> I did not want to write a Common Lisp compiler in C++ (shudder).
> This project started as an archaic Sexp walking Lisp interpreter that grew into a very slow Common Lisp Sexp walking interpreter within which I wrote a reasonably fast self-hosting Common Lisp compiler that generates LLVM-IR.
> 
> It would have been easier to rely on ECL's interpreter to do that. It is very small and the bytecodes interpreter already may run _all_ of Common Lisp at a reasonable speed.
>  

Probably, but I didn't know how to use the ECL byte-code interpreter so I wrote my own.
Yours is quite a bit faster than mine and is undoubtably more robust.
As far as I can tell mine is complete albeit slow but I only need mine to bootstrap the compiler.
Since my compiler has JIT compilation I do compile-in-eval once it boots and then everything gets a lot faster.

> I didn't know that when I started. I know that now - the bridge doesn't require the compiler and LLVM backend 
> - the CL/C++ bridge is all C++ template programming, lots and lots of template programming. 
> It's styled after the boost::python library if you are familiar with it.
> Once I started writing the compiler and exposing and using the LLVM backend I just kept going.
> Although, having intimate familiarity with the Common Lisp implementation I wrote has greatly facilitated the development of the bridge.
> 
> I did not know about Boost::Python. On reading it, it seems that, though horrible, it would be easy to port to any Common Lisp out there. If I were to write that, though, I would not do it as they do, with such undecipherable template code -- templates perhaps yes, but at least attempting something more readable.

C++ template programming is not pretty but using template programming hands the problem of identifying C++ types and parsing C++ code to the C++ compiler where it is done properly. The other approach is to create something like SWIG, which attempts to parse C (it does a good job) and C++ (it does a lousy job) and then generate interface code.

For a comparison of the boost::python (template programming) approach and the SWIG parse/code-generation approach see: https://dev.lsstcorp.org/trac/wiki/SwigVsBoostPython

>  
> We could get it to work with ECL as well if you want it - although it's going to take some work.
> 
> As I said, I believe this can be ported to _any_ Common Lisp. 

"Can be ported" - certainly. These are Turing complete languages and we can do anything computable within them. It's a question of how much time porting requires.  
I don't think it will be much trouble to port what I've done to ECL - other CL's… I don't know anything about their internals.
Doing it the SWIG way - that's a different question.

But there is a reason no one has done this (interfaced the more complicated language features of C++ and CL) before.
There is also a reason why SWIG exists and why boost::python exists.
I've taken the boost::python approach to interfacing C++ to CL - this has not been done before.

>  
> Then we could incorporate the LLVM-IR generating compiler into ECL.
> 
> I would love this, but I would love it even more if the LLVM-IR used the current compiler -- we do some pretty nifty optimizations and type inference there which any too straightforward LLVM implementation will miss.
>  
> 1) C++ classes behave like CL structures.
> 
> They are opaque, then, not exposing any inheritance? Or do you allow class B which inherits from A to use the same functions as A? Do you expose methods from inherited classes with the same name, as in Python, or with different names, as in SWIG?

Using my (boost::python-like) approach C++ classes have either single or multiple inheritance, the inheritance structure is duplicated in CL.
Methods are exposed once with one name, in the lowest level class that implements them and they are inherited by every class that derives from that class.

>  
> 3) Memory management is currently handled using reference counted C++ shared_ptr/weak_ptr.  I plan to add mark-and-sweep GC later.
> 
> Even with shared_ptr there are problems. Note the examples of SWIG, where you have class class A { class B *b; } and then return field "b" of A to Common Lisp. How do you handle ownership there? Boost::Python is mind bloging and too ugly / fragile in this respect.

Memory management is complicated, there is no way around that. 
If I return a shared_ptr from C++ to CL, it is no trouble as all objects in CL are handled with shared_ptrs. 
They are reference counted and destructed when their ref. count goes to zero.
If I return a raw pointer from C++ to CL it is wrapped in a class that is told what to do with that pointer when the wrapper class is destructed.  It can be told to leave it alone or own it and destroy it when the wrapper is destructed.

> 
> Best,
> 
> Juanjo
> 
> -- 
> Instituto de Física Fundamental, CSIC
> c/ Serrano, 113b, Madrid 28006 (Spain) 
> http://juanjose.garciaripoll.googlepages.com

Cheers,

.Chris.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/ecl-devel/attachments/20130305/5b269e7c/attachment.html>