[Ecls-list] Announce: A new C++ based implementation of Common Lisp based on the ECL CL code

[Christian Schafmeister]

On Mar 4, 2013, at 4:23 PM, Juan Jose Garcia-Ripoll <juanjose.garciaripoll at gmail.com> wrote:

> On Mon, Mar 4, 2013 at 5:30 PM, Christian Schafmeister <chris.schaf at verizon.net> wrote:
> I've created a new implementation of Common Lisp that has a core written in C++ completely from scratch but hosts the ECL Common Lisp source code.
> 
> Sounds like a lot of work.
>  
> It hosts all of the ECL Common Lisp code in the ecl/src/lsp and ecl/src/clos directories of the ECL source tree.
> I do not use any part of the ECL CL->C compiler or the byte code compiler.
> I've implemented my own CL interpreter and CL->LLVM-IR->native code compiler.
> 
> I do not understand why you reimplemented the interpreter, but the LLVM thingy looks interesting.

I needed a CL interpreter which would host the compiler (which I wrote in Common Lisp) which calls the LLVM-IR library to generate llvm::Module objects which contains llvm::Function objects that are Just-In-Time compiled into machine code and called via C function pointers from the Common Lisp or C++ code. 
I did not want to write a Common Lisp compiler in C++ (shudder).
This project started as an archaic Sexp walking Lisp interpreter that grew into a very slow Common Lisp Sexp walking interpreter within which I wrote a reasonably fast self-hosting Common Lisp compiler that generates LLVM-IR.

>  
> It's purpose is to make possible the seamless interfacing of C++ libraries with Common Lisp code and vice versa.
> Within this new implementation it is trivial to interface complex C++ libraries containing classes, virtual functions, overloaded functions etc with Common Lisp code.
> 
> Sorry, but I do not understand this. There are different things in interfacing Common-Lisp with C++, and none of them requires building a new compiler or a LLVM backend.

I didn't know that when I started. I know that now - the bridge doesn't require the compiler and LLVM backend 
- the CL/C++ bridge is all C++ template programming, lots and lots of template programming. 
It's styled after the boost::python library if you are familiar with it.
Once I started writing the compiler and exposing and using the LLVM backend I just kept going.
Although, having intimate familiarity with the Common Lisp implementation I wrote has greatly facilitated the development of the bridge.

>  
> Why - you ask?  I'd be happy to explain over a beer or coffee sometime.
> In a nutshell - I want this for my research and prior to my writing it there was no Common Lisp implementation that interfaces seamlessly with C++.
> ECL does a great job with "C" but "C++" is a much more complex language to interface with and I have a lot of C++ code that I need to interface with.
> 
> If you state that ECL cannot interface with C++ I would like to see that statement backed with examples. EQL is one example that interfacing is not impossible and, probably, not that hard once you can write C++ inside Common Lisp code.

I did not state that ECL cannot interface with C++ - I know better than that. 

On the other hand, I need it to be very easy to interface CL to C++ because I have a lot of C++ and nothing I saw in the CL world seemed to do the job.

Now if I want to interface a C++ function foo(int x, double y) to CL I add the following code to my C++ code.

af_def("CORE","foo",&foo);

Or the macro:
Defun(foo);

That's it for simple cases.  If the function needs lambda-list processing then it is:
af_def("CORE","foo",&foo,"(x y &optional (z 999))");

That specifies the package, the function-name in CL, the pointer to the function and the lambda-list as a string which is parsed and associated with the wrapper for the function.

If I want to expose the C++ class Foo with method Foo::bar (default lambda-list processing) and Foo::baz (lambda-list processing)  I currently copy-and-paste-and-edit a wrapper class Foo_O into a header file (I should be able to eliminate the need for this with more template programming) and I add the following to my C++ code.

class_<Foo_O>()
    .def("bar",&Foo::bar)
    .def("baz",&Foo::baz,"(a b &key (gamma (+ a b) gamma-p))")
;

That's it for this simple example.  Different function arity, lambda-list processing, argument type-conversion, result-conversion, memory-management, multiple and single inheritance, virtual-function handling is all done automatically with C++ template programming.

We could get it to work with ECL as well if you want it - although it's going to take some work.
Then we could incorporate the LLVM-IR generating compiler into ECL.


> 
> But most important, the issue of interfacing Common Lisp and C++ consists on many different problems that I do not see how you solve at the same time with the help of LLVM
> 
> 1. Integrating the C++ type system and the Common Lisp one
> 2. Being able to call C++ functions from Common Lisp and viceversa
> 3. Integrating C++ and Common Lisp memory management
> 4. Possibly automating the generation of wrappers for C++ code.
> 5. Resolving overloading at runtime instead of at compilation time.
> ...
> 
> Many issues, some of which are mentioned in SWIG, http://www.swig.org/Doc1.3/SWIGPlus.html but even many of which they (working long in the interface between C++ and dynamic languages) do not have an answer to.

SWIG ran out of steam for me pretty early on. 

I've had to develop solutions to all of these problems - so I'm familiar with them.

1) C++ classes behave like CL structures.
2) C++ functions calling CL and vice versa is taken care of.
3) Memory management is currently handled using reference counted C++ shared_ptr/weak_ptr.  I plan to add mark-and-sweep GC later.
4) Virtual functions, regular functions and overloaded functions are all handles. When you have overloaded functions with different types you have to disambiguate them by getting a pointer to each function. There is no way to avoid this.


> Juanjo
> 
> -- 
> Instituto de Física Fundamental, CSIC
> c/ Serrano, 113b, Madrid 28006 (Spain) 
> http://juanjose.garciaripoll.googlepages.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/ecl-devel/attachments/20130304/cf2c8a75/attachment.html>