[armedbear-devel] Question about jnew-runtime-class

Tue Oct 9 13:27:31 UTC 2012

On Mon, Oct 1, 2012 at 11:06 PM, archisman rudra <archi.rudra at gmail.com> wrote:
> Following up on this,
>
> I have some code by modifying some of the call chain for compile,
> jvm-compile etc in compiler-pass2.lisp that lets me generate class files
> where the compiled code for top level functions are stored. I have checked
> that I can load this and call this (and managed to do this without
> initializing the interpreter) from pure java land. One case that is not
> covered is a set of defuns within a let binding, for example
>
> (let (y) (defun get-y() y) (defun set-y (x) (setq y x)) (defun add-x (x) (+
> x y)))
>
> right now compile-to-file only lets me compile these separately, and creates
> separate symbols static fields in each file separately and tries to
> initialize them in the static initialization blocks using a Lisp.recall
> function call. There is something funky going on in the compilation re
> whether this is in memory or not, but each function gets its own class and
> there does not seem an easy way to compile just the let form  in the top
> level.
>
> I will write and tests to see if my changes to jnew-runtime-class can work
> as well, and would like to co-ordinate on how to do more extensive testing
> and merging.
>
> Archi

Hi Archi,

sorry for not following up for a long time. I really appreciate what
you're doing, but unfortunately I have little time to dedicate to ABCL
these days. If you are willing to cook up a patch, I could review it.
Also, I'm rarely on #abcl on IRC, but we can schedule a virtual
meeting there to better discuss the details if you want (and to hear
ideas from other devs as well!).

Cheers,
Alessio

> On Fri, Sep 21, 2012 at 11:52 AM, Alessio Stalla <alessiostalla at gmail.com>
> wrote:
>>
>> On Fri, Sep 21, 2012 at 4:39 PM, archisman rudra <archi.rudra at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > hopefully this is the correct list. If not please let me know.
>>
>> Hi,
>>
>> yes, you are writing to the correct list - welcome!
>>
>> > Background:
>> > I am trying to use abcl to generate jars to send off to a hadoop cluster
>> > for
>> > execution.
>>
>> Interesting! I'm answering because I wrote the current (incomplete)
>> version of jnew-runtime-class. Just FYI, you can find some notes about
>> it at <http://trac.common-lisp.net/armedbear/wiki/JavaFfi/RuntimeClass>.
>>
>> > The svn version of the jnew-runtime-class does not store the
>> > actual method bodies in the generated class file, but rather loads the
>> > generated class file and sets the relevant field. Here is the relevant
>> > code
>> > :
>> >
>> >     (multiple-value-bind (class-file method-implementation-fields)
>> >         (apply #'java::%jnew-runtime-class class-name stream args)
>> >       (sys::put-memory-function memory-class-loader
>> >                                 class-name
>> > (sys::%get-output-stream-bytes
>> > stream))
>> >       (let ((jclass (java:jcall "loadClass" memory-class-loader
>> > class-name)))
>> >         (dolist (method method-implementation-fields)
>> >           (setf (java:jfield jclass (car method)) (cdr method)))
>> >         jclass))))
>> >
>> >
>> > This means that the generated jar file cannot be easily used from
>> > standard
>> > java code.
>>
>> This is a byproduct of the design - although the design is mine, and
>> hasn't been discussed much among the other developers, so I'm not
>> representing anyone but myself. Let me clarify below.
>>
>> > I would like to add some code in the static initialization block
>> > of generated class file to do something equivalent. I will have to save
>> > the
>> > lambda expression for the method and have been experimenting as follows:
>> >
>> > (defvar *cf*)
>> > (defvar *mif*)
>> >
>> > (with-open-file (stream "test.class" :direction :output :element-type
>> > '(unsigned-byte 8))
>> >   (multiple-value-bind (class-file method-implementation-field)
>> >       (java::%jnew-runtime-class "test" stream :methods
>> > `(("forty_two_method1" :int () ,(lambda (this) 42))))
>> >     (setf *cf* class-file)
>> >     (setf *mif* method-implementation-field)))
>> >
>> > (with-open-file (stream "test_forms" :direction :output)
>> >   (dolist (m *mif*)
>> >     (system:dump-form (list (car m) (function-lambda-expression (cdr
>> > m)))
>> > stream)))
>> >
>> >
>> > This assumes that the lambda expression does not refer to any variable
>> > or
>> > user defined function, but perhaps the asdf-jar contrib might help here.
>> >
>> > I was going to write the relevant code and start using it for my
>> > project,
>> > but I was wondering if I am duplicating work that others have already
>> > done
>> > or are doing and whether there are other libraries (jffli perhaps, I am
>> > not
>> > very familiar with it) that have similar functionality. Also if there
>> > are
>> > other interactions with other parts of the system that I am perhaps
>> > missing.
>> >
>> > I would e grateful for any advice.
>>
>> What I wanted to achieve is to have methods that can be:
>>  * arbitrary functions, including closures
>>  * possibly redefined after the class has been loaded. Redefinition is
>> possible by using symbols as method implementations.
>>
>> There's also a hidden goal that is to keep the implementation simple
>> enough, and compiling Lisp code to the body of methods in an arbitrary
>> class is not easy at the moment.
>>
>> What explicitly is NOT a requirement is to have the generated class be
>> independent from ABCL (the methods are implemented in Lisp and will
>> always need runtime support). So, no matter how you arrange things,
>> you will never be able to distribute your class without abcl.jar. Note
>> that it is the same for Clojure, Scala, Groovy, etc.
>>
>> I don't know specifically how Hadoop works, but, as I see things, you need
>> to
>>  1. provide abcl.jar + your class to it
>>  2. have Hadoop execute enough code to load ABCL and initialize your
>> class.
>>
>> Now, probably the key point is who loads your class.
>>  * if you can arrange to have ABCL load it and pass it to Hadoop, you
>> should be good with the current design.
>>  * if Hadoop wants to load it by itself from a jar, then yes, you have
>> to somehow dump the method implementations somewhere and have the
>> class auto-load them from there when it is initialized.
>>
>> For the second case, I propose an intermediate design that is, imho,
>> simpler than dumping arbitrary forms: special-case the situation when
>> a method implementation is a symbol. In that case, directly generate
>> an instruction in the class file to find that symbol in its package
>> and assign it to the appropriate field. If the implementation of a
>> method is not a symbol nothing changes. That way, you will still need
>> to init ABCL before loading your class, but, when all your method
>> implementations are symbols, the class will be able to initialize
>> itself without external support. If some methods are proper functions,
>> instead, the class won't be self-contained, but will need some Lisp
>> code to be executed to be properly initialized.
>>
>> Does it make sense to you? Do you find it acceptable? If yes, I can
>> give a shot at implementing it, if you want.
>>
>> Peace,
>> Alessio
>>
>> --
>> Some gratuitous spam:
>>
>> http://ripple-project.org Ripple, social credit system
>> http://villages.cc Villages.cc, Ripple-powered community economy
>> http://common-lisp.net/project/armedbear ABCL, Common Lisp on the JVM
>> http://code.google.com/p/tapulli my current open source projects
>> http://www.manydesigns.com/ ManyDesigns Portofino, open source
>> model-driven Java web application framework
>
>

-- 
Some gratuitous spam:

http://ripple-project.org Ripple, social credit system
http://villages.cc Villages.cc, Ripple-powered community economy
http://common-lisp.net/project/armedbear ABCL, Common Lisp on the JVM
http://code.google.com/p/tapulli my current open source projects
http://www.manydesigns.com/ ManyDesigns Portofino, open source
model-driven Java web application framework