[cffi-devel] offer of code contribution - with-array-as-foreign-pointer

[JTK]

Thanks for all your links.  I'm learning a lot.  

> You may find Tamas' experiences with various Lisp implementations useful:
> 
> http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.html


This is indeed interesting, but the idea of keeping track of all these implementations is very daunting.
That's CFFI's job ;)


> Some Lisp implementations have an alternative to pinning arrays, which
> is instead to allocate the array in memory that the GC promises never
> to move ("static arrays").  The following library wraps that
> functionality:
> 
> http://gitorious.org/iolib/static-vectors/

That is especially interesting; I wish that I had know about it.  After looking at SBCL's source,
It appears that SBCL does real pinning, not just suspending GC (at least for x86, the only
platform where you worry about threads).


> My sense is that this is the right way to go: it's safer (no calling
> malloc() and free()), and it's more scalable in parallel (no need to
> interrupt or pause the GC). 



Here's my argument why copying to malloc'ed memory  is the most sensible compromise,
after messing with this issue for a few years (and passing Lisp arrays directly):

* it uses the hard work that the authors of CFFI have done to understand various Lisp 
   implementations.   CFFI has a broad user base and is here to stay.    Things like  
   gitorious.org/iolib/static-vectors/ and FFA are impressive,  but they depend on one or two 
   dedicated people  digging through many compilers' source code and often inferring 
   unpublished internals.     
 
*  It's what the Lisp compiler writers expect you to do, by providing an FFI and malloc.

*  In an age of 64 bit machines, insufficient address space for malloc should be much less 
    of a problem.    I just allocated and freed 1 billion bytes on my laptop in both CCL and SBCL.

*  The penalty of copying isn't too bad.  Copying is O(N), and the things you do in foreign memory are usually
    O(N^a) or AxO(N) for A>>1.   For instance, matrix inversion is N^(3/2) for an N element array.
    The exception might be some sort of  very simple IO with no processing, but why is that being done in FFI-land?    

*  If a implementation is slow to copy into foreign memory, then it will be slow
    for everything else, and the copying will be the least of one's worries.    




-John