[cffi-devel] offer of code contribution - with-array-as-foreign-pointer

[Liam Healy]

You might also want to take a look at foreign-array in Grid Structured
Data (GSD), which uses iolib and is built around Tamas's "AFFI"
(affine).  GSLL uses it extensively, and we are working on porting
other libraries.  Formerly, I had built foreign array access on ffa,
but iolib is more portable.  Also, I've come to think that seeing the
same array from both the Lisp and foreign side is not all that
important; with grid:gref you reference the foreign element just like
aref.  However I can imagine if you use CL array functions and foreign
code together, this would be handy.

http://repo.or.cz/w/gsd.git
http://common-lisp.net/project/gsll/

The gsll-devel mailing list is a good place to bring up issues as this
is under ongoing development.

Liam

On Sat, Sep 11, 2010 at 11:49 PM, JTK <jetmonk at gmail.com> wrote:
>
> Thanks for all your links.  I'm learning a lot.
>
>> You may find Tamas' experiences with various Lisp implementations useful:
>>
>> http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.html
>
>
> This is indeed interesting, but the idea of keeping track of all these implementations is very daunting.
> That's CFFI's job ;)
>
>
>> Some Lisp implementations have an alternative to pinning arrays, which
>> is instead to allocate the array in memory that the GC promises never
>> to move ("static arrays").  The following library wraps that
>> functionality:
>>
>> http://gitorious.org/iolib/static-vectors/
>
> That is especially interesting; I wish that I had know about it.  After looking at SBCL's source,
> It appears that SBCL does real pinning, not just suspending GC (at least for x86, the only
> platform where you worry about threads).
>
>
>> My sense is that this is the right way to go: it's safer (no calling
>> malloc() and free()), and it's more scalable in parallel (no need to
>> interrupt or pause the GC).
>
>
>
> Here's my argument why copying to malloc'ed memory  is the most sensible compromise,
> after messing with this issue for a few years (and passing Lisp arrays directly):
>
> * it uses the hard work that the authors of CFFI have done to understand various Lisp
>   implementations.   CFFI has a broad user base and is here to stay.    Things like
>   gitorious.org/iolib/static-vectors/ and FFA are impressive,  but they depend on one or two
>   dedicated people  digging through many compilers' source code and often inferring
>   unpublished internals.
>
> *  It's what the Lisp compiler writers expect you to do, by providing an FFI and malloc.
>
> *  In an age of 64 bit machines, insufficient address space for malloc should be much less
>    of a problem.    I just allocated and freed 1 billion bytes on my laptop in both CCL and SBCL.
>
> *  The penalty of copying isn't too bad.  Copying is O(N), and the things you do in foreign memory are usually
>    O(N^a) or AxO(N) for A>>1.   For instance, matrix inversion is N^(3/2) for an N element array.
>    The exception might be some sort of  very simple IO with no processing, but why is that being done in FFI-land?
>
> *  If a implementation is slow to copy into foreign memory, then it will be slow
>    for everything else, and the copying will be the least of one's worries.
>
>
>
>
> -John
>
>
> _______________________________________________
> cffi-devel mailing list
> cffi-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
>