[cffi-devel] offer of code contribution - with-array-as-foreign-pointer
Mark Hoemmen
mark.hoemmen at gmail.com
Sun Sep 12 02:21:16 UTC 2010
On Sat, Sep 11, 2010 at 19:22, JTK <jetmonk at gmail.com> wrote:
>>
>> You might be disappointed (or pleased, perhaps) to see that someone
>> (Tamas K. Papp, in particular) has already done this:
>>
>> http://tkpapp.blogspot.com/2007/12/announcement-ffa.html
>
> I feel a bit chagrined that I didn't find Papp's AAF before putting a lot of effort into
> my WAAF.
>
> I've looked at the differences between FFA and my WAAF.
>
> From my admittedly quick reading of the docs, FFA passes Lisp arrays to
> foreign land rather than copying for SBCL and maybe other Lisps. This involves
> suspending garbage collection. On the balance, I think that in an era of multithreaded
> 64 bit Lisps and relatively abundant memory, it is better not to mess with Lisp's
> internals (like suspending GC), and it is safer to allocate foreign memory and
> copy.
Allocating foreign memory is what Tamas calls "Approach 1" in this
blog post. Rif's "fnv" (foreign numeric vectors) package (to which
Tamas links in that post) does exactly this: space for vectors and
matrices is only allocated in foreign memory, so you know the GC won't
move it around or collect it. If I recall correctly, Tamas chose his
Approach 2 because (a) supposedly, some Lisp implementations severely
restrict the available heap for malloc() calls, and (b) Tamas was
mainly interested in SBCL, on which Approach 2 is faster.
> eg, SBCL will pin arrays if you ask it, but I understand that this is done by
> globally suspending GC, which is not necessarily good when other threads are running.
>
> In fact, I was forced to write this after a multithreaded SBCL application froze mysteriously
> in foreign code with GC suspended.
You may find Tamas' experiences with various Lisp implementations useful:
http://tkpapp.blogspot.com/2010/05/upgraded-array-element-types-and-pinned.html
Some Lisp implementations have an alternative to pinning arrays, which
is instead to allocate the array in memory that the GC promises never
to move ("static arrays"). The following library wraps that
functionality:
http://gitorious.org/iolib/static-vectors/
My sense is that this is the right way to go: it's safer (no calling
malloc() and free()), and it's more scalable in parallel (no need to
interrupt or pause the GC). If you don't allocate your arrays in that
way, it's reasonable to ask for payment of a performance tax (either
copying into foreign memory, or pausing the GC). Anyway, that's just
my opinion; I definitely don't speak for CFFI or any Lisp
implementation :-)
mfh
More information about the cffi-devel
mailing list