[cffi-devel] Converting foreign structures with CFFI generic functions

[Liam Healy]

On Sun, Sep 4, 2011 at 3:23 PM, Luís Oliveira <luismbo at gmail.com> wrote:
>
> On Sun, Sep 4, 2011 at 3:33 AM, Liam Healy <lnp at healy.washington.dc.us> wrote:
> > I've written a macro define-structure-conversion to flesh out what I see as
> > the way to go.  Eventually, I think this would be best as a couple of key
> > arguments in the defcstruct, similar to FSBV's current :constructor and
> > :deconstructor arguments.
>
> I'm not convinced we need a macro like that. Certainly not one
> specific to structures. Defining translate-* methods is indeed a bit
> more verbose, but it's also plain CLOS, which makes it clear and
> flexible. (E.g., you easily can use inheritance and multiple dispatch
> when defining CFFI types if you already know CLOS.)

My target audience is people like me, scientists and engineers who
want to make a foreign numeric library work and get on with their
lives.  They might not know what CLOS is, they might not ever have
written a defmethod.  Why should they?  Even people who do know what
these are should not have to rewrite boilerplate over and over; that
is exactly what macros are designed for: to hide implementation
details.

Here's what I have now in FSBV:

(defcstruct (complex :constructor complex :deconstructor (realpart imagpart))
  (dat :double :count 2))

You can get a pretty good sense of what the :constructor and
:deconstructor args do quickly, and I bet most people could write
their own defcstruct and appropriate converter quickly with no trips
to the manual.  The expansion is quite ugly, but no one cares because
no one looks at it, not even me any more.  While your methods below
are a lot simpler than what FSBV expands this defcstruct to, they
still involve implementation detail irrelevant to the user that will
involve (for me anyway) many trips to the manual to get right, or
perhaps big chunks of cut-and-paste.  I'd like to eliminate that.  No
one would be forced to use the macro, you could still write the
defmethods by hand as you propose.  Macros eliminate or minimize cut
and paste.

> I am stuck however on recursive conversion of slots.  What I have works for
> the complex example, but if I built on that with a  "real-and-complex"
> structure and try to convert to foreign, I get an error because the complex
> slot is already converted by the time it gets in the setf.  Even with an
> existing pointer, as would be obtained from foreign-slot-pointer, the only
> function I have at my disposal for translating the slot is
> translate-to-foreign which creates a new foreign struct, it does not write
> to an existing struct.  It seems I need something like a setf function.

>    (defcstruct (real-and-complex :class real-and-complex)
>     (x :double)
>     (c complex-double-c))
>
>    (define-structure-conversion value real-and-complex list (x c)
>      ;; Make foreign
>      ;;(setf x (first value) c (convert-to-foreign (second value)
> 'complex-double-c))
>      (setf x (first value) c (second value))
>      ;; Make CL
>      (list x c))
>
> I see a couple of issues raised by your "real-and-complex" example:
>  (1) recursive translations.
>  (2) differentiating between value/reference semantics for the
> translation of aggregate types.
>  (3) reusing translation code for both semantics.
>
> Regarding (1), at first sight, it seems to me that if we implement
> some sort of default translation from structures to CLOS objects, we
> can handle recursiveness there. User-provided translations are on
> their own, i.e. they should call convert-foreign-objects themselves.
> (We can export a way to iterate through the slots of a structure if we
> don't already.) BTW, it occurs to me that a default cstruct to CL
> conversion could use opaque objects whose accessors handle the
> translations (and possibly cache them).

I don't envision making CLOS objects the CL equivalent of foreign
structures I need (I assume you mean of CLOS object of metaclass
standard-class), but I have no objection to providing some default
methods.   My inclination would be to make the default CL equivalent a
list.   That's how it is for fsbv:defcstruct.   It is unclear to me
how making CLOS objects simplifies the recursive conversion task.

I don't understand your BTW statement.

>
> Regarding (2), I agree that we need a translate-into-foreign-memory
> function that takes 3 arguments: source value, type, and a target
> pointer. (IIUC, this would be the setf function you were referring
> to.) This will also be useful to properly integrate the array type
> with structures, BTW.

Yes, this is what I meant by a setf function -- right now in FSBV, I
have two functions, the :constructor which is analogous to CFFI's
translate-from-foreign, and the :deconstructor which sets components
of an existing foreign structure from the Lisp, unlike
translate-to-foreign which creates a new foreign structure.  This may
be called by the (setf object) function; the former is called by the
#'object function.

>
> Having solved (2) that way, (3) should be straightforward. At some
> point, there'll always be a pointer to somewhere, so
> translate-to-foreign can call translate-into-foreign-memory, if
> applicable.
>

If CFFI had a translate-into-foreign-memory, then a
translate-to-foreign could be an ordinary function:
(translate-into-foreign-memory source type (foreign-alloc type))
Wouldn't that do everything it needs to?

(BTW, I get confused by foreign-alloc's :initial-element/contents
arguments.  Is this calling translate-to-foreign?  It seems like it
presumes some kind of translation facility.  Or does it simply do
nothing if type is a foreign structure?)

> So, under this proposed scheme, your COMPLEX example would look something like:
>
> (defcstruct (complex :class complex-type)
>  (real :double)
>  (imag :double))
>
> (defmethod translate-into-foreign-memory ((value complex) (type complex-type) p)
>  (with-foreign-slots ((real imag) p 'complex)
>    (setf real (realpart value)
>          imag (imagpart value))))
>
> (defmethod translate-from-foreign (p (type complex-type))
>  (with-foreign-slots ((real imag) p 'complex)
>    (complex real imag)))
>
> This would be adequate to implement translation both when embedding
> the structure and when passing as a value. (We could also implement a
> TRANSLATE-TO-FOREIGN if we wanted to do automatic translation when
> passing by reference as well. I guess we probably would want want.)
>
> I like that this transfers the responsibility of allocating a
> (temporary?) structure to libffi-specific code.

I don't understand why you call out libffi specifically.  Certainly
libffi isn't the only library that might want to refer to a foreign
structure.  And yes, the responsibility is on the user of the library
to allocating it; the way I do it in FSBV is with-foreign-objects
instead of something like the translate-to-foreign,
free-translated-object pair because the lifetime is well-defined
within one function call.  Providing both is the best, however.

>
> The REAL-AND-COMPLEX example would in turn look like this:
>
> (defcstruct (real-and-complex :class real-and-complex-type)
>  (x :double)
>  (c complex))
>
> (defmethod translate-into-foreign-memory (value (type real-and-complex-type) p)
>  (setf (foreign-slot-value p 'real-and-complex 'x) (first value))
>  (convert-into-foreign-memory (second value)
>                               'complex
>                               (foreign-slot-pointer p 'real-and-complex 'c)))
>
> (defmethod translate-from-foreign (p (type real-and-complex-type))
>  (with-foreign-slots ((x c) p 'real-and-complex)
>    (list x c)))
>
> Does this look good to you plumbing-wise? Regarding the external API,
> do you still think we need some sort of convenience macro around this?
> (I guess the real-and-complex example would benefit from a
> WITH-FOREIGN-SLOT-PLACES along the lines of WITH-FOREIGN-PLACE
> described in <https://bugs.launchpad.net/cffi/+bug/688532>.)

Getting there; yes I definitely want the macro to hide implementation
details like the calls to foreign-slot-pointer, with-foreign-slots
parts, and the recursive *-into-foreign-memory, so that I can just
give a pair:

(first value) (second value)
(list x c)

or whatever would involve zero CFFI function/macro calls, and let the
macro do the work for me, as with FSBV's :constructor and
:deconstructor (which may be oversimplifying).

>
> We'll want matching converto-into-foreign-memory and
> expand-into-foreign-memory functions for this new
> translate-into-foreign-memory.
>

I don't know what expand-into-foreign-memory is.

> Cheers,
>
> --
> Luís Oliveira
> http://r42.eu/~luis/

Thanks,

Liam