[cffi-devel] Re: [fetter-devel] Quick CFFI update

[James Bielman]

Kenny Tilton <ktilton at nyc.rr.com> writes:

> My guess is that it is easier to invent one uniform high-level API
> atop a few low-level FFI capabilities than take a dozen high-level
> APIs and come up with a consistent super-high-level API. But no one
> is saying that except me. And CY. :)

Err, if I understand you right this is exactly what CFFI does!

For example, let's define the C timeval structure using CFFI:

(defcstruct timeval
  (tv-sec :long)
  (tv-usec :long))

The macroexpansion of this definition is 100% ANSI CL.  No calls to
the host FFI are made, it just defines a CLOS instance associated with
TIMEVAL containing the layout of the slots in the TIMEVAL structure.
(Actually that's not entirely true, it has to ask the Lisp FFI for the
size and alignment of the :LONG type, but that's not very difficult).

Now, let's allocate a TIMEVAL on the heap:

(let ((tv (foreign-object-alloc 'timeval)))
  ...)

FOREIGN-OBJECT-ALLOC is a function (again, ANSI CL) that ends up using
the CFFI-SYS (implementation layer) interface, to allocate enough
bytes to hold the structure:

  (foreign-alloc 8)

Which ends up calling some sort of MALLOC function defined by the host
FFI (every FFI I've seen exports this primitive already).

If we set a slot in TV:

  (setf (foreign-slot-value tv 'timeval 'tv-sec) 100)

FOREIGN-SLOT-VALUE is another portable function (but will be inlined
via a compiler-macro since the types are constant), that uses a
CFFI-SYS primitive to actually frob the memory:

  (setf (cffi-sys:%mem-ref tv :long 0) 100)

%MEM-REF is a more complicated primitive; this expression is
equivalent to:

  *(((long *) tv) + 0) = 100;

and is also inlined via compiler macros to, for example, on SBCL:

  (setf (sb-sys:sap-ref-32 tv 0) 100)

Again, every FFI I've looked at so far has this primitive in some form
or another (CLISP didn't have a fast one, although I think one is
being worked on).

Most of the CFFI-SYS primitives are like this, just frobbing memory
and doing operations on pointers, and there doesn't seem to be too
much trouble porting these.

There are some more complex primitives for doing block memory
transfers and getting pointers to the data in Lisp vectors, but these
can be implemented in terms of other primitives if necessary.

The only primitive that is really problematic is FOREIGN-FUNCALL,
which is like OpenMCL's EXTERNAL-CALL macro.  It provides a method for
calling a foreign function without using a declarative interface.

For example:

? (foreign-funcall "sqrtf" :float 16.0 :float)

The last argument to FOREIGN-FUNCALL is the return type, preceding
keyword/value pairs are the types and values of the function arguments.

I like to short-circuit the host FFI's declarative interface to
functions because this macro makes it very easy to deal with varargs
functions (which I need to write the portable Objective-C bridge I
want to write down the road).

Now, users (well, hypothetical users anyway) don't generally use this
directly, they use DEFCFUN, which is like UFFI:DEF-FUNCTION:

(defcfun "sqrtf" :float
  (x :float))

If there's a compromise to be made, this is where I would suggest
making it.  Hello-C could redefine DEFCFUN to macroexpand into the
host FFI's declarative interface (DEF-FOREIGN-CALL, etc), instead of
using FOREIGN-FUNCALL.  (It would still need to invoke any type
translators defined in the argument and result types, to do things
like convert Lisp strings to/from C strings).

I suspect that, making this compromise, it would be easy to support
every reasonable host FFI for the Lisps you are interested in.

I suggest that you look over Luis' Allegro CL port of the CFFI-SYS
interface at:

http://common-lisp.net/project/cffi/darcs/cffi-luis/src/cffi-allegro.lisp

The interface is really not all that big, and apart from
FOREIGN-FUNCALL it's pretty simple stuff.

So, to get back to your original question, what we gain with a minimal
interface is we can define a little bit of core functionality and get
everything else "for free".

> I am anxious to get the Actual Landscape (in my mind's eye I see a
> 2D table, rows being Lisps, columns being needed support hooks for
> CFFI) to sharpen this discussion. I am also anxious, where gaps are
> found, for Luis to contact vendors/maintainers and get them to sign
> on to making necessary changes ASAP. Hopefully the prospect of being
> excoriated on c.l.l by Demon Kenny will bring cooperation. :)

Hopefully I've been able to convince you somewhat that the interface
really isn't all that complicated, and the hooks needed are pretty
basic (and likely to already exist), except for FOREIGN-FUNCALL which
could be done differently to ease development.

> In a sense, i guess what we are doing is a Pitamnesque substandard
> or a CLRFI, except instead of undertaking a process a few of are
> undertaking some coding. I always prefer coding, myself. :)

Basically, although I think there's a much better chance of getting
things to happen when you can enter the discussion with a working
implementation, instead of arguing over some specification.

And now I'm going up to the roof to watch the fireworks over Puget
Sound. :)

James