[Ecls-list] CFFI and DFFI

[Juan Jose Garcia-Ripoll]

On Wed, Jul 22, 2009 at 4:49 AM, Luis Oliveira<luismbo at gmail.com> wrote:
> Juan Jose Garcia-Ripoll
> <juanjose.garciaripoll at googlemail.com> writes:
>
>> CFFI has some weird assumptions about what a foreign function
>> interface can and can not do. It assumes everything has to be dynamic
>> while ECL prefers to compile things (the resulting code is more
>> efficient)
>
> CFFI is indeed inspired on the FFIs of SBCL, CCL, Allegro, Lispworks,
> etc, which have similar implementation strategies, and are all very
> different from ECL.

The fact that many lisp implementations opt for some FFI strategy does
not mean that is the ideal or the only possibility and it makes our
life (ECL maintainers) more difficult. UFFI, for instance, has a
simpler set of function types and arguments it supports (pointers,
ints, floats, etc) because they are less likely to change with the
ABI.

Work on ECL's DFFI only began because of CFFI. DFFI is a mixture of
assembly and C code to generate dynamically function calls to foreign
functions. I must say I myself do not like it. I was only able to code
the x86/32 bit version, M. Goffioul hacked the x64_64 bits port
without even knowing x86_64 assembler. But that is all. Should we want
to use it on other platforms I would need to contact a skilled (or
reckless) coder that does it for us. But even then I would not trust
it as much as the code generated by a C compiler, which does know the
ABI very well, including the choices of registers it does when passing
structures and other things.

The fact that ECL is forced to use DFFI because all other lisps do is
kind of awkward. ECL uses a C compiler, my gosh, that should put us on
an advantage position, having all the type information, being able to
code a FFI call with just one line of C, no need for building callback
code on the fly, or other hacks... However, as it is, with CFFI this
has not been possible.

>> does not care that functions with varargs ("..." in C) may
>> be different from other functions (making our compiled code really
>> really unportable)
>
> Does ECL's DFFI have any sort of varargs functionality? If so, we could
> use it. How is UFFI any different in that regard BTW?

I just cited an example back from my recollections when initially
working on the CFFI port. CFFI dropped all type information, varargs
was completely ignored, this and other problems made it impossible for
us to produce a single C function call that was type accurate and
could be used by the compiler. You can see traces of the problems in
the current cffi-ecl port, with code like
           (let ((types (mapcar #'ecl-type->c-type types)))
             ;; On AMD64, the following code only works with the extra
             ;; argument ",...". If this is not present, functions
             ;; like sprintf do not work
which is due to the fact that C functions are not really called with
the types they should.

To this you should add that CFFI only contemplates dynamically loaded
libraries, no possibility of letting a proper linker resolve symbols:
one has to use always dlopen, etc, as you said, because other lisps
do. In ECL's UFFI we can just compile in the library.

> We're certainly open to patches/suggestions that help us better support
> ECL's static and dynamic FFIs.

Right now I do not have time to work on that. To begin with, it should
be possible to use CFFI as just a kind of lisp-SWIG, in which files
with foreign function calls are compiled before being loaded. That,
and keeping more accurate information, allowing libraries to be
statically specified, not just dynamically loaded, creating compiler
macros for the foreign functions so that they expand into C-INLINE
calls, etc, would be a start towards a more efficient implementation
of CFFI on ECL (and also on GCL, btw)

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com