[Ecls-list] CFFI and DFFI

[Juan Jose Garcia-Ripoll]

On Wed, Jul 22, 2009 at 3:12 PM, Luis Oliveira<luismbo at gmail.com> wrote:
>> UFFI, for instance, has a simpler set of function types and arguments
>> it supports (pointers, ints, floats, etc) because they are less likely
>> to change with the ABI.
>
> Can you be more specific? What CFFI types are problematic?

As I said, passing structures and varargs is not supported by UFFI. To
remove this limitation in a robust and portable way,

> Perhaps using libffi (for DFFI) would be a good, solid approach?
> <http://sourceware.org/libffi/>

I do not know about libffi. Is it just a child of libffcall? Because
that one is no less a hack than our code is, but that is not the
point. Those libraries, just like DFFI, represent an unnecessary
overhead for something that could be done efficiently in ECL.

> Having a dynamic FFI seems useful for interpreted code, for libraries
> that for some reason or another really do have to be accessed via
> dlopen/dlsym, and during development -- e.g., using the REPL.

Yes, but many people are happy with SWIG, even in interpreted
environments (scheme, python, etc) and that is not dynamic at all. As
another user pointed out, you can always compile a particular FFI you
need. CFFI could allow both development models, focusing on the static
one for platforms, such as GCL or ECL, on which this might be
advantageous.

> BTW, regarding the OP's problem, some UFFI libraries such as CLSQL pass
> non-constant arguments to UFFI:LOAD-FOREIGN-LIBRARY as well. I assume
> that would be a problem for ECL as well.

That is not the only possibility. When the library is not constant we
can resort to the non-constant version, which will be much slower, but
UFFI is simple enough that we can intercept the name of the library
and pass it to the linker when the file using UFFI is compiled. This
could also be done with CFFI, but its backend only provides pointers,
not really names.

>> I just cited an example back from my recollections when initially
>> working on the CFFI port. CFFI dropped all type information, varargs
>> was completely ignored, this and other problems made it impossible for
>> us to produce a single C function call that was type accurate and
>> could be used by the compiler.
>
> At what point does CFFI "drop all type information"? That doesn't sound
> right. Again, I'm not aware of any way to define varargs functions in
> UFFI/DFFI. If there were, CFFI could use it.

UFFI does not vararg functions it because it is not really portable.
Even on the same platform different compilers may choose different
conventions. But in any case, CFFI does not make use of the vararg
information either. The code that generates the function call in the
CFFI-ECL port has to compile _all_ function calls as varargs and hope
that works. Please have a look at it. The reason is that the CFFI
backend has not been passed that information to the code that
generates the function call. I explain it below.

>> To this you should add that CFFI only contemplates dynamically loaded
>> libraries, no possibility of letting a proper linker resolve symbols:
>> one has to use always dlopen, etc, as you said, because other lisps
>> do. In ECL's UFFI we can just compile in the library.
>
> I'm pretty sure I've done that with CFFI and ECL on darwin/ppc, so I
> don't think that claim is correct.

As I said, I do not know the full status of CFFI as of now. I am
talking about what I saw at the time and what I have read by a cursory
inspection of the code you have at the repo right now.

>> Right now I do not have time to work on that. To begin with, it should
>> be possible to use CFFI as just a kind of lisp-SWIG, in which files
>> with foreign function calls are compiled before being loaded.
>
> That's how CFFI-using libraries are usually loaded, so no problem
> there. (I should mention again that having a dynamic FFI seems useful
> nevertheless.)

In this case, as I said before, if instead of using the prototypes
based on dlopen + dlsym (I mean loading library and searching for
symbols), if the current backend was allowed to use the C names of the
functions and produce a C code that is compiled once and used, then I
believe it should be possible to create better prototypes and even
keep compiler macros that expand later calls, making them more
efficient.

>> That, and keeping more accurate information
>
> What information?

The accurate list of argument types and prototype of a function, such
that we can produce an valid C function call. Currently CFFI uses a
PRODUCE-FUNCTION-POINTER-CALL that takes a pointer and the types of
the arguments the user declares to pass, not the actual declaration of
the function. Hence if you call SPRINTF with a string and a float you
are not using the fact that it takes a variable number of arguments,
as I said before, but rather treating it as a function that takes a
string and a float, which is incorrect.

Ideally, CFFI should also allow specifying C headers that provide the
same information and let the backend either use the header or the type
information which is provided.

>> allowing libraries to be statically specified
>
> The only problem I see here is how to specify alternatives.
> E.g.: (cffi:load-foreign-library '(:or "libfoo.so.1" "libfoo.so"))
> How can we do this why ECL's static FFI?

Is it really such a big problem in the C world? Don't we have -l flags
that take care of resolving library names? The backend could be taught
to strip the library names, passing the search path to the linker
together with a list of -lA -lB ... etc

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com