[Ecls-list] FFI paradigms (was Re: ECL 10.2.1 RC)

[Matthew Mondor]

On Sat, 13 Feb 2010 15:40:37 +0100
Juan Jose Garcia-Ripoll <juanjose.garciaripoll at googlemail.com> wrote:

> A test case would be nice. I did not investigate this further because of the
> lack of information -- and also because I tend to forget problems unless
> reminded frequently :-) It may well be that ECL does not have enough
> information about alignment in structures and is thus wrongly identifying
> the location of a slot value.

I will try to soon provide example code to reproduce the issue.
I will also further explain in this post why it may be quite a
challenge to get UFFI structure accessors working right.

> But I also realized that support for other C99
> > types, and some useful grovelling features were missing from UFFI.
> 
> Since I had to commit an unrelated fix, I used this chance to include your
> patch. The support for C99 types should now be available in UFFI.

Thanks; when I wrote the above paragraph I was even refering to other
types which I didn't think of at the time of my patch as well, i.e.
bool and size_t, but I think the list could also grow if we really went
that route (although admitedly those inttypes are probably pretty good
to have).

The other issue about the lack of grovelling I was writing about had to
do with needing to access C-side constant values and structures easily
(i.e. which CFFI or the native inline FFI allow), but with UFFI, one
would have to presently either write code to parse C headers or to
manually copy them.  And there is stuff like <sys/types.h> defining
system-specific types which usually are typedefs to other internal
macros (often from machine/*.h) and ultimately some grovelling code
would have to recursively do that parsing, as well as the FFI system
support the types.  Moreover, UFFI's C-struct definition macro does not
allow to supply the actual field and types as case-sensitive strings,
so it couldn't easily generate the equivalent inline C code.

To explain in less words: the difference between CFFI ctype, cstruct and
constant/constantenum which accept C strings versus UFFI def-constant and
def-struct which have to guess how it'll map to binary, is enormous.

With inline FFI this becomes much simpler as the file is #included, C
struct field accessor functions can also be built with macros
generating inline code which already knows all that's needed, using
strings like CFFI allows.  So I've pretty much have abandonned the idea
of using UFFI for ECL-specific contribs, I'll stick to macros on top of
the existing c-inline FFI system.  Since the native inline FFI will
insert ecl_*() convert calls as necessary to convert between C-native
and the supported FFI types (and the inline-C can do it where required),
having a wide range of C-native types support becomes less of an issue.

The API of my macros is subject to change and refinement, but an example:

Some C definitions:

#define SDL_SWSURFACE   0x00000000      /* Surface is in system memory */
#define SDL_HWSURFACE   0x00000001      /* Surface is in video memory */
/* etc */

typedef struct {
        Sint16 x, y;
        Uint16 w, h;
} SDL_Rect;


Custom macros using inline FFI mapping to the C definitions:

(include-c-header "<SDL/SDL.h>")

; These now can use the actual C macros, whatever their actual definition.
(define-c-constants
  '((+sdl_swsurface+ "SDL_SWSURFACE")
    (+sdl_hwsurface+ "SDL_HWSURFACE")))
; etc

; This can easily generate c-inline accessors and even constructor and
; print functions, and there's no need to include all fields, only the ones
; we want to export to CL.  Alignment is dealt with in actual C code.
; Moreover, we can rely on the actual C struct field types and not worry
; about converting to an equivalent standard C type in a restricted set.
(define-c-structure :sdl-rect "SDL_Rect" '((:x :fixnum "x" "Sint16")
                                           (:y :fixnum "y" "Sint16")
                                           (:w :fixnum "w" "Uint16")
                                           (:h :fixnum "h" "Uint16")))

Since this provides the C-side types as strings, as well as the CL-side type to convert from/to, a wide range of known FFI types becomes less of an issue.

Versus UFFI:

; Those have to be copied from C header files and would have to be updated if said
; files change.
(def-constant +sdl-swsurface+ #x00000000)
(def-constant +sdl-hwsurface+ #x00000001)
; etc

; The backend has little information to guess the actual binary representation
; to generate accessors.
(def-struct sdl-rect
  (x :int16-t)
  (y :int16-t)
  (w :uint16-t)
  (h :uint16-t))


So to resume, using portable UFFI the C99 inttypes are not even known
(even less the sys/types ones), even if ECL's UFFI knew about more
types, they couldn't be used portably, and an incentive to use UFFI is
portability.  Secondly, there's no way for c-inline code to reliably be
generated from that definition.  Third, alignment issues are possible
since this doesn't map to c-inline code easily (the C structure might
as well have been declared to be packed with a GCC attribute, for
instance, or the C compiler ABI might align fields to 32-bit or 64-bit,
or even 16-bytes for performance internally)...
-- 
Matt