[fetter-devel] Re: CFFI progress

[James Bielman]

Kenny Tilton <ktilton at nyc.rr.com> writes:

> Well, no, the goal is a truly universal FFI, in both functionality
> (adding callbacks) and Lisps supported. We /thought/ that would mean
> extending UFFI, but CFFI has us wondering if a better approach is
> possible. And let's not forget the possibility of a hybrid
> approach. I know UFFI on CMUCL has a history of performance
> problems. Maybe we go the CFFI route with CMUCL and let other Lisps'
> FFIs handle things themselves?

While this may be possible, it seems to me like it may be a lot of
extra work.  IMHO if you're going to go to the trouble of implementing
your own foreign type system, you may as well make it portable.

If you really want a hybrid approach, it may be easier to try to use
the alien interface to obtain information at macroexpansion time but
still use it for defining foreign types (handwavy but I dont know the
details, most likely you'd need to get some stuff exported).

> I see two concerns:
>
> - will every Lisp support CFFI with the necessary low-level ops
>
> - are the low-level ops exported / stable / reliable / supported?
>
> - will CFFI sometimes be less effective than the implementation FFI?
>   For example, Franz indicates their native sockets package works
>   better with their MP code, so in MP situations would run better than
>   a generic solution using winsock or native BSD sockets via the FFI.

While I'm not so ambitious as to try to write some sort of CLRFI and
get every CL vendor to implement it, I am trying to take a slightly
longer-term view, and at least convince each vendor to make a core set
of functionality available and documented/exported/etc.

For example, the reason 'cffi-allegro.lisp' doesn't exist yet is
because I want to get Franz's input on how best to implement the
CFFI-SYS specification for their Lisp, as they do not export some of
the pieces I need.

(Unfortunately I have essentially no time to work on any of this for
most of July, but it will get done eventually).

> That was just a long argument against worrying about backward
> compatibility.

Ah yes, who cares about that?! :)

>> In CFFI, I've pretty much thrown out the UFFI interface and written
>> my own.  In particular, everything that can conceivably be a
>> function is a function, in contrast to UFFI which consists almost
>> entirely of macros (many of which can be confusing WRT what gets
>> evaluated on which Lisp).
>>
> Yep. I think KMR realizes this and would cure in UFFI2.

It's hard though, because the host Lisp FFIs can vary wildy in how
they evaluate their arguments.  For example, to use OpenMCL's high
level FFI, almost everything has to be constant at macroexpansion
time.  UFFI even has to do some gross things like intern keyword
symbols at macroexpansion time for building structure references.

I don't see that there's any way around these sorts of problems as
long as you are trying to paper over each Lisp's high-level
interface.

>> CFFI-UFFI-COMPAT has quite a bit of value to me because I can build
>> existing libraries like CLSQL, get a performance boost on SBCL, and
>> not have to change the code (and indeed I've successfully built the
>> CLSQL-SQLite3 backend using CFFI-UFFI-COMPAT).
>>
> Yep. Was that no changes at all, or "just a few" tweaks? Up above I
> was saying that, if and only if we had to break backward compatibility
> with UFFI, that that would not be the end of the world if we kept the
> breakage to a minimum.

No changes at all, but I did read through the code first to make sure
it didn't have any embedded arrays or nested structures, since the
compatibility layer doesn't handle these right yet.

The compatibility layer was basically a for-fun afterthought I had,
and if I can't make it work for the bulk of the UFFI-ized libraries
out there, I'll probably scrap it.

James