[fetter-devel] Re: CFFI progress

[Kenny Tilton]

James Bielman wrote:

>[Hi Kenny, I guess we should have talked about FFIs at ILC too, eh?]
>
yes, who knew? i was delighted to discover who was behind cffi. :)

>
>I'm just entering this discussion, so I'm not quite sure what you guys
>are planning to do with Hello-C / CFFI / etc.
>
>(Stop me if you know all this already, just making sure since I've
>never really officially announced this library or anything...)
>
>As I understand it, the goal of Hello-C is to add callbacks and
>support for additional Lisp implementations to a fork of UFFI.
>
Well, no, the goal is a truly universal FFI, in both functionality 
(adding callbacks) and Lisps supported. We /thought/ that would mean 
extending UFFI, but CFFI has us wondering if a better approach is 
possible. And let's not forget the possibility of a hybrid approach. I 
know UFFI on CMUCL has a history of performance problems. Maybe we go 
the CFFI route with CMUCL and let other Lisps' FFIs handle things 
themselves?

>
>CFFI is a pretty different animal, in that it tries to define an
>low-level interface for frobbing memory and foreign functions, then
>implement the remainder of the foreign type system in portable code.
>Check out the CFFI-SYS draft specification available at
>http://common-lisp.net/project/cffi for details on the system
>interface.
>
Good lord, man, you wrote a spec?! I do not even document things when 
they are released to the public! :)

>
>CFFI can also be more efficient than UFFI on CMUCL and SBCL since it
>uses system area pointers directly instead of consing up alien objects
>that carry around redundant type information at run-time (and require
>lots of type/inline declarations to optimize away).
>
OK.

>
>However, it's also less mature than UFFI.  In fact, it's only 3 weeks
>old, and doesn't support as many Lisp implementations, because CFFI is
>asks a bit more of the Lisp implementor (providing the low-level
>operations to implement the CFFI-SYS interface).
>
I see two concerns:

- will every Lisp support CFFI with the necessary low-level ops

- are the low-level ops exported / stable / reliable / supported?

- will CFFI sometimes be less effective than the implementation FFI? For 
example, Franz indicates their native sockets package works better with 
their MP code, so in MP situations would run better than a generic 
solution using winsock or native BSD sockets via the FFI.

Again, a hybrid approach means not every Lisp must support CFFI.

>
>Anyway, I'll elaborate a little on this list of open issues:
>
>Kenny Tilton <ktilton at nyc.rr.com> writes:
>
>  
>
>>Luis Oliveira wrote:
>>    
>>
>>>[ideas for things to work on in cffi]
>>>
>>>  -> Foreign globals. 
>>>      
>>>
>>Sounds reasonable. I just never encountered that. This means, however,
>>an autogenerated C DLL to create a C interface?
>>    
>>
>
>UFFI does this already with DEF-FOREIGN-VAR.  I think all Lisps
>supported by UFFI can do this without generating any sort of C stubs.
>
Doh! Actually, this reminds me: I have done a lot of FFI work, but I 
have managed to get by with a subset of FFI capablities, so I have some 
blind spots.

>
>  
>
>>>  -> FOREIGN-SLOT-VALUE: let it take multiple "indices" for directly
>>>     accessing structs inside structs, arrays inside structs,
>>>      
>>>
>>Hmm, not appearing clearly in my mind's eye. But I get the drift/value
>>if feasible.
>>    
>>
>
>The idea here is to make FOREIGN-SLOT-VALUE sort of like AREF:
>
>(defcstruct timeval
>  (tv-secs :long)
>  (tv-usecs :long))
>
>(defcstruct employee
>  (name :char 255)
>  (hire-date timeval))
>
>(with-foreign-object (emp employee)
>  ;;                        obj  type     slot-1     slot-2...
>  (setf (foreign-slot-value emp 'employee 'hire-date 'tv-secs) 10204242)
>  ;; This is like 'emp.hire_date.tv_secs' in C...
>  ...)
>
Gotcha. Again, I have gotten by with merely functional FFI, never hack 
data structures. Looks like some introspection is needed, but nothing 
serious.

>
>  
>
>>>  -> Type-checking pointer interface. 
>>>      
>>>
>>OK, interesting. I went the other way: one type-trusting interface,
>>one autoconverting interface.
>>    
>>
>
>Indeed, I don't really care too much about the type-checking, but
>others might.  One of reasons I like defining the bulk of the foreign
>type system in terms of low-level memory operations is that we can
>experiment with different foreign type systems in a portable way.
>
>This is certainly not a high priority for me though.
>
>  
>
>>"automatic access" has practically zero value. We will not diverge
>>from UFFI gratuitously, so if it was a good interface it will
>>survive largely unchanged even with the new underpinnings. If it was
>>a sucky interface, we /want/ to kill it. If the latter obtains,
>>automatic access will not fly. If, however, the UFFI API survives
>>largely intact, a set of UFFI bindings can be transformed in a few
>>hours. This equals "practically zero value" for backward
>>compatibility.
>>    
>>
>
>I'm not really sure what you mean here.
>
That was just a long argument against worrying about backward compatibility.

>  In CFFI, I've pretty much
>thrown out the UFFI interface and written my own.  In particular,
>everything that can conceivably be a function is a function, in
>contrast to UFFI which consists almost entirely of macros (many of
>which can be confusing WRT what gets evaluated on which Lisp).
>
Yep. I think KMR realizes this and would cure in UFFI2.

>
>CFFI-UFFI-COMPAT maps the thrown-out UFFI interface back to my new
>interface (minus some lossage currently with arrays, CFFI does arrays
>quite differently from UFFI).
>
Yep, this is a fine approach. Forget UFFI, then recover it later as an 
optional, alternative API.

>
>CFFI-UFFI-COMPAT has quite a bit of value to me because I can build
>existing libraries like CLSQL, get a performance boost on SBCL, and
>not have to change the code (and indeed I've successfully built the
>CLSQL-SQLite3 backend using CFFI-UFFI-COMPAT).
>
Yep. Was that no changes at all, or "just a few" tweaks? Up above I was 
saying that, if and only if we had to break backward compatibility with 
UFFI, that that would not be the end of the world if we kept the 
breakage to a minimum.

-- 
Kenny

Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film

"If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page