[cffi-devel] must cffi-sys::with-pointer-to-vector-data go?

[Hoehle, Joerg-Cyril]

Hi,

David Lichteblau wrote:
[I don't know if David joined cffi-devel, maybe people did not get his reply?]

>But my impression was that CFFI, in contrast to UFFI, is willing to
>implement features even if not every Lisp supports them.
I had that impression too :)

So let's analyse further.

I'm going to argue for an -- if at all --
optional cffi:feature:shareable/vector.
Don't provide a poor or problematic version!

CFFI would need to document that this API is *not* available
everywhere, because the implementation cost of mimicking it is way to
high (see below).  Instead, code that wishes to benefit from this
feature must either tell it's restricted to implementation XYZ (lousy),
or use the #+/-cffi:feature to provide an *alternate*, portable
implementation.

I say lousy, because as SW complexity grows, people will not remember
all the places the pinnable vector is used, which will make later
porting a challenge.
Seems similar to the trouble with pointer aliasing in C.

If alternative code is provided, why not just avoid #+/- and use a
single code?  I believe it's fast enough for plenty of uses.


>However, I don't think that the concept of WITH-POINTER-TO-VECTOR-DATA
>is entirely bogus just because some backends use a bad implementation.

Indeed, I advocate explicitly *not* providing a bogus or slow
implementation.  If some Lisps needs WITHOUT-GC to match it, better
not provide the macro with them!


> * The commercial Lisps: Don't know, but they'll implement it if 
>   their paying customers start asking for it.  ;-)

I'm still convinced that the benefit is too low to add that,
commercial or not.  It's presumably hairy to add as an afterthought.
There are many many thing that would provide more value to Lispers.

As an example, if foreign-memory where useable as Lisp arrays (and
therefore as sequences), this would render many intended uses of
with-pointer-to-vector-data superfluous.  It leads to more Lisp-level
code, instead of FFI stuff.  And I believe it's easier to
implement, and provides more benefit, in more situations.


>interface would have to be changed so that [...]
>is guaranteed to work for any vector.

Indeed.  That's a excellent requirement.
Prefer consistency over restrictions.
Generally, programmers should test "optimized" software
with displaced vectors instead and witness the crashes :-(
Not every array is simple...


BTW, I don't know whether it's appropriate to worry too much about
efficiency in cl+ssl now.  It uses Flexi-streams, of which Edi Weitz
says "FLEXI-STREAMS is currently not optimized towards performance".
It uses GRAY streams, which is the first thing I'd look to avoid in a
quest for hopefully fast i/o.  An interface built upon READ-CHAR
cannot be fast.  You need block operations.
Simple streams and the like can get at it, not GRAY.
This is a sad situation of I/O in Lisp, indeed.


Here's a sketch of my argumentation against providing
with-pointer-to-vector-data for those Lisps that can't pin. Please
excuse my lack of time for further development.

Consider general availabibility of such a macro.
People may use it for 1MB large vectors.
Consider calling the FFI GUI event loop inside its scope.
This will cause callbacks to Lisp.  Clearly, WITHOUT-GC would be an
application killer (not killer application).

Now consider my :in/:out suggestion.  I argue it's still not enough as follows:

(elided). [partial :in requires :out+in, so as not to kill already
copied bytes, thus is a waste.  Enter :start :end keywords for more
precision about the touched area.]

Even if :in/:out plus :start + :end indices were added, it would not
be enough in some cases.  And there we notice that we've come
extremely close to providing such precise information about what bytes
move where, that we could, instead explicitly call memory-vector-copy
operations.  Why not do it right away?

Regards,
	Jorg Hohle