[cffi-devel] must cffi-sys::with-pointer-to-vector-data go?

[Hoehle, Joerg-Cyril]

Hi,
[I'm not member of cl-ssl@ and cross-posting is hairy, so I'd prefer this to remain in cffi-devel. I CC'ed to David Lichteblau, author of cl+ssl for comments.]

CL+SSL's using that macro caused me to think as follows:

CFFI-SYS says:
;;;# Shareable Vectors
;;; This interface is very experimental.  WITH-POINTER-TO-VECTOR-DATA
;;; should be defined to perform a copy-in/copy-out if the Lisp
;;; implementation can't do this.

cffi-sys::with-pointer-to-vector-data, as is, is highly problematic.
Trying to get the base address of a Lisp vector in memory is
unportable and subject to subtle errors with a moving GC.  I'll try
and suggest a better API below.

On CMUCL, it's implementend using sys:without-gcing, which is sign
enough of a problem.

This remembers me, 20 years ago, of similar bad use of SI::DISABLE-XYZ
(I forgot the name) on Symbolics machines, and on the Amiga computer,
where programmers were repeatedly told not to make wrong (ab)use of
Forbid() (disable multitasking) or Disable() (disable interrupts).

I could understand it if it were restricted to CFFI internal use, for
a couple of highly optimized vector copying operations.  Everything
else should be a big NO-NO.  It's use is worse than the often repeated
"EVAL is EVIL".

Sadly, its mere existance encourages some use.
E.g. CL+SDL uses it :-(

This is a very bad idea. Let me explain.
Cl+ssl:stream-read-byte (in stream.lisp) uses it for ssl-read and
installs a callback handler.  UNIX signaling seems alo involved, as
the error codes and loop structure indicate.

Now consider what happens when a user has installed a SERVE-FD-EVENT
handler (e.g. running SLIME).  While cl+ssl is looping and waiting for
ssl-read to complete, arbitrary Lisp code can be called by the signal
handlers and the event dispatcher.  All of this within the context of
sys:without-gcing.  This is calling for trouble.
And I did not yet mention other treads getting to run.

Back on the Amiga computer, Forbid/Disable() were accepted in a few
cases, and programmers were told to quickly exit the protected
section.  People used it mostly from assembly, but also C.  OS calls
within this section almost always are a bad idea, since it might
either break out of Forbid() and reenable multitasking, or hang
because important interrupts would not get served. This could break
invariants of the OS or the application, which would cause random
crashes some random time later.

As noted, I could understand use of gc:witout-gcing and
with-pointer-to-data when kept strictly inside CFFI and used locally,
e.g. to copy data from foreign to/from a Lisp array.

Any other use is leading to problems, typically hard to debug.  I hope
I made this clear.

There is IMHO no reason for CL+SSL to use such a function. If it wants
objects at a fixed address, it should use foreign-alloc'ed memory.

Try to explain your C/Java/Perl/tcl programmer friend: "I can
only use SSL when resorting to sys::without-gc".  "Huh? In my language
& environment, no such dirty hack is necessary.  It's clearly superior".

BTW, cl+ssl:stream-read-sequence uses
                (replace thing buf :start1 start :end1 (+ start length))
anyway.  So there's copying even when sharing.  Instead it could copy
from the foreign buffer to the user supplied "thing".  This raises the
question of whether copying to an arbitrary array-element-type is
supported by CFFI's emerging memory<->vector block copy API.


Now let's move toward a better design:

>From a CFFI perspective, the following comment:
;;; WITH-POINTER-TO-VECTOR-DATA should be defined to perform a
;;; copy-in/copy-out if the Lisp implementation can't do this.
IMHO shows an inversed design.

With the interface and recommendation as is, nobody knows which of a
copy-in and/or copy-out is needed.  The macro would do both, just to
be safe.  Implementations would suffer a double speed penalty.

The need for copy-in or copy-out must be indicated by the
programmer, similarly to :in and :out parameter modes.

I believe the design should be the opposite: an efficient copy-in or
copy-out may resort to with-pointer-to-vector-data and possibly to
si::without-gcing (is that thread-safe at all?) to quickly copy the
vector and do nothing more than that (no callbacks, no signals, etc.).


To return to the CL+SSL example, I suggest to use a foreign-alloc'ed
buffer for ssl-read etc., then copy that into a Lisp vector.

This copying could use CFFI's emerging memory block interface and rely
on CFFI to be fast.

To implement that block copy, CFFI could use
with-pointer-to-vector-data.  However I believe it's now superfluous:
In implementations were such a function is available at all
(e.g. cmucl), the native code compiler can already translate Lisp code
to an efficient loop from one array to the other.  Safely.

I mean that what w-p-t-v-d does (sys:vector-SAP etc.) can be restricted to a few internal functions within cffi-{cmucl,*}.lisp and need no dangerous general macro wrapper.


I think this is the best one can achieve portably, without resorting
to very specialised features like IIRC Allegro's ability to allocate
Lisp vectors at non-moving locations.  I haven't yet investigated how
one could make transparent use of such a feature, even though any
Allegro user would find it suboptimal if cl+ssl (or any other library)
would not make best use of her/his Lisp implementation. (Oh well, I'll
rant about portable libraries another time. :)


On a final note, cffi-cmucl contains
>     (let ((,ptr-var (sys:vector-sap ,vector)))
This won't work with displaced etc. (non simple) vectors.
It's not the common case, but it should be supported, or the
restriction documented (not that it matters if w-p-t-v-d is dropped
anyway).

I always welcome comments.

Regards,
	Jörg Höhle