SV: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant

Tue Aug 22 18:18:47 UTC 2006

I am not sure, I have been looking for good documentation. I might exist
somewhere, though? I am guessing int in FFI is closer connected to int
as is defined by C and thereby the OS? Somebody can probably tell us...

It seems also to be a problem computing the size of a buffer, as well,
which causes problems for strings. I will take a closer look tomorrow,
tonight I have to struggle with some deadlines...

Petter

-----Opprinnelig melding-----
Fra: Ian Eslick [mailto:eslick at csail.mit.edu] 
Sendt: 22. august 2006 15:43
Til: Petter Egesund; eslick at media.mit.edu
Kopi: Robert L. Read; Ian Eslick; elephant-devel at common-lisp.net
Emne: RE: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with
Elephant

I don't have internet at the moment, but what is the definition of int
vs integer in the sb-alien foreign interface?  Just want to make sure we
aren't asking for trouble in the future.  (i.e. is int signed or
unsigned vs integer?  Is it 32bit for compatibility or whatever the clib
it's linked to says it is?)

Ian

-----Original Message-----
From: "Petter Egesund" <Petter.Egesund at kunnskapsforlaget.no>
To: eslick at media.mit.edu
Cc: "Robert L. Read" <read at robertlread.net>; "Ian Eslick"
<ieslick at common-lisp.net>; elephant-devel at common-lisp.net
Sent: 8/22/06 8:18 AM
Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with
Elephant

Hi;

I have done some debugging. It seems the problem of serializing integers
is caused by mixing of the types int and integer in the foreign
interface?

I change the function read-int in the package memutil.lisp to:

(defun pread-int (buf offset)
  "Read a 32-bit signed integer from a foreign char buffer."
  (declare (optimize speed (safety 1) (debug 1))
	   (type (alien (* char)) buf)
	   (type fixnum offset))
  (print "Offset: ") (print offset)
  (the (signed-byte 32)
    (deref (cast (sap-alien (sap+ (alien-sap buf) offset) (* char))
		 (* int))))) ;; !!!!!!!!!!!!! Changed from (* integer)

After this the integer-test runs fine. I will do some more testing.

Cheers,

Petter Egesund

-----Opprinnelig melding-----
Fra: Ian Eslick [mailto:eslick at csail.mit.edu] 
Sendt: 21. august 2006 16:11
Til: Petter Egesund
Kopi: Robert L. Read; eslick at media.mit.edu; Ian Eslick
Emne: Re: SV: SV: SV: SV: SV: [elephant-devel] Memory error with
Elephant

Interesting.  That explains the source of our problem (serializing
integers) but I still don't understand why.  Even if the 4 byte
assumption in lisp keeps us from expanding the buffer streams
appropriately the write should still write the first 8 bytes correctly
and thus be able to read it correctly.  It almost looks like it is
writing 4 bytes but reading 8.

1) Try replacing the #x45 with #x00 in my test code

The other test is to see what is actually being put into the buffer
stream.

2) replace #x45 with #x40302010 and add the following code instead of
the buffer-read-int at the end:

(loop for i from 0 upto 16 do
    (format t "byte ~A: ~A~%" i (buffer-read-byte bs)))

Ian

Petter Egesund wrote:
>  
>
> ----------------------------------------------------------------------
> --
> *Fra:* Robert L. Read [mailto:read at robertlread.net]
> *Sendt:* 19. august 2006 22:26
> *Til:* Petter Egesund
> *Kopi:* eslick at media.mit.edu; Ian Eslick
> *Emne:* Re: SV: SV: SV: SV: [elephant-devel] Memory error with 
> Elephant
>
> Personally, the fact that buffer-write-int in memutil.lisp assumes a 
> 4-byte integer:
> (defun buffer-write-int (i bs)
>   "Write a 32-bit signed integer."
>   (declare (optimize (speed 3) (safety 0))
>    (type buffer-stream bs)
>    (type (signed-byte 32) i))
>   (with-struct-slots ((buf buffer-stream-buffer)
>       (size buffer-stream-size)
>       (len buffer-stream-length))
>     bs      
>     (let ((needed (+ size 4)))
>       (when (> needed len)
> (resize-buffer-stream bs needed))
>       (write-int buf i size)
>       (setf size needed)
>       nil)))
> While the c-code version of write-int in libmemutil.c uses the 
> "sizeof(int)" paradigm seems fragile on a 64-bit architecture:
>
> void write_int(char *buf, int num, int offset) {
>   memcpy(buf+offset, &num, sizeof(int)); }
>
> I would like to know if "sizeof(int)" is 8 or 4 as compiled.
> A simple C program (or reading the f-ing manual) would answer that 
> question.
> An awful hack that would let us test things would be to hard-wire "4"
> in place of
> "sizeof(int)" in the libmemutil.c file;  I don't know enough about 
> UFFI and sap-alien to know if that would work.
>
> A better solution is to expose a C-function that simply gives the size

> of an integer in bytes to LISP (it could be added to libmemutil.c 
> file) and then have memutil.lisp use that in place of the assumption 
> "4".)
>
> How this changes the typing in memutil.lisp, I'm not sure ---- can 
> we/should we force the
> AMD64 to use 32-bit integers, or should we rewrite memutil.lisp to 
> determine based on its compilation environment the size of an integer?

> Either solution is better than what we have now.
>
>
> On Sat, 2006-08-19 at 20:53 +0200, Petter Egesund wrote:
>> It looks like below. Certainly not 35.
>>
>> Petter
>>
>> -------
>>
>> ELE> (defun fixnum-test ()
>>      (with-buffer-streams (bs)
>>        (buffer-write-int #x23 bs)
>>        (buffer-write-int #x45 bs)
>>        (reset-buffer-stream bs)
>>        (buffer-read-int bs)))
>> ; in: LAMBDA NIL
>> ;     (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 69 ELEPHANT::BS)
>> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET 
>> WHEN ; --> COND IF PROGN ; ==>
>> ;   (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS
>> ;                                          ELEPHANT-MEMUTIL::NEEDED)
>> ;
>> ; note: doing signed word to integer coercion (cost 20)
>>
>> ;     (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 35 ELEPHANT::BS)
>> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET 
>> WHEN ; --> COND IF PROGN ; ==>
>> ;   (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS
>> ;                                          ELEPHANT-MEMUTIL::NEEDED)
>> ;
>> ; note: doing signed word to integer coercion (cost 20) ; ; 
>> compilation unit finished
>> ;   printed 2 notes
>> FIXNUM-TEST
>> ELE> (fixnum-test)
>> 296352743459
>> ELE> 
>>
>>
>>
>>
>>  