SV: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant

Tue Aug 22 18:36:19 UTC 2006

There are definitely some issues with string calculations.  I've had
some issues dealing
with allegro's unicode support.

Look in memutil.lisp:
The byte-length macro

Look in serializer.lisp:
(method serialize %serialize) the string case statements

Serialization of strings involves:
1) Serialize appropriate string tag based on Lisp character width for
string type
    In SBCL this is 1 byte or 4 bytes depending on Unicode support
2) Serialize the total string length (bytes * num chars).  The
byte-length macro
    is used for this
3) Serialize the array of characters using buffer-write-string

Hopefully that will help you figure out where the mismatch occurs.

Cheers,
Ian

Petter Egesund wrote:
> I am not sure, I have been looking for good documentation. I might exist
> somewhere, though? I am guessing int in FFI is closer connected to int
> as is defined by C and thereby the OS? Somebody can probably tell us...
>
> It seems also to be a problem computing the size of a buffer, as well,
> which causes problems for strings. I will take a closer look tomorrow,
> tonight I have to struggle with some deadlines...
>
> Petter
>
>
> -----Opprinnelig melding-----
> Fra: Ian Eslick [mailto:eslick at csail.mit.edu] 
> Sendt: 22. august 2006 15:43
> Til: Petter Egesund; eslick at media.mit.edu
> Kopi: Robert L. Read; Ian Eslick; elephant-devel at common-lisp.net
> Emne: RE: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with
> Elephant
>
> I don't have internet at the moment, but what is the definition of int
> vs integer in the sb-alien foreign interface?  Just want to make sure we
> aren't asking for trouble in the future.  (i.e. is int signed or
> unsigned vs integer?  Is it 32bit for compatibility or whatever the clib
> it's linked to says it is?)
>
> Ian
>
> -----Original Message-----
> From: "Petter Egesund" <Petter.Egesund at kunnskapsforlaget.no>
> To: eslick at media.mit.edu
> Cc: "Robert L. Read" <read at robertlread.net>; "Ian Eslick"
> <ieslick at common-lisp.net>; elephant-devel at common-lisp.net
> Sent: 8/22/06 8:18 AM
> Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with
> Elephant
>
> Hi;
>
> I have done some debugging. It seems the problem of serializing integers
> is caused by mixing of the types int and integer in the foreign
> interface?
>
> I change the function read-int in the package memutil.lisp to:
>
> (defun pread-int (buf offset)
>   "Read a 32-bit signed integer from a foreign char buffer."
>   (declare (optimize speed (safety 1) (debug 1))
> 	   (type (alien (* char)) buf)
> 	   (type fixnum offset))
>   (print "Offset: ") (print offset)
>   (the (signed-byte 32)
>     (deref (cast (sap-alien (sap+ (alien-sap buf) offset) (* char))
> 		 (* int))))) ;; !!!!!!!!!!!!! Changed from (* integer)
>
>
> After this the integer-test runs fine. I will do some more testing.
>
> Cheers,
>
> Petter Egesund
>
>
>
>  
>
> -----Opprinnelig melding-----
> Fra: Ian Eslick [mailto:eslick at csail.mit.edu] 
> Sendt: 21. august 2006 16:11
> Til: Petter Egesund
> Kopi: Robert L. Read; eslick at media.mit.edu; Ian Eslick
> Emne: Re: SV: SV: SV: SV: SV: [elephant-devel] Memory error with
> Elephant
>
> Interesting.  That explains the source of our problem (serializing
> integers) but I still don't understand why.  Even if the 4 byte
> assumption in lisp keeps us from expanding the buffer streams
> appropriately the write should still write the first 8 bytes correctly
> and thus be able to read it correctly.  It almost looks like it is
> writing 4 bytes but reading 8.
>
> 1) Try replacing the #x45 with #x00 in my test code
>
> The other test is to see what is actually being put into the buffer
> stream.
>
> 2) replace #x45 with #x40302010 and add the following code instead of
> the buffer-read-int at the end:
>
> (loop for i from 0 upto 16 do
>     (format t "byte ~A: ~A~%" i (buffer-read-byte bs)))
>
> Ian
>
> Petter Egesund wrote:
>   
>>  
>>
>> ----------------------------------------------------------------------
>> --
>> *Fra:* Robert L. Read [mailto:read at robertlread.net]
>> *Sendt:* 19. august 2006 22:26
>> *Til:* Petter Egesund
>> *Kopi:* eslick at media.mit.edu; Ian Eslick
>> *Emne:* Re: SV: SV: SV: SV: [elephant-devel] Memory error with 
>> Elephant
>>
>> Personally, the fact that buffer-write-int in memutil.lisp assumes a 
>> 4-byte integer:
>> (defun buffer-write-int (i bs)
>>   "Write a 32-bit signed integer."
>>   (declare (optimize (speed 3) (safety 0))
>>    (type buffer-stream bs)
>>    (type (signed-byte 32) i))
>>   (with-struct-slots ((buf buffer-stream-buffer)
>>       (size buffer-stream-size)
>>       (len buffer-stream-length))
>>     bs      
>>     (let ((needed (+ size 4)))
>>       (when (> needed len)
>> (resize-buffer-stream bs needed))
>>       (write-int buf i size)
>>       (setf size needed)
>>       nil)))
>> While the c-code version of write-int in libmemutil.c uses the 
>> "sizeof(int)" paradigm seems fragile on a 64-bit architecture:
>>
>> void write_int(char *buf, int num, int offset) {
>>   memcpy(buf+offset, &num, sizeof(int)); }
>>
>> I would like to know if "sizeof(int)" is 8 or 4 as compiled.
>> A simple C program (or reading the f-ing manual) would answer that 
>> question.
>> An awful hack that would let us test things would be to hard-wire "4"
>> in place of
>> "sizeof(int)" in the libmemutil.c file;  I don't know enough about 
>> UFFI and sap-alien to know if that would work.
>>
>> A better solution is to expose a C-function that simply gives the size
>>     
>
>   
>> of an integer in bytes to LISP (it could be added to libmemutil.c 
>> file) and then have memutil.lisp use that in place of the assumption 
>> "4".)
>>
>> How this changes the typing in memutil.lisp, I'm not sure ---- can 
>> we/should we force the
>> AMD64 to use 32-bit integers, or should we rewrite memutil.lisp to 
>> determine based on its compilation environment the size of an integer?
>>     
>
>   
>> Either solution is better than what we have now.
>>
>>
>> On Sat, 2006-08-19 at 20:53 +0200, Petter Egesund wrote:
>>     
>>> It looks like below. Certainly not 35.
>>>
>>> Petter
>>>
>>> -------
>>>
>>> ELE> (defun fixnum-test ()
>>>      (with-buffer-streams (bs)
>>>        (buffer-write-int #x23 bs)
>>>        (buffer-write-int #x45 bs)
>>>        (reset-buffer-stream bs)
>>>        (buffer-read-int bs)))
>>> ; in: LAMBDA NIL
>>> ;     (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 69 ELEPHANT::BS)
>>> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET 
>>> WHEN ; --> COND IF PROGN ; ==>
>>> ;   (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS
>>> ;                                          ELEPHANT-MEMUTIL::NEEDED)
>>> ;
>>> ; note: doing signed word to integer coercion (cost 20)
>>>
>>> ;     (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 35 ELEPHANT::BS)
>>> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET 
>>> WHEN ; --> COND IF PROGN ; ==>
>>> ;   (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS
>>> ;                                          ELEPHANT-MEMUTIL::NEEDED)
>>> ;
>>> ; note: doing signed word to integer coercion (cost 20) ; ; 
>>> compilation unit finished
>>> ;   printed 2 notes
>>> FIXNUM-TEST
>>> ELE> (fixnum-test)
>>> 296352743459
>>> ELE> 
>>>
>>>
>>>
>>>
>>>  
>>>       
>
>   