[cmucl-ticket] Re: #7: Bug in read-vector

[cmucl]

#7: Bug in read-vector
--------------------------+-------------------------------------------------
  Reporter:  jcunningham  |       Owner:  somebody   
      Type:  defect       |      Status:  new        
  Priority:  major        |   Milestone:             
 Component:  Core         |     Version:  19c        
Resolution:               |    Keywords:  read-vector
--------------------------+-------------------------------------------------
Old description:

> I have been trying to read large binary files of floating point data
> using CMUCL (19c). I thought I would have to do it using some form of
> FFI and went to comp.lang.lisp for help getting that working. I
> succeeded. But Duane Rettig at Allegro suggested it would be easier to
> use 'read-vector. So I tried that as follows:
>
> (let ((vec (make-array 10 :element-type 'double-float)))
>   (with-open-file (os "d10.bin")
>     (read-vector vec os)
>     (print vec)))
>
> where "d10.bin" is a double-float binary file containing 10
> elements. When I try to read the file it produces the following error:
>
> Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER:
>    #\Null is not of type DOUBLE-FLOAT
>
> Here is the C-code code I used to produce the file:
>
> #include <iostream>
> #include <fstream>
> #include <complex>
> using namespace std;
> int main()
> {
>   int n=10;
>   double *d = new double[n];
>   for (int i=0; i<n; i++)
>     d[i] = i;
>   FILE *of = fopen("d10.bin", "wb");
>   fwrite(f,8,n,of);
>   fclose(of);
>   return 0;
> }
>

> Here is Duane's sample code that he says works in Allegro along with
> comments:
> (copied from
> <http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe>)
> ...................................................................
> "You must be using a simple-streams implementation from another lisp.
> Allegro CL doesn't have a KERNEL package.
>
> What you're seeing above is a bug; you should report it to that
> lisp's support team.
>
> It works fine in Allegro CL (for which you can download the Express
> Edition for free):
>
> [edited formating for readability on Trac - jkc]
> (defvar *x*
>   (make-array 10 :element-type 'double-float
>      :initial-contents
>      (loop for i from 0.0d0 to 9.0d0 collect i)))
>
> #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)
>
> (with-open-file (s "z.dat" :direction :io
>   :if-exists :overwrite
>   :if-does-not-exist :create)
> (write-vector *x* s))
> 80
> (defvar *y* (make-array 10 :element-type 'double-float :initial-element
> 10.0d0))
>
> (with-open-file (s "z.dat")
> (read-vector *y* s))
>
> *y*
> #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)
>
> --
> Duane Rettig"
>

> Further correspondence with Raymond Toy corroborates this is a bug:
>
> "Yes, this does appear to be a bug in the implementation of
> read-vector.
>
> You can, however, achieve what you want by opening the file with an
> element-type of, say, (unsigned-byte 8), instead of the default
> 'character.
>
> I'll have to read some more to understand how read-vector interacts
> with the stream element type.  It seems, though, that the element-type
> of the vector overrides the element-type of the stream, more or less.
> Currently, a stream element type of character basically causes the
> code to read in characters.
>
> Ray"
>
> If you need more information, let me know and I'll see what I can do.
>
> -jeff

New description:

 I have been trying to read large binary files of floating point data
 using CMUCL (19c). I thought I would have to do it using some form of
 FFI and went to comp.lang.lisp for help getting that working. I
 succeeded. But Duane Rettig at Allegro suggested it would be easier to
 use 'read-vector. So I tried that as follows:
 {{{
 (let ((vec (make-array 10 :element-type 'double-float)))
   (with-open-file (os "d10.bin")
     (read-vector vec os)
     (print vec)))
 }}}
 where "d10.bin" is a double-float binary file containing 10
 elements. When I try to read the file it produces the following error:

 Type-error in KERNEL::OBJECT-NOT-DOUBLE-FLOAT-ERROR-HANDLER:
    #\Null is not of type DOUBLE-FLOAT

 Here is the C-code code I used to produce the file:
 {{{
 #include <iostream>
 #include <fstream>
 #include <complex>
 using namespace std;
 int main()
 {
   int n=10;
   double *d = new double[n];
   for (int i=0; i<n; i++)
     d[i] = i;
   FILE *of = fopen("d10.bin", "wb");
   fwrite(f,8,n,of);
   fclose(of);
   return 0;
 }
 }}}

 Here is Duane's sample code that he says works in Allegro along with
 comments:
 (copied from
 <http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/67876101085aee82/05a20cfcd11f8fbe?hl=en#05a20cfcd11f8fbe>)
 ...................................................................
 "You must be using a simple-streams implementation from another lisp.
 Allegro CL doesn't have a KERNEL package.

 What you're seeing above is a bug; you should report it to that
 lisp's support team.

 It works fine in Allegro CL (for which you can download the Express
 Edition for free):

 [edited formating for readability on Trac - jkc]
 {{{
 (defvar *x*
   (make-array 10 :element-type 'double-float
      :initial-contents
      (loop for i from 0.0d0 to 9.0d0 collect i)))

 #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)

 (with-open-file (s "z.dat" :direction :io
   :if-exists :overwrite
   :if-does-not-exist :create)
 (write-vector *x* s))
 80
 (defvar *y* (make-array 10 :element-type 'double-float :initial-element
 10.0d0))

 (with-open-file (s "z.dat")
 (read-vector *y* s))

 *y*
 #(0.0d0 1.0d0 2.0d0 3.0d0 4.0d0 5.0d0 6.0d0 7.0d0 8.0d0 9.0d0)
 }}}
 --
 Duane Rettig"


 Further correspondence with Raymond Toy corroborates this is a bug:

 "Yes, this does appear to be a bug in the implementation of
 read-vector.

 You can, however, achieve what you want by opening the file with an
 element-type of, say, (unsigned-byte 8), instead of the default
 'character.

 I'll have to read some more to understand how read-vector interacts
 with the stream element type.  It seems, though, that the element-type
 of the vector overrides the element-type of the stream, more or less.
 Currently, a stream element type of character basically causes the
 code to read in characters.

 Ray"

 If you need more information, let me know and I'll see what I can do.

 -jeff

Comment (by rtoy):

 First I added a bit of markup so that the code bits are formatted better.

 Now for the bug report, there are a couple of issues here.  CMUCL has an
 extension, READ-VECTOR, which is similar to the SIMPLE-STREAM's READ-
 VECTOR, but not quite.  In particular, it produces the bug that you see.
 However, if you {{{(require 'simple-streams)}}}, READ-VECTOR will be
 replaced with the simple-stream version.  Then, the following snip of code
 works as Duane mentions:

 {{{
 (let ((vec (make-array 10 :element-type 'double-float)))
            (with-open-file (os "d10.bin" :class 'stream:file-simple-
 stream)
              (stream:read-vector vec os) (print vec)))
 }}}

 The important part is that you open the file with the appropriate class.

 Not sure what to do.  We can change READ-VECTOR to match simple-stream
 READ-VECTOR.
 (BTW, READ-VECTOR is much faster than STREAM:READ-VECTOR.)

-- 
Ticket URL: <http://trac.common-lisp.net/cmucl/ticket/7>
cmucl <http://common-lisp.net/project/cmucl>
cmucl