From edi at agharta.de  Mon Jul 14 07:36:50 2008
From: edi at agharta.de (Edi Weitz)
Date: Mon, 14 Jul 2008 09:36:50 +0200
Subject: [flexi-streams-devel] Re: flexi-streams 1.0.3 and CMUCL 19e
In-Reply-To: <487AB898.8050901@gmail.com> (Eugene Ossintsev's message of
	"Sun, 13 Jul 2008 22:23:20 -0400")
References: <487AB898.8050901@gmail.com>
Message-ID: <uprpg3oj1.fsf@agharta.de>

Hi Eugene,

On Sun, 13 Jul 2008 22:23:20 -0400, Eugene Ossintsev <eugoss at gmail.com> wrote:

> I was trying to compile flexi-streams 1.0.3 with CMUCL 19e and it
> stopped with the following message in ascii.lisp :
>
> --------------------------------------------------------
> Type-error in KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER:
>     65533 is not of type (UNSIGNED-BYTE 8)
>     [Condition of type TYPE-ERROR]
> --------------------------------------------------------
>
> I don't know if it's my fault or not, I'm not experienced enough
> yet. But as I see, in MAKE-DECODING-TABLE the function MAKE-ARRAY
> has :element-type 'char-code-integer.
>
> CHAR-CODE-INTEGER is defined as
>
> (deftype char-code-integer ()
>    "The subtype of integers which can be returned by the function CHAR-CODE."
>    '(integer 0 #.(1- char-code-limit)))
>
> But at least in CMUCL 19e CHAR-CODE-LIMIT is 256, that is not enough
> to hold 65533. I'm afraid it's about Unicode-non-Unicode characters,
> right?

Please send but reports to the mailing list (see Cc).

I'm not sure what the best thing to do is w.r.t. CMUCL.  Should we
declare CHAR-CODE-INTEGER to be the same as FIXNUM on CMUCL?  Or
(integer 0 65535)?  Or should at least MAKE-DECODING-TABLE be
modified?  Opinions?

Cheers,
Edi.

PS: Eugene, as a workaround please try the following in mapping.lisp
    and report to the list if you still have problems:

(defun make-decoding-table (list)
  "Creates and returns an array which contains the elements in the
list LIST and has an element type that's suitable for character
codes."
  (make-array (length list)
              :element-type 'fixnum
              :initial-contents list))


From eugoss at gmail.com  Wed Jul 16 03:00:31 2008
From: eugoss at gmail.com (Eugene Ossintsev)
Date: Tue, 15 Jul 2008 23:00:31 -0400
Subject: [flexi-streams-devel] flexi-streams 1.0.3 and CMUCL 19e
Message-ID: <487D644F.3080709@gmail.com>

Hi,

I was trying to compile flexi-streams 1.0.3 with CMUCL 19e and it stopped with 
the following message in ascii.lisp :

--------------------------------------------------------
Type-error in KERNEL::OBJECT-NOT-TYPE-ERROR-HANDLER:
    65533 is not of type (UNSIGNED-BYTE 8)
    [Condition of type TYPE-ERROR]
--------------------------------------------------------

As I see, in MAKE-DECODING-TABLE the function MAKE-ARRAY has
:element-type 'char-code-integer.


And CHAR-CODE-INTEGER is defined as

--------------------------------------------------------
(deftype char-code-integer ()
   "The subtype of integers which can be returned by the function CHAR-CODE."
   '(integer 0 #.(1- char-code-limit)))
--------------------------------------------------------

But in CMUCL CHAR-CODE-LIMIT is 256, that is not enough to hold 65533.


If I modify mapping.lisp the following way (see also the attached patch):

--------------------------------------------------------
  (deftype char-code-integer ()
    "The subtype of integers which can be returned by the function CHAR-CODE."
-  '(integer 0 #.(1- char-code-limit)))
+  #-:cmu '(integer 0 #.(1- char-code-limit))
+  #+:cmu '(integer 0 65533))
--------------------------------------------------------


it's compiled with CMUCL successfully and a simple Hunchentoot web application 
of mine runs without any problem.

But if I try to perform the tests from flexi-streams/test I get a bunch of 
successful results

--------------------------------------------------------
FLEXI-STREAMS-TEST> (run-tests)
Test (using COPY-STREAM) "kafka_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [INPUT]
  --> (:UTF-8 :EOL-STYLE :LF) [OUTPUT].

Test (using COPY-STREAM) "kafka_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [IO]
  --> (:UTF-8 :EOL-STYLE :LF) [OUTPUT].

Test (using COPY-STREAM) "kafka_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [INPUT]
  --> (:UTF-8 :EOL-STYLE :LF) [IO].

   [[ skipped; too many to put them all here ]]
--------------------------------------------------------


followed by some failures (all with the hebrew* files):

--------------------------------------------------------
Test (using COPY-STREAM) "hebrew_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [INPUT]
  --> (:UTF-8 :EOL-STYLE :LF) [OUTPUT].
  Test failed!!!

   [[ skipped ]]

Test (using COPY-STREAM) "hebrew_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [IO]
  --> (:UTF-8 :EOL-STYLE :CRLF) [IO].
  Test failed!!!

Test (using COPY-STREAM) "hebrew_utf8_lf.txt" (:UTF-8 :EOL-STYLE :LF) [INPUT]
  --> (:ISO-8859-8 :EOL-STYLE :LF) [OUTPUT].
--------------------------------------------------------


and eventually, hello the debugger!

--------------------------------------------------------
#\? (code NIL) is not in this encoding.
    [Condition of type EXTERNAL-FORMAT-ENCODING-ERROR]

Restarts:
  0: [ABORT] Return to SLIME's top level.
  1: [ABORT] Return to Top-Level.

Backtrace:
   0: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR #<FLEXI-STREAMS::FLEXI-8-BIT-FORMAT 
(:ISO-8859-8 :EOL-STYLE :LF) {5A537DA5}> "~S (code ~A) is not in this encoding." 
#\? NIL)
   1: (FLEXI-STREAMS::WRITE-SEQUENCE* #<#1=unavailable-arg> #<#1#> #<#1#> #<#1#> 
...)
   2: (TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE #<#1=unavailable-arg> #<#1#> 
#<#1#> #<#1#> ...)
   3: (LISP::WRITE-LINE* ":???? ??? ????? ?? ????? ??? ?????? 	?  1" 
#<FLEXI-OUTPUT-STREAM {59392C6D}> 0 41)
   4: (COPY-STREAM #<Stream for file 
"/home/eugene/lisp/flexi-streams-1.0.3/test/hebrew_utf8_lf.txt"> 
#<FLEXI-STREAMS::FLEXI-UTF-8-FORMAT (:UTF-8 :EOL-STYLE :LF) {5A537D75}> #<Stream 
for file "/tmp/odd-streams-test/hebrew_latin8_lf.txt"> 
#<FLEXI-STREAMS::FLEXI-8-BIT-FORMAT (:ISO-8859-8 :EOL-STYLE :LF) {5A537DA5}>)
   5: (COPY-FILE 
#P"/home/eugene/lisp/flexi-streams-1.0.3/test/hebrew_utf8_lf.txt" 
#<FLEXI-STREAMS::FLEXI-UTF-8-FORMAT (:UTF-8 :EOL-STYLE :LF) {5A537D75}> 
#P"/tmp/odd-streams-test/hebrew_latin8_lf.txt" 
#<FLEXI-STREAMS::FLEXI-8-BIT-FORMAT (:ISO-8859-8 :EOL-STYLE :LF) {5A537DA5}> ...)
   6: (COMPARE-FILES "hebrew_utf8_lf.txt" #<FLEXI-STREAMS::FLEXI-UTF-8-FORMAT 
(:UTF-8 :EOL-STYLE :LF) {5A537D75}> "hebrew_latin8_lf.txt" 
#<FLEXI-STREAMS::FLEXI-8-BIT-FORMAT (:ISO-8859-8 :EOL-STYLE :LF) {5A537DA5}>)
   7: (RUN-TESTS)
  --more--
--------------------------------------------------------



I don't know if it's the same or similar for the other Common Lisp 
implementations, but that patch allows flexi-streams at least to be compiled 
with CMUCL.

I know it's far from being perfect. I hope there are much more experienced 
people here, who know how to fix it.



- Eugene
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mapping.lisp.diff
Type: text/x-diff
Size: 453 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/flexi-streams-devel/attachments/20080715/401d428c/attachment.diff>

From edi at agharta.de  Fri Jul 25 10:00:18 2008
From: edi at agharta.de (Edi Weitz)
Date: Fri, 25 Jul 2008 12:00:18 +0200
Subject: [flexi-streams-devel] New release 1.0.4
Message-ID: <uiquuxozx.fsf@agharta.de>

ChangeLog:

  Version 1.0.4
  2008-07-25
  Cosmetic surgery on test suite

Download:

  http://weitz.de/files/flexi-streams.tar.gz