[slime-devel] CMUCL unicode strings breaks slime

[Helmut Eller]

* Raymond Toy [2010-10-02 16:30] writes:

>> 2) Introduce variants of length and read-sequence that use the same
>>    notion of character as Emacs.  Kinda messy and probably slow, but
>>    relatively easy.
>
> I don't know slime internals, but wouldn't you only need a special
> version of length and read-sequence for cmucl with unicode?  The normal
> length/read-sequence would be fine for everyone else.

Well, I wrote the string with prin1 to a file with CMUCL and read it
back with

(with-open-file (f "/tmp/x.utf8" :external-format :utf-8)
  (length (read f)))

in other implementations.  The results are
CMUCL, Allegro: 4
SBCL, CCL, CLISP: 3,
ABCL: 6

Lispworks didn't want to read the file and ECL didn't accept utf-8
(probably pilot error).

ABCL probably ignores the external format argument, but if they use a
Java-like representation then it would be 4.

Anyway CMUCL and Allegro have the same problem.

>> 
>> 3) Switch from character streams to binary streams so that we can use
>>    byte counts instead of character counts.  This has several
>>    advantages:
>>     - surrogate pairs are no problem
>>     - don't need flexi-streams for Lispworks
>
> Why does Lispworks need flexi-streams? 

Because Lispworks' socket interface has no (documented) option to
specify an encoding.  You can write your own stream classes and
flexi-streams does that (presumably on top of binary streams).

> Does this have to do with using read-byte on character streams or
> read-char on binary streams?

No. At least SLIME doesn't need that.

>>     - it would be easier to switch encoding after connecting
>>     - read/write-sequence is probably faster on byte streams
>>    disadvantageous:
>>     - more consing, and Emacs's GC isn't that good
>>     - need a string-to/from-bytearray function for every backend
> Doesn't every backend already have such a function?  Of course, someone
> has to hook that up, but at least it doesn't have to be written from
> scratch.

Yes, most Lisps will have something like that but we still need to
figure out how it's called for each one.

>>     - breaks third party backends
> Sounds like a show stopper to me.

That's the least of my worries.

Helmut