[babel-devel] [cffi-devel] utf-16

[Luís Oliveira]

On Mon, May 31, 2010 at 1:32 PM, Helmut Eller <heller at common-lisp.net> wrote:
> Is this assertion
>
> (assert
>  (equal
>  (cffi:with-foreign-string (data "foobar" :encoding :utf-16)
>    (cffi:foreign-string-to-lisp data :count 6 :encoding :utf-16))
>  "foobar"))
>
> expected to fail?

FOREIGN-STRING-ALLOC currently doesn't insert BOM markers and the
:UTF-16 encoder uses native endianness. The :UTF-16 decoder defaults
to big endian when a BOM marker is not present, as recommended by the
Unicode specification. That's why the assertion fails.

One solution is to use :UTF-16BE or :UTF-16LE explicitly, which
unfortunately aren't implemented in Babel yet.

-- 
Luís Oliveira
http://r42.eu/~luis/