From luismbo at gmail.com  Mon May 31 18:57:44 2010
From: luismbo at gmail.com (=?ISO-8859-1?Q?Lu=EDs_Oliveira?=)
Date: Mon, 31 May 2010 19:57:44 +0100
Subject: [babel-devel] [cffi-devel] utf-16
In-Reply-To: <m2vda42qvv.fsf@common-lisp.net>
References: <m2vda42qvv.fsf@common-lisp.net>
Message-ID: <AANLkTimLRlz1vAs9FN_XUGxNgXLDC5WDaB2oJtzcQX1T@mail.gmail.com>

On Mon, May 31, 2010 at 1:32 PM, Helmut Eller <heller at common-lisp.net> wrote:
> Is this assertion
>
> (assert
> ?(equal
> ?(cffi:with-foreign-string (data "foobar" :encoding :utf-16)
> ? ?(cffi:foreign-string-to-lisp data :count 6 :encoding :utf-16))
> ?"foobar"))
>
> expected to fail?

FOREIGN-STRING-ALLOC currently doesn't insert BOM markers and the
:UTF-16 encoder uses native endianness. The :UTF-16 decoder defaults
to big endian when a BOM marker is not present, as recommended by the
Unicode specification. That's why the assertion fails.

One solution is to use :UTF-16BE or :UTF-16LE explicitly, which
unfortunately aren't implemented in Babel yet.

-- 
Lu?s Oliveira
http://r42.eu/~luis/