[tbnl-devel] UTF-8 problems -- patch

[Ivan Shvedunov]

Hello.
Well, I've promised this patch somewhat earlier, but I didn't have
time to complete it... I've discovered several problems with TBNL's
handling of UTF-8. Namely, there was a problem with url-decode in
util.lisp which was turning UTF-8 urlencoded strings into something
incomprehensible, and also there was problem with Content-Length in
modlisp.lisp which was causing UTF-8 content to be truncated.

The attached patch works only with SBCL. I mean that it shouldn't
break other Lisps, but
proper unicode hanling is implemented only for SBCL. I've tried to
make it work with Allegro demo/LispWorks Personal Edition, but with no
luck. Well, concerning Allegro, the problem here is that sockets that
are used to talk to mod_lisp are set to latin-1 encoding for some
reason, most likely KMRCL needs to be fixed a bit, again,
unfortunatelly I just have no time to complete this. As of LispWorks,
I just don't know how to turn a string into series of octets and vice
versa using current encoding - i.e. I didn't find something like
Allegro/SBCL octets-to-string/string-to-octets there.

Concerning implementation - I've introduced :tbnl-unicode feature that
is set for supported
Unicode-aware Lisps in specials.lisp (I'm setting it for Allegro and
SBCL, thogh it doesn't help Allegro much). Also I've added supporting
funcs, bytes-to-string and string-to-bytes (defined only when
#+tbnl-unicode) that do the dirty job of string conversion.

Ivan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tbnl-0.5.5-unicode.patch
Type: text/x-patch
Size: 3861 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/tbnl-devel/attachments/20050728/5cfe3f96/attachment.bin>