[cells-gtk-devel] UTF-8 support -- Patch

[Peter Hildebrandt]

Hi Ingo,

> thank you for your cumulative patch. I'm really looking forward to 
> playing with cells-trie-view, cairo and the threading code.

The cells-tree-view works nicely (c.f. the demo in test-gtk).

The cairo stuff is still in the making.  I'm trying to factor out the 
common elements of a drawing area canvas and then use that as a 
superclass for the cairo-drawing-area (using Tamas' cl-cairo2) and the 
gl-drawing-area (using gtk-gl-ext).

I need both for my project (the cairo part is for a graphic network 
editor, the gl part for a 3d physics simulator based on cl-ode), so 
chances are I'll finish them soon.


>> I fixed that one, too.  The problem was roughly that the string was
>> converted to utf-8 *twice*.  cells-gtk read a utf-8 string back from
>> gtk, forgot to decode it, then encoded it *again* and sent it back to gtk
> 
> In my hands utf-8-to-lisp didn't work for multi byte characters 
> like #\REGISTERED_SIGN. I changed it to
> 
>    (defun utf-8-to-lisp (str)
>       (when str
>        #+sbcl (sb-ext:octets-to-string 
>                  (sb-ext:string-to-octets str :external-format :latin1) 
>                  :external-format :utf-8)
>        #-(or sbcl) str))
> 
> and now everything works like a charm.

You got me there.  Things are more complicated than both of us thought 
(I believe).  Your stuff is obvious, and that's what I had at first. 
Turned out that e.g. the entry widget returned crap with that.

I believe you spotted the problem (see below).

> Btw, find the patch in my git repository at
> http://public.efil.de/gitweb/?p=cells-gtk/.git;a=commitdiff;h=48c5592f984460269bb09415b472a8495db73251


Yep.  Saw that.  Maybe there's a point in having a common develop 
CVS/SVN/git somewhere?  (I prefer CVS because  then I don't have to 
reconfigure emacs ;-))

> Btw', what do you think about putting all utf8 trancoding directly 
> into gtk-ffi's macro def-gtk-function?

Actually, check out gtk-ffi.lisp:

(defmacro def-gtk-function (library name return-type arguments)
[...]
(let
   ((result
      ,(let ((fn `(,gtk-name ,@(mapcar #'(lambda (arg)
              (if (eql (cadr arg) :gtk-string)								 
`(lisp-to-utf-8 ,(car arg))
									    (car arg)))
							arguments))))
[...]

i.e. whenever a function accepts a gtk-string as a parameter, it is 
first processed lisp-to-utf-8.

>    (if (equal return-type :string)
>        (gtk-string-to-lisp result)
>        result)
> 
> to have all conversions handled transparently.

This one is more tricky (Yes, I thought about it, too).  :gtk-string as 
a return type is a pointer, not a lisp data structure.  That is, you 
can't process it directly using sb-ext:octets-to-string.  The calling 
function usually makes some uffi call to create a lisp string from the 
pointer.

OTOH, there is only five or six functions returning strings (grep 
:gtk-string$ *.lisp).  Most of them are file boxes, and they seem to be 
fine.

> Lisp-to-gtk-string should be similar to to-gtk-string.
> Note, to-gtk-string uses g-locale-to-utf8 which is itself 
> defined using def-gtk-lib-functions.

And here things get complicated.  It is up to the calling function in 
cells-gtk to handle to-gtk-string, local-to-utf8 etc.  And it looks like 
cells-gtk sometimes does it one way and sometimes the other.

The title/label/text setting functions plainly pass the lisp string and 
do not care about locale conversions.  So the def-gtk-lib... hook that 
is currently in place does the right thing.  Reading stuff back from an 
entry is easy, since cffi:translate-from-foreign is the right place for 
the hook.

The tree-view/list-view is a lot more complex, since they use the whole 
locale-to-utf8 conversion.  There are g-values and this runtime-typing 
function (in gtk-utilities I believe) ... and honestly, I don't know 
where you should hook in there.

> I'd just be interested in what you think of the approach
> and whether I missed something here.

I think you understand the mess at least as well as I do.  Where to go 
from here is another question.  I believe your patch breaks the entry 
widget.

The problem really seems to be that we do the utf-8 conversion sometimes 
twice.  And I am not quite sure how to deal with this.

Maybe it is an option to "blacklist" the locale/utf8 functions so that 
they don't automatically call the sb-ext conversion?

If you want to look into that, go ahead.  I'm working on the object 
inspector (an extension of the tree view) and the drawing areas right 
now, so it's all yours.

Peter


>> ... (BTW, it is "whether" ;-) )
>>
> 
> Danke, die beide bekomme ich immer durcheinander.
> Mal sehn' Wetter ich das in Zukunft hinbekomme :)

:-) Viel Erfolg!

> Ingo
> 
>