From leslie.polzer at gmx.net  Tue Jul 29 21:02:46 2008
From: leslie.polzer at gmx.net (Leslie P. Polzer)
Date: Tue, 29 Jul 2008 23:02:46 +0200 (CEST)
Subject: [cl-smtp-devel] Encoding issues
Message-ID: <61866.88.73.249.231.1217365366.squirrel@mail.stardawn.org>


Hi,

I'm having trouble getting my German umlauts and other
special characters across the SMTP pipe.

Without any effort by me, some parts get annotated inline:

  Sei gegr??t -> Sei gegr=?ISO-8859-15?Q?=FC=DFt

and some vanish completely:

  ben?tigt -> bentigt

I tried converting my UTF8 strings to latin1 via flexi-streams
and Babel:

  (babel:octets-to-string
    (babel:string-to-octets
      "Sei gegr??t, ben?tigt"
      :encoding :utf-8)
    :encoding :latin-1)

-> "Sei gegr????t, ben??tigt"

but CL-SMTP refuses the body with:

  debugger invoked on a FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR
    in thread #<THREAD "initial thread" {A72A5D9}>:
  #\VULGAR_FRACTION_ONE_QUARTER is not in this encoding.

Partial backtrace:

0: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR
    #<FLEXI-STREAMS::FLEXI-8-BIT-OUTPUT-STREAM {B35E001}>
    "~S is not in this encoding.")
1: ((SB-PCL::FAST-METHOD FLEXI-STREAMS::CHAR-TO-OCTETS
     (FLEXI-STREAMS::FLEXI-8-BIT-OUTPUT-STREAM T T))
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>)
2: ((SB-PCL::FAST-METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE
     (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM STRING T T))
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>
    #<unavailable argument>)
3: (WRITE-STRING
    "?"
    #<FLEXI-STREAMS::FLEXI-8-BIT-OUTPUT-STREAM {B35E001}>
    :START
    0
    :END
    NIL)
4: (FLEXI-STREAMS:STRING-TO-OCTETS #<unavailable argument>)
5: ((SB-PCL::FAST-METHOD SB-GRAY:STREAM-WRITE-CHAR [...]

Also, latin1 seems to be hard-coded at a bunch of locations
in CL-SMTP.

I'm using Hans' BKNR version of CL-SMTP and would
appreciate any help.

  Cheers,

    Leslie

-- 
LinkedIn Profile: http://www.linkedin.com/in/polzer
Blog: http://blog.viridian-project.de/



From hans at huebner.org  Tue Jul 29 21:21:21 2008
From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=)
Date: Tue, 29 Jul 2008 23:21:21 +0200
Subject: [cl-smtp-devel] Encoding issues
In-Reply-To: <61866.88.73.249.231.1217365366.squirrel@mail.stardawn.org>
References: <61866.88.73.249.231.1217365366.squirrel@mail.stardawn.org>
Message-ID: <fd2982320807291421m2fe4ea53u6ca7d0acbfba5a48@mail.gmail.com>

Hi Leslie,

I cannot really help much with CL-SMTP:SEND-SMTP as I am not using it
myself and don't understand the code quite well.  I use
CL-SMTP:WITH-SMTP-MAIL and CL-MIME for my mail composition purposes,
which gives me more control over the format and encoding of the email.

That said, some general comments:  All non-ASCII characters in email
headers are automatically encoded using quoted printable encoding.  In
bodies, no such automatic conversion is performed.

Have a look at http://bknr.net/trac/browser/trunk/projects/bos/m2/mail-generator.lisp#L53,
which is the mail sending routine in one of my projects.  It works
with non-ASCII characters.  You may be able to adopt it for your
purposes.

Good luck,
Hans

On Tue, Jul 29, 2008 at 23:02, Leslie P. Polzer <leslie.polzer at gmx.net> wrote:
>
> Hi,
>
> I'm having trouble getting my German umlauts and other
> special characters across the SMTP pipe.
>
> Without any effort by me, some parts get annotated inline:
>
>  Sei gegr??t -> Sei gegr=?ISO-8859-15?Q?=FC=DFt
>
> and some vanish completely:
>
>  ben?tigt -> bentigt
>
> I tried converting my UTF8 strings to latin1 via flexi-streams
> and Babel:
>
>  (babel:octets-to-string
>    (babel:string-to-octets
>      "Sei gegr??t, ben?tigt"
>      :encoding :utf-8)
>    :encoding :latin-1)
>
> -> "Sei gegr????t, ben??tigt"
>
> but CL-SMTP refuses the body with:
>
>  debugger invoked on a FLEXI-STREAMS:FLEXI-STREAM-ENCODING-ERROR
>    in thread #<THREAD "initial thread" {A72A5D9}>:
>  #\VULGAR_FRACTION_ONE_QUARTER is not in this encoding.
>
> Partial backtrace:
>
> 0: (FLEXI-STREAMS::SIGNAL-ENCODING-ERROR
>    #<FLEXI-STREAMS::FLEXI-8-BIT-OUTPUT-STREAM {B35E001}>
>    "~S is not in this encoding.")
> 1: ((SB-PCL::FAST-METHOD FLEXI-STREAMS::CHAR-TO-OCTETS
>     (FLEXI-STREAMS::FLEXI-8-BIT-OUTPUT-STREAM T T))
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>)
> 2: ((SB-PCL::FAST-METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE
>     (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM STRING T T))
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>
>    #<unavailable argument>)
> 3: (WRITE-STRING
>    "?"
>    #<FLEXI-STREAMS::FLEXI-8-BIT-OUTPUT-STREAM {B35E001}>
>    :START
>    0
>    :END
>    NIL)
> 4: (FLEXI-STREAMS:STRING-TO-OCTETS #<unavailable argument>)
> 5: ((SB-PCL::FAST-METHOD SB-GRAY:STREAM-WRITE-CHAR [...]
>
> Also, latin1 seems to be hard-coded at a bunch of locations
> in CL-SMTP.
>
> I'm using Hans' BKNR version of CL-SMTP and would
> appreciate any help.
>
>  Cheers,
>
>    Leslie
>
> --
> LinkedIn Profile: http://www.linkedin.com/in/polzer
> Blog: http://blog.viridian-project.de/
>
> _______________________________________________
> cl-smtp-devel mailing list
> cl-smtp-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/cl-smtp-devel
>


From leslie.polzer at gmx.net  Wed Jul 30 09:04:40 2008
From: leslie.polzer at gmx.net (Leslie P. Polzer)
Date: Wed, 30 Jul 2008 11:04:40 +0200 (CEST)
Subject: [cl-smtp-devel] Encoding issues
In-Reply-To: <fd2982320807291421m2fe4ea53u6ca7d0acbfba5a48@mail.gmail.com>
References: <61866.88.73.249.231.1217365366.squirrel@mail.stardawn.org>
	<fd2982320807291421m2fe4ea53u6ca7d0acbfba5a48@mail.gmail.com>
Message-ID: <61664.88.73.215.40.1217408680.squirrel@mail.stardawn.org>


> Have a look at
> http://bknr.net/trac/browser/trunk/projects/bos/m2/mail-generator.lisp#L53,
> which is the mail sending routine in one of my projects.  It works
> with non-ASCII characters.  You may be able to adopt it for your
> purposes.

Thanks, that helped a lot! For me, it's now:

  (cl-smtp:with-smtp-mail (smtp "localhost" from (list to))
    (format smtp "Subject: ~A
Content-Type: text/plain; charset=ISO-8859-1
Mime-Version: 1.0
Content-Transfer-Encoding: quoted-printable

~A" subject (cl-qprint:encode body)))

I haven't managed to use a charset different from latin1
(like UTF8 or latin15), but that's okay for the time being.

  Leslie