From rflug05 at gmail.com Mon Aug 4 22:38:03 2008 From: rflug05 at gmail.com (Nick Allen) Date: Tue, 5 Aug 2008 00:38:03 +0200 Subject: [cl-smtp-devel] re: Encoding issues Message-ID: > I haven't managed to use a charset different from latin1 > (like UTF8 or latin15), but that's okay for the time being. Hello Leslie I managed to get utf-8 working with SEND-EMAIL w/ the following http://paste.lisp.org/display/64693 take care Nick From rflug05 at gmail.com Sat Aug 9 11:32:02 2008 From: rflug05 at gmail.com (Nick Allen) Date: Sat, 9 Aug 2008 13:32:02 +0200 Subject: [cl-smtp-devel] Re: Encoding issues In-Reply-To: References: Message-ID: On Tue, Aug 5, 2008 at 12:38 AM, Nick Allen wrote: >> I haven't managed to use a charset different from latin1 >> (like UTF8 or latin15), but that's okay for the time being. > > Hello Leslie > > I managed to get utf-8 working with SEND-EMAIL w/ the following > > http://paste.lisp.org/display/64693 using this code user invites (a user invites another user to a service) got most of our emails stuck in spam filters I've amended the paste http://paste.lisp.org/display/64693#1 I'm not an expert on any of this (in fact I really know nothing about email at all, most of this was accomplished w/ late-night guess and checking), but this code has got 300+ sometimes-ASCII-sometimes-UTF-8 invites sent successfully over the last 3 days w/out any problems except for some what we think are some misconfigured mail-servers in Poland. take care Nick From leslie.polzer at gmx.net Sat Aug 9 20:05:16 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 9 Aug 2008 22:05:16 +0200 (CEST) Subject: [cl-smtp-devel] Re: Encoding issues In-Reply-To: References: Message-ID: <62778.88.73.221.58.1218312316.squirrel@mail.stardawn.org> > I'm not an expert on any of this (in fact I really know nothing about > email at all, most of this was accomplished w/ late-night guess and > checking), but this code has got 300+ sometimes-ASCII-sometimes-UTF-8 > invites sent successfully over the last 3 days w/out any problems > except for some what we think are some misconfigured mail-servers in > Poland. Thanks again! But are you sure you have covered all cases with tests? I have experienced two problems so far: 1. Body is ASCII, Subject contains non-ASCII parts -> email/ascii is used, which fails because of the subject. 2. .qprint-encode/utf-8-email-subject doesn't do what it should for me. Consider: (defun .qprint-encode/utf-8-email-subject (string) (cl-ppcre:regex-replace-all "\\S+" string (list (lambda (word) (format t "word: ~A~%" word) (if (not (.unicode-p word)) word (format nil "=?UTF-8?Q?~A?=" (.qprint-encode/utf-8 word))))) :simple-calls t)) (this is your version with just one format line added for debugging) With tracing enabled: (.qprint-encode/utf-8-email-subject "??? ??? abc") 0: (.QPRINT-ENCODE/UTF-8-EMAIL-SUBJECT "??? ??? abc") 0: .QPRINT-ENCODE/UTF-8-EMAIL-SUBJECT returned "??? ??? abc" NIL "??? ??? abc" So it doesn't really do anything, and the regex doesn't seem to match any word... Am I doing something wrong? Leslie From rflug05 at gmail.com Tue Aug 12 00:32:14 2008 From: rflug05 at gmail.com (Nick Allen) Date: Tue, 12 Aug 2008 02:32:14 +0200 Subject: Fwd: [cl-smtp-devel] Re: Encoding issues In-Reply-To: References: <62778.88.73.221.58.1218312316.squirrel@mail.stardawn.org> Message-ID: I meant to CC this to the whole list ---------- Forwarded message ---------- From: Nick Allen Date: Tue, Aug 12, 2008 at 2:27 AM Subject: Re: [cl-smtp-devel] Re: Encoding issues To: "Leslie P. Polzer" > Thanks again! But are you sure you have covered all cases with tests? no, I'm not sure at all. We have been using this in a specific situations where we send a user invite message in either all ascii or all unicode. I don't know enough about email stuff yet to implement real tests (beyond "did you get the email? yes. was the unicode screwed up. yes"), although I'd like to when I get a chance... > > I have experienced two problems so far: > > 1. Body is ASCII, Subject contains non-ASCII parts > -> email/ascii is used, which fails because of the subject. you are right > > 2. .qprint-encode/utf-8-email-subject doesn't do what it should > for me. Consider: > ..[snip].. > > So it doesn't really do anything, and the regex doesn't seem > to match any word... this regex works fine for me. what cl-ppcre are you using? I'm using the one from http://bknr.net/svn/ediware I don't know if this could have a difference... I made a darcs repo we can use while trying to figure it out darcs get http://common-lisp.net/project/bpm/trivial-email-utf-8 browsable at http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi?r=bpm-trivial-email-utf-8;a=tree it implements SEND-EMAL*, which is just like CL-SMTP:SEND-EMAIL except it knows about qprinting unicode, and INVITE-USER which is like SEND-EMAIL* but for sending email on behalf of a user I haven't had a chance to test it (either with "did you get it?" or real testing), since I'm on my laptop at the moment and am not in a position to connect to our mail server. I will try to do this as soon as I get a chance... take care Nick From leslie.polzer at gmx.net Fri Aug 15 20:00:13 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 15 Aug 2008 22:00:13 +0200 (CEST) Subject: Fwd: [cl-smtp-devel] Re: Encoding issues In-Reply-To: References: <62778.88.73.221.58.1218312316.squirrel@mail.stardawn.org> Message-ID: <60431.84.157.51.234.1218830413.squirrel@mail.stardawn.org> Very nice, thanks for the repo. The branch mismatch is easy to fix, of course. The word encoding problem seems to be a bit more serious, though: >> 2. .qprint-encode/utf-8-email-subject doesn't do what it should >> for me. Consider: >> ..[snip].. >> >> So it doesn't really do anything, and the regex doesn't seem >> to match any word... > > this regex works fine for me. what cl-ppcre are you using? I'm using > the one from > > http://bknr.net/svn/ediware > > I don't know if this could have a difference... This is weird. I just tested on two machines, both of them with cl-ppcre 1.3.2. The following snippet: (cl-ppcre:regex-replace-all "\\S+" "Hello, world!" (list (lambda (word) "MATCH")) :simple-calls t) returns "Hello, world" on one and "MATCH MATCH" on the other... Leslie From rflug05 at gmail.com Fri Aug 15 23:02:52 2008 From: rflug05 at gmail.com (Nick Allen) Date: Sat, 16 Aug 2008 01:02:52 +0200 Subject: Fwd: [cl-smtp-devel] Re: Encoding issues In-Reply-To: <60431.84.157.51.234.1218830413.squirrel@mail.stardawn.org> References: <62778.88.73.221.58.1218312316.squirrel@mail.stardawn.org> <60431.84.157.51.234.1218830413.squirrel@mail.stardawn.org> Message-ID: > This is weird. I just tested on two machines, both of them > with cl-ppcre 1.3.2. The following snippet: > > (cl-ppcre:regex-replace-all "\\S+" "Hello, world!" > (list (lambda (word) "MATCH")) :simple-calls t) > > returns "Hello, world" on one and "MATCH MATCH" on the other... > This is weird. I just tested on two machines, both of them > with cl-ppcre 1.3.2. The following snippet: > > (cl-ppcre:regex-replace-all "\\S+" "Hello, world!" > (list (lambda (word) "MATCH")) :simple-calls t) > > returns "Hello, world" on one and "MATCH MATCH" on the other... Leslie Can you post this on the CL-PPCRE list? Nick From leslie.polzer at gmx.net Tue Aug 19 08:51:56 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 19 Aug 2008 10:51:56 +0200 (CEST) Subject: Fwd: [cl-smtp-devel] Re: Encoding issues Message-ID: <60881.84.157.42.78.1219135916.squirrel@mail.stardawn.org> Had a chance to test TRIVIAL-EMAIL-UTF-8 now. Seems to work great! It's not really trivial, though, is it? ;) Thanks a lot, Nick. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ Stardawn Consulting http://www.stardawn.org/