From avodonosov at yandex.ru Sun Mar 23 03:22:12 2008 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sun, 23 Mar 2008 05:22:12 +0200 Subject: [cl-smtp-devel] patch: 1) newline conversions; 2) ssl modes Message-ID: <1689548457.20080323052212@yandex.ru> Hello. Current cl-smtp code assumes that socket stream converts #\Newline to one byte 0xA (LF). It is not necessary true. For instance in CLISP on Windows default newline conversion is CRLF. I.e. by #\Return #\NewLine we send CRCRLF to server instead of CRLF. The suggestion is to create binary socket stream and wrap it by flexi-stream with explicitly specified newline conversion. The second issue is SSL functionality. As Tim Ritchey noted in his blog (http://redromelogic.com/blog/display?id=20) SSL connection to the port 465 do not requires STARTTLS negotiation. In the mail agent The Bat also there are three modes for smtp connection: - usual (default port 25) - secure to standard port (STARTTLS, default port 25) - secure to special port (TLS, default port 465). So I would suggest instead of boolean parameter SSL introduce parameter SECURE with three possible values: nil, :tls and :starttls. And make default value of the PORT depend on the SECURE value. The changes proposed are in the attached patch. BTW, it would be good to update the tarball linked from the project page and from cliki to current CVS state. Best regards, -- Anton -------------- next part -------------- A non-text attachment was scrubbed... Name: cl-smtp.diff Type: application/octet-stream Size: 13928 bytes Desc: not available URL: From hans at huebner.org Sun Mar 23 18:26:04 2008 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Sun, 23 Mar 2008 19:26:04 +0100 Subject: [cl-smtp-devel] Patch to support sending raw messages Message-ID: Hi, I recently spent some time with cl-smtp to make it support sending raw messages - I am using cl-mime for my message composition purposes, and I did not want to extend cl-smtp to support yet more headers. While at it, I refactored cl-smtp a bit: - Sending a command and checking the response is in the SMTP-COMMAND function - SMTP protocol handling and message composition is separated a bit - Some error reporting is now done through specific condition classes. This makes it possible to handle invalid recipients programmatically without having to re-send the whole message (cl-smtp did not report which recipient failed) - Some docstrings and comments have been added - I fixed STARTTLS so that an EHLO command is sent after STARTTLS. This is required by some servers to switch them into ESMTP mode - Determination of the AUTH protocol is now automatic, by reading the features list sent by the server as a response to the EHLO command. The new WITH-SMTP-MAIL macro is what I am now using to send my raw messages. The rest of the API is mostly unchanged (except that the method specified in the AUTHENTICATION argument is now ignored). When I was finished with my initial hacking round today, I found that some of the changes overlap with what Anton Vodonosov has recently posted. My bad! I have then added his functionality to support raw TLS in addition to STARTTLS. The SSL keyword argument may now either be NIL (no TLS), T or :STARTTLS (use STARTTLS as before) or :TLS for raw TLS. What I did not find in his patch was the mentioned change in CR/LF handling. I'd like to fix that, too, but I would want to do that on the stream level rather than requiring the message to be all in memory. One could certainly spend more time with enhancing cl-smtp. Before further hacking, I'd like to hear if anyone else has needs or plans and if Jan would agree to update cl-smtp with these changes. Please review and let me know what you think. Thanks! -Hans -------------- next part -------------- A non-text attachment was scrubbed... Name: cl-smtp.diff Type: application/octet-stream Size: 27221 bytes Desc: not available URL: From avodonosov at yandex.ru Sun Mar 23 22:53:54 2008 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Mon, 24 Mar 2008 00:53:54 +0200 Subject: [cl-smtp-devel] Patch to support sending raw messages In-Reply-To: References: Message-ID: <1668532236.20080324005354@yandex.ru> Hans wrote: > What I did not find in his patch was the mentioned change in > CR/LF handling. I'd like to fix that, too, but I would want to do > that on the stream level rather than requiring the message to be all > in memory. What I mean is the following lines: (defun open-smtp-connection (sock &key authentication secure) (let ((stream (flex:make-flexi-stream sock :external-format '(:latin-1 :eol-style :lf)))) and underlying socket stream is binary: (defun send-smtp (host from to subject message &key port cc bcc reply-to extra-headers html-message display-name authentication attachments buffer-size secure) (assert-one-of secure '(nil :tls :starttls)) (when (null port) (setf port (if (eq secure :tls) 465 25))) (let* ((sock (usocket:socket-stream (usocket:socket-connect host port :element-type '(unsigned-byte 8)))) I.e. I encountered problem with CRLF transformation in initial communication with server (HELLO, EHLO, etc.), even before sending a message body. > Before further hacking, I'd like to hear if anyone else has needs > or plans ... > Please review and let me know what you think. Thanks! > -Hans As for me, I'm interested in having one common version of the library, so that my soft will be compatible with it and I will be able to easy migrate to new releases. So I'll take a more careful look at your patch little bit later and if find something to discuss, send my comments. Best regards, -Anton From hans at huebner.org Mon Mar 24 07:58:57 2008 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 24 Mar 2008 08:58:57 +0100 Subject: [cl-smtp-devel] Patch to support sending raw messages In-Reply-To: <1668532236.20080324005354@yandex.ru> References: <1668532236.20080324005354@yandex.ru> Message-ID: Hi Anton and whoever is interested, I made yet another patch to CL-SMTP so that it performs automatic conversion of non-ASCII characters in email headers to quoted characters according to RFC2047. This is implemented as a specialized stream class SMTP-OUTPUT-STREAM that is used during message output. The SMTP-OUTPUT-STREAM class also does conversion of #\Newline to #\Return #\Newline - This might be better done with a properly set up FLEXI-STREAMS encoding; I'm not sure. One thing that might usefully be added to SMTP-OUTPUT-STREAM is dot masking. Dot masking is not currently automatic for the WITH-SMTP-MAIL macro, and that ought to change. I will propably split the cl-smtp.lisp file in the future into two files: smtp.lisp will contain only SMTP protocol handling functionality and compose-message.lisp will contain the message composition functionality. Further thoughts? Let me know. -Hans From hans at huebner.org Mon Mar 24 07:59:26 2008 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 24 Mar 2008 08:59:26 +0100 Subject: [cl-smtp-devel] Patch to support sending raw messages In-Reply-To: References: <1668532236.20080324005354@yandex.ru> Message-ID: Forgot the patch URL: http://bknr.net/trac/changeset/2790 2008/3/24, Hans H?bner : > Hi Anton and whoever is interested, > > I made yet another patch to CL-SMTP so that it performs automatic > conversion of non-ASCII characters in email headers to quoted > characters according to RFC2047. This is implemented as a specialized > stream class SMTP-OUTPUT-STREAM that is used during message output. > The SMTP-OUTPUT-STREAM class also does conversion of #\Newline to > #\Return #\Newline - This might be better done with a properly set up > FLEXI-STREAMS encoding; I'm not sure. > > One thing that might usefully be added to SMTP-OUTPUT-STREAM is dot > masking. Dot masking is not currently automatic for the > WITH-SMTP-MAIL macro, and that ought to change. > > I will propably split the cl-smtp.lisp file in the future into two > files: smtp.lisp will contain only SMTP protocol handling > functionality and compose-message.lisp will contain the message > composition functionality. > > Further thoughts? Let me know. > > > -Hans > From hans at huebner.org Mon Mar 24 08:12:48 2008 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 24 Mar 2008 09:12:48 +0100 Subject: [cl-smtp-devel] Patch to support sending raw messages In-Reply-To: References: <1668532236.20080324005354@yandex.ru> Message-ID: Sorry - My last patch failed to build on SBCL because of a package lock violation. I have fixed that in our Subversion repository. For the moment, it is propably best to test with the SVN version (svn://svn.bknr.net/svn/trunk/thirdparty/cl-smtp) until Jan has decided if the stuff is merged into his CVS. For browsing, use this URL: http://bknr.net/trac/browser/trunk/thirdparty/cl-smtp -Hans 2008/3/24, Hans H?bner : > Forgot the patch URL: http://bknr.net/trac/changeset/2790 > > 2008/3/24, Hans H?bner : > > > Hi Anton and whoever is interested, > > > > I made yet another patch to CL-SMTP so that it performs automatic > > conversion of non-ASCII characters in email headers to quoted > > characters according to RFC2047. This is implemented as a specialized > > stream class SMTP-OUTPUT-STREAM that is used during message output. > > The SMTP-OUTPUT-STREAM class also does conversion of #\Newline to > > #\Return #\Newline - This might be better done with a properly set up > > FLEXI-STREAMS encoding; I'm not sure. > > > > One thing that might usefully be added to SMTP-OUTPUT-STREAM is dot > > masking. Dot masking is not currently automatic for the > > WITH-SMTP-MAIL macro, and that ought to change. > > > > I will propably split the cl-smtp.lisp file in the future into two > > files: smtp.lisp will contain only SMTP protocol handling > > functionality and compose-message.lisp will contain the message > > composition functionality. > > > > Further thoughts? Let me know. > > > > > > -Hans > > > From jan.idzikowski at knowledgetools.de Mon Mar 24 19:52:49 2008 From: jan.idzikowski at knowledgetools.de (Jan Idzikowski) Date: Mon, 24 Mar 2008 20:52:49 +0100 Subject: [cl-smtp-devel] Patch to support sending raw messages In-Reply-To: References: <1668532236.20080324005354@yandex.ru> Message-ID: <87ve3bew6m.wl@suse.knowledgetools.de> Hello Hans, i'm currently in vacation (a small village near the baltic see) with sporadic internet access. Your patches looks good for me, nice work, i like the idea, to split cl-smtp.lisp in a protocol handling functionality part and a message composition functionality part. The quoting of non-ASCII characters in email headers is (was) pretty high on my priority list. I'll be back in Berlin on 30.03. and i'll take care to merge your patches in the CL-SMTP repository and i'll have a look at the #\Return #\Newline problem. Best regards, Jan Idzikowski From hans at huebner.org Mon Mar 24 21:04:32 2008 From: hans at huebner.org (=?ISO-8859-1?Q?Hans_H=FCbner?=) Date: Mon, 24 Mar 2008 22:04:32 +0100 Subject: [cl-smtp-devel] Patch to support sending raw messages In-Reply-To: <87ve3bew6m.wl@suse.knowledgetools.de> References: <1668532236.20080324005354@yandex.ru> <87ve3bew6m.wl@suse.knowledgetools.de> Message-ID: Hi Jan, enjoy your remaining vacation! Let me know when you are back and I'll send you an updated diff. Cheers, Hans