From wws at clozure.com Wed May 5 02:41:17 2010 From: wws at clozure.com (Bill St. Clair) Date: Tue, 04 May 2010 22:41:17 -0400 Subject: [drakma-devel] https through a proxy Message-ID: <4BE0DACD.3060102@clozure.com> I have an application where I need to do https requests through a proxy server. It hangs in Drakma 1.1.0. I found the fix, for Lispworks only, at http://www.mail-archive.com/drakma-devel at common-lisp.net/msg00200.html . I guess Edi didn't integrate it yet. I made it work in CCL 1.5, meaning it will probably work everywhere, but I did't test Allegro. My fix is a little ugly, as I'm setting a slot with no writer inside of the chunga stream, but it works for me. Patch below. -Bill St. Clair wws at clozure.com ==== diff -rN old-drakma/request.lisp new-drakma/request.lisp 429c429,430 < (let (http-stream must-close done) --- > (let ((proxying-https? (and proxy (not stream) (eq :https (puri:uri-scheme uri)))) > http-stream raw-http-stream must-close done) 437,438c438,440 < (use-ssl (or force-ssl < (eq (uri-scheme uri) :https)))) --- > (use-ssl (and (not proxying-https?) > (or force-ssl > (eq (uri-scheme uri) :https))))) 470a473 > (setq raw-http-stream http-stream) 498a502,528 > (when proxying-https? > ;; Setup a tunnel through the proxy server to the > ;; final destination. > (write-http-line "CONNECT ~A:~A HTTP/1.1"(puri:uri-host uri) > (or (puri:uri-port uri) 443)) > (write-http-line "Host: ~A:~A" (puri:uri-host uri) > (or (puri:uri-port uri) 443)) > (write-http-line "") > (force-output http-stream) > ;; Check we get a 200 response before proceeding. > (let ((line (read-status-line http-stream *header-stream*))) > (unless (eq (second line) 200) > (error "Unable to establish HTTPS tunnel through proxy."))) > ;; Got a connection. We have to read a blank line, > ;; turn on SSL, and then we can transmit. > (read-line* http-stream) > #+:lispworks > (comm:attach-ssl raw-http-stream :ssl-side :client) > #-lispworks > (setf (slot-value (flexi-stream-stream http-stream) 'chunga::real-stream) ;evil > #+:allegro > (socket:make-ssl-client-stream raw-http-stream) > #-:allegro > (let ((s raw-http-stream)) > (cl+ssl:make-ssl-client-stream > (cl+ssl:stream-fd s) > :close-callback (lambda () (close s)))))) 513c543 < (cond (proxy (render-uri uri nil)) --- > (cond ((and proxy (not proxying-https?)) (render-uri uri nil)) From edi at agharta.de Fri May 7 19:43:17 2010 From: edi at agharta.de (Edi Weitz) Date: Fri, 7 May 2010 21:43:17 +0200 Subject: [drakma-devel] https through a proxy In-Reply-To: <4BE0DACD.3060102@clozure.com> References: <4BE0DACD.3060102@clozure.com> Message-ID: Hi Bill, Thanks for the patch. I can't really read it because your email client broke it, but I think I get the idea. Yes, I didn't integrate the old patch and IIRC I didn't do it because it was a LW-only solution. Obviously, I don't like the new solution either because it uses unexported symbols (and you can't change the underlying stream for a reason). As far as I understand, all this happens while the headers are still sent and while chunking isn't really needed. I haven't really thought about the details, but maybe it would be possible to change the order of how and when the streams are wrapped and use only exported functionality. In the worst case, one could "unwrap" the stream, attach SSL, and then "rewrap" it. Of course, the easiest solution would be if CL+SSL and AllegroCL had something similar to LispWorks' comm:attach-ssl. Anyway, if someone wants to work on a "clean" solution to this problem, I'll happily review it. As usual: http://weitz.de/patches.html Thanks again, Edi. On Wed, May 5, 2010 at 4:41 AM, Bill St. Clair wrote: > I have an application where I need to do https requests through a proxy > server. It hangs in Drakma 1.1.0. I found the fix, for Lispworks only, > at > http://www.mail-archive.com/drakma-devel at common-lisp.net/msg00200.html . > I guess Edi didn't integrate it yet. I made it work in CCL 1.5, meaning > it will probably work everywhere, but I did't test Allegro. My fix is a > little ugly, as I'm setting a slot with no writer inside of the chunga > stream, but it works for me. Patch below. > > -Bill St. Clair > wws at clozure.com > > ==== > > diff -rN old-drakma/request.lisp new-drakma/request.lisp > 429c429,430 > < ? ? (let (http-stream must-close done) > --- >> ? ? (let ((proxying-https? (and proxy (not stream) (eq :https > (puri:uri-scheme uri)))) >> ? ? ? ? ? ?http-stream raw-http-stream must-close done) > 437,438c438,440 > < ? ? ? ? ? ? ? ? ? (use-ssl (or force-ssl > < ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(eq (uri-scheme uri) :https)))) > --- >> ? ? ? ? ? ? ? ? ? ?(use-ssl (and (not proxying-https?) >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(or force-ssl >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(eq (uri-scheme uri) :https))))) > 470a473 >> ? ? ? ? ? ? ? (setq raw-http-stream http-stream) > 498a502,528 >> ? ? ? ? ? ? ? (when proxying-https? >> ? ? ? ? ? ? ? ? ;; Setup a tunnel through the proxy server to the >> ? ? ? ? ? ? ? ? ;; final destination. >> ? ? ? ? ? ? ? ? (write-http-line "CONNECT ~A:~A > HTTP/1.1"(puri:uri-host uri) >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(or (puri:uri-port uri) 443)) >> ? ? ? ? ? ? ? ? (write-http-line "Host: ~A:~A" (puri:uri-host uri) >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(or (puri:uri-port uri) 443)) >> ? ? ? ? ? ? ? ? (write-http-line "") >> ? ? ? ? ? ? ? ? (force-output http-stream) >> ? ? ? ? ? ? ? ? ;; Check we get a 200 response before proceeding. >> ? ? ? ? ? ? ? ? (let ((line (read-status-line http-stream > *header-stream*))) >> ? ? ? ? ? ? ? ? ? (unless (eq (second line) 200) >> ? ? ? ? ? ? ? ? ? ? (error "Unable to establish HTTPS tunnel through > proxy."))) >> ? ? ? ? ? ? ? ? ;; Got a connection. We have to read a blank line, >> ? ? ? ? ? ? ? ? ;; turn on SSL, and then we can transmit. >> ? ? ? ? ? ? ? ? (read-line* http-stream) >> ? ? ? ? ? ? ? ? #+:lispworks >> ? ? ? ? ? ? ? ? (comm:attach-ssl raw-http-stream :ssl-side :client) >> ? ? ? ? ? ? ? ? #-lispworks >> ? ? ? ? ? ? ? ? (setf (slot-value (flexi-stream-stream http-stream) > 'chunga::real-stream) ;evil >> ? ? ? ? ? ? ? ? ? ? ? #+:allegro >> ? ? ? ? ? ? ? ? ? ? ? (socket:make-ssl-client-stream raw-http-stream) >> ? ? ? ? ? ? ? ? ? ? ? #-:allegro >> ? ? ? ? ? ? ? ? ? ? ? (let ((s raw-http-stream)) >> ? ? ? ? ? ? ? ? ? ? ? ? (cl+ssl:make-ssl-client-stream >> ? ? ? ? ? ? ? ? ? ? ? ? ?(cl+ssl:stream-fd s) >> ? ? ? ? ? ? ? ? ? ? ? ? ?:close-callback (lambda () (close s)))))) > 513c543 > < ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(cond (proxy (render-uri uri nil)) > --- >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(cond ((and proxy (not > proxying-https?)) (render-uri uri nil)) > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > From wws at clozure.com Sun May 9 11:36:46 2010 From: wws at clozure.com (Bill St. Clair) Date: Sun, 09 May 2010 07:36:46 -0400 Subject: [drakma-devel] https through a proxy (Edi Weitz) Message-ID: <4BE69E4E.309@clozure.com> On 5/8/10 12:00PM, drakma-devel-request at common-lisp.net wrote: > Date: Fri, 7 May 2010 21:43:17 +0200 > From: Edi Weitz > Subject: Re: [drakma-devel] https through a proxy > To: General interest list for Drakma and Chunga > > Thanks for the patch. I can't really read it because your email > client broke it, but I think I get the idea. Yes, I didn't integrate > the old patch and IIRC I didn't do it because it was a LW-only > solution. Obviously, I don't like the new solution either because it > uses unexported symbols (and you can't change the underlying stream > for a reason). > > As far as I understand, all this happens while the headers are still > sent and while chunking isn't really needed. I haven't really thought > about the details, but maybe it would be possible to change the order > of how and when the streams are wrapped and use only exported > functionality. In the worst case, one could "unwrap" the stream, > attach SSL, and then "rewrap" it. Of course, the easiest solution > would be if CL+SSL and AllegroCL had something similar to LispWorks' > comm:attach-ssl. > > Anyway, if someone wants to work on a "clean" solution to this > problem, I'll happily review it. I cleaned up my patch. It now contains no double-colons. It ends up wrapping with the flexi-stream and chunking-stream twice in the https through a proxy case, as not wrapping didn't work (it may work to wrap the proxy communication with only a flexi-stream, but I didn't try that). I also fixed a bug that caused it to include on the first (GET) line to the server the entire URI instead of only the path part of it when a request through a proxy with keep-alive gets redirected. Patch attached. -Bill -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 100508.diff URL: From reddaly at gmail.com Mon May 17 10:27:58 2010 From: reddaly at gmail.com (Red Daly) Date: Mon, 17 May 2010 03:27:58 -0700 Subject: [drakma-devel] Cookie-parsing patch for chunga Message-ID: Cookies often have quoted-strings, as described in http://www.w3.org/Protocols/rfc2109/rfc2109 . This patch adds quoted string support to the auxiliary cookie parsing functionality provided by Chunga. *crossing my fingers that I formatted the patch properly* - Red -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: quoted-strings-in-cookies.patch Type: text/x-patch Size: 1213 bytes Desc: not available URL: From edi at agharta.de Wed May 19 15:16:53 2010 From: edi at agharta.de (Edi Weitz) Date: Wed, 19 May 2010 17:16:53 +0200 Subject: [drakma-devel] New version 1.2.0 Message-ID: Available from http://weitz.de/ as usual. ChangeLog below, thanks to anyone who sent patches or bug reports. Edi. Version 1.2.0 2010-05-19 Introduced *REMOVE-DUPLICATE-COOKIES-P* (Ryan Davis) Enabled https through a proxy (Bill St. Clair and Dave Lambert) Bugfix for redirect of a request through a proxy (Bill St. Clair) Export PARSE-COOKIE-DATE Safer method to render URIs Allowed for GET/POST parameters without a value (seen on Lotus webservers) From edi at agharta.de Wed May 19 15:25:03 2010 From: edi at agharta.de (Edi Weitz) Date: Wed, 19 May 2010 17:25:03 +0200 Subject: [drakma-devel] New Chunga version 1.1.1 Message-ID: And, again on weitz.de, there's also a new release of Chunga. Version 1.1.1 2010-05-19 Read quoted cookie values (Red Daly) From xach at xach.com Wed May 19 18:36:07 2010 From: xach at xach.com (Zach Beane) Date: Wed, 19 May 2010 14:36:07 -0400 Subject: [drakma-devel] New version 1.2.0 In-Reply-To: (Edi Weitz's message of "Wed, 19 May 2010 17:16:53 +0200") References: Message-ID: <87r5l7pwmg.fsf@hangup.portland.xach.com> Edi Weitz writes: > Available from http://weitz.de/ as usual. ChangeLog below, thanks to > anyone who sent patches or bug reports. This version does not compile for me; make-ssl-stream in util.lisp has an extra paren at the end of line 334. Zach From edi at agharta.de Wed May 19 19:41:45 2010 From: edi at agharta.de (Edi Weitz) Date: Wed, 19 May 2010 21:41:45 +0200 Subject: [drakma-devel] New version 1.2.1 Message-ID: Yeah, sorry, I should have been more careful. At least this was reported by three persons in a couple of minutes which gives me the warm and fuzzy feeling that there are enough users to catch mistakes... :) Thanks. Bugfix release online. Edi. On Wed, May 19, 2010 at 8:36 PM, Zach Beane wrote: > Edi Weitz writes: > >> Available from http://weitz.de/ as usual. ?ChangeLog below, thanks to >> anyone who sent patches or bug reports. > > This version does not compile for me; make-ssl-stream in util.lisp has > an extra paren at the end of line 334. > > Zach > >