From edi at agharta.de Wed May 7 12:41:01 2008 From: edi at agharta.de (Edi Weitz) Date: Wed, 07 May 2008 08:41:01 -0400 Subject: [drakma-devel] New Chunga release 0.4.2 Message-ID: ChangeLog: Version 0.4.2 2008-05-07 Flush stream when switching chunking off (patch by Hans H?bner) Download: http://weitz.de/files/chunga.tar.gz From lispercat at gmail.com Fri May 9 18:48:10 2008 From: lispercat at gmail.com (Andrei Stebakov) Date: Fri, 9 May 2008 14:48:10 -0400 Subject: [drakma-devel] http-request returns nil Message-ID: I am using http-request to upload files on some web-site. The problem is that sometimes the function returns a nil response so I just have to repeat it again with the same parameters. Sometimes it take up to 8 retries to make the request return a non-nil response. I wonder if you saw it before or how can I debug it? The function doesn't take long to return with nil result. I mean the whole file upload can take 10 seconds, so the http-request may return nil within those 10 seconds. Is some timeout or the connection gets broken between the hosts? There is no exception raised as well, it just silently returns nil. Thank you, Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: From edi at agharta.de Fri May 9 19:21:21 2008 From: edi at agharta.de (Edi Weitz) Date: Fri, 09 May 2008 15:21:21 -0400 Subject: [drakma-devel] http-request returns nil In-Reply-To: (Andrei Stebakov's message of "Fri, 9 May 2008 14:48:10 -0400") References: Message-ID: On Fri, 9 May 2008 14:48:10 -0400, "Andrei Stebakov" wrote: > I am using http-request to upload files on some web-site. The > problem is that sometimes the function returns a nil response so I > just have to repeat it again with the same parameters. Sometimes it > take up to 8 retries to make the request return a non-nil response. > I wonder if you saw it before or how can I debug it? The function > doesn't take long to return with nil result. I mean the whole file > upload can take 10 seconds, so the http-request may return nil > within those 10 seconds. Is some timeout or the connection gets > broken between the hosts? There is no exception raised as well, it > just silently returns nil. Hmm, it should definitely not just silently return NIL, but we probably need a tiny little bit more info from you: Which Lisp? Which operating system? Which version of Drakma and Chunga? Have you tried to trace the headers when this happens? http://weitz.de/drakma/#*header-stream* Is this a public URL so we can try to reproduce your problem? Edi. From lispercat at gmail.com Fri May 9 19:37:47 2008 From: lispercat at gmail.com (Andrei Stebakov) Date: Fri, 9 May 2008 15:37:47 -0400 Subject: [drakma-devel] http-request returns nil In-Reply-To: References: Message-ID: Unfortunately it's not a public url, and it uses some authorization tokens in the request, so I can't disclose it. I am using drakma-0.11.5 and chunga-0.4.2 on Ubuntu with sbcl-1.0.15. I am going to use *header-stream* to see if it gives some clue. Andrew On Fri, May 9, 2008 at 3:21 PM, Edi Weitz wrote: > On Fri, 9 May 2008 14:48:10 -0400, "Andrei Stebakov" > wrote: > > > I am using http-request to upload files on some web-site. The > > problem is that sometimes the function returns a nil response so I > > just have to repeat it again with the same parameters. Sometimes it > > take up to 8 retries to make the request return a non-nil response. > > I wonder if you saw it before or how can I debug it? The function > > doesn't take long to return with nil result. I mean the whole file > > upload can take 10 seconds, so the http-request may return nil > > within those 10 seconds. Is some timeout or the connection gets > > broken between the hosts? There is no exception raised as well, it > > just silently returns nil. > > Hmm, it should definitely not just silently return NIL, but we > probably need a tiny little bit more info from you: > > Which Lisp? Which operating system? Which version of Drakma and > Chunga? Have you tried to trace the headers when this happens? > > http://weitz.de/drakma/#*header-stream* > > Is this a public URL so we can try to reproduce your problem? > > Edi. > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edi at agharta.de Fri May 9 19:44:15 2008 From: edi at agharta.de (Edi Weitz) Date: Fri, 09 May 2008 15:44:15 -0400 Subject: [drakma-devel] http-request returns nil In-Reply-To: (Andrei Stebakov's message of "Fri, 9 May 2008 15:37:47 -0400") References: Message-ID: On Fri, 9 May 2008 15:37:47 -0400, "Andrei Stebakov" wrote: > I am using drakma-0.11.5 and chunga-0.4.2 on Ubuntu with > sbcl-1.0.15. I am going to use *header-stream* to see if it gives > some clue. If that doesn't help, try to use something like Wireshark. Or, alternatively, try with another Lisp and/or from another OS. Sometimes that makes a difference. (Which would be a bug in Drakma because it shouldn't.) From rsynnott at gmail.com Sat May 10 00:23:11 2008 From: rsynnott at gmail.com (Robert Synnott) Date: Sat, 10 May 2008 01:23:11 +0100 Subject: [drakma-devel] http-request returns nil In-Reply-To: References: Message-ID: <24f203480805091723x625c9739rd0c22b950d018665@mail.gmail.com> Do other clients have similar issues; are you sure it's not a server issue? Rob From lispercat at gmail.com Sat May 10 00:45:41 2008 From: lispercat at gmail.com (Andrei Stebakov) Date: Fri, 9 May 2008 20:45:41 -0400 Subject: [drakma-devel] http-request returns nil In-Reply-To: <24f203480805091723x625c9739rd0c22b950d018665@mail.gmail.com> References: <24f203480805091723x625c9739rd0c22b950d018665@mail.gmail.com> Message-ID: I can't say for other clients. Even if it's a server issue, there should be some reason why the function returns nil. If server just breaks connection, there should be an exception, right? If it's a timeout, again, should be some way to know it. In this case it's probably not timeout as the function doesn't take long to return. Probably some exception is just masked somewhere in the guts of the function. Andrew On Fri, May 9, 2008 at 8:23 PM, Robert Synnott wrote: > Do other clients have similar issues; are you sure it's not a server issue? > Rob > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.j.lambert at gmail.com Mon May 19 14:57:21 2008 From: d.j.lambert at gmail.com (Dave Lambert) Date: Mon, 19 May 2008 15:57:21 +0100 Subject: [drakma-devel] HTTPS through proxy Message-ID: Hi, I need to contact HTTPS servers through a proxy. I've managed to get this to work by having the http-request function issue an HTTP CONNECT commmand in order to tunnel SSL through the proxy: the patch is below (against 0.11.5). On the downside, it only works for Lispworks. Cheers, Dave --- a/request.lisp +++ b/request.lisp @@ -416,7 +416,8 @@ LispWorks 5.0 or higher." (t (setq content (alist-to-url-encoded-string parameters external-format-out) content-type "application/x-www-form-urlencoded"))))) - (let (http-stream must-close done) + (let ((proxying-https? (and proxy (eq :https (puri:uri-scheme uri)))) + http-stream raw-http-stream must-close done) (unwind-protect (progn (let ((host (or (and proxy (first proxy)) @@ -424,8 +425,8 @@ LispWorks 5.0 or higher." (port (cond (proxy (second proxy)) ((uri-port uri)) (t (default-port uri)))) - (use-ssl (or force-ssl - (eq (uri-scheme uri) :https)))) + (use-ssl (and (not proxying-https?) + (or force-ssl (eq (uri-scheme uri) :https))))) #+(and :lispworks5.0 :mswindows (not :lw-does-not-have-write-timeout)) (when use-ssl @@ -445,6 +446,7 @@ LispWorks 5.0 or higher." #-:lispworks (usocket:socket-stream (usocket:socket-connect host port :element-type 'octet)))) + (setq raw-http-stream http-stream) (when (and use-ssl ;; don't attach SSL to existing streams (not stream)) @@ -473,6 +475,24 @@ LispWorks 5.0 or higher." (format http-stream "~?~C~C" fmt args #\Return #\Linefeed)) (write-header (name value-fmt &rest value-args) (write-http-line "~A: ~?" name value-fmt value-args))) + (when proxying-https? + ;; Setup a tunnel through the proxy server to the + ;; final destination. + (write-http-line "CONNECT ~A:~A HTTP/1.1" (puri:uri-host uri) + (or (puri:uri-port uri) 443)) + (write-http-line "Host: ~A:~A" (puri:uri-host uri) + (or (puri:uri-port uri) 443)) + (write-http-line "") + (force-output http-stream) + ;; Check we get a 200 response before proceeding. + (let ((line (read-status-line http-stream *header-stream*))) + (unless (eq (second line) 200) + (error "Unable to establish HTTPS tunnel through proxy."))) + ;; Got a connection. We have to read a blank line, + ;; turn on SSL, and then we can transmit. + (read-line* http-stream) + #+:lispworks + (comm:attach-ssl raw-http-stream :ssl-side :client)) (when (and (not parameters-used-p) parameters) (setf (uri-query uri) From edi at agharta.de Mon May 19 15:54:38 2008 From: edi at agharta.de (Edi Weitz) Date: Mon, 19 May 2008 17:54:38 +0200 Subject: [drakma-devel] HTTPS through proxy In-Reply-To: (Dave Lambert's message of "Mon, 19 May 2008 15:57:21 +0100") References: Message-ID: On Mon, 19 May 2008 15:57:21 +0100, "Dave Lambert" wrote: > I need to contact HTTPS servers through a proxy. I've managed to > get this to work by having the http-request function issue an HTTP > CONNECT commmand in order to tunnel SSL through the proxy: the patch > is below (against 0.11.5). On the downside, it only works for > Lispworks. Thanks. Is this based on some standard or does it cater to one specific server? Drakma's (limited) proxy functionality didn't work for you? > --- a/request.lisp > +++ b/request.lisp Whatever email app you used ruined the patch. Can you send it as an attachment or with another program? Cheers, Edi. From d.j.lambert at open.ac.uk Mon May 19 16:14:25 2008 From: d.j.lambert at open.ac.uk (Dave Lambert) Date: Mon, 19 May 2008 17:14:25 +0100 Subject: [drakma-devel] HTTPS through proxy In-Reply-To: References: Message-ID: 2008/5/19 Edi Weitz : > On Mon, 19 May 2008 15:57:21 +0100, "Dave Lambert" wrote: > >> I need to contact HTTPS servers through a proxy. I've managed to >> get this to work by having the http-request function issue an HTTP >> CONNECT commmand in order to tunnel SSL through the proxy: the patch >> is below (against 0.11.5). On the downside, it only works for >> Lispworks. > Drakma's (limited) proxy functionality didn't work for you? Drakma's HTTP proxy support is just great, but I also need to create an HTTPS connection from behind a firewall. This patch adds just that HTTPS functionality. There was a previous request for this on the mailing list: http://common-lisp.net/pipermail/drakma-devel/2007-November/000212.html > Is this based on some standard or does it cater to one specific > server? The approach is standard: Use the HTTP CONNECT method to get a tunnel through the proxy, and then send SSL on the same connection. It's covered in HTTP 1.1: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.9 I've tested it on my local proxy, and a couple of HTTPS end points. > Whatever email app you used ruined the patch. Can you send it as an > attachment or with another program? That's a shame (it's Gmail). Patch attached this time. Cheers, Dave -------------- next part -------------- A non-text attachment was scrubbed... Name: https-proxy.diff Type: text/x-patch Size: 3084 bytes Desc: not available URL: From edi at agharta.de Tue May 20 06:33:43 2008 From: edi at agharta.de (Edi Weitz) Date: Tue, 20 May 2008 08:33:43 +0200 Subject: [drakma-devel] HTTPS through proxy In-Reply-To: (Dave Lambert's message of "Mon, 19 May 2008 17:14:25 +0100") References: Message-ID: On Mon, 19 May 2008 17:14:25 +0100, "Dave Lambert" wrote: > There was a previous request for this on the mailing list: > > http://common-lisp.net/pipermail/drakma-devel/2007-November/000212.html Yup. Seems I forgot... :) > The approach is standard: Use the HTTP CONNECT method to get a > tunnel through the proxy, and then send SSL on the same connection. > It's covered in HTTP 1.1: > http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.9 > > I've tested it on my local proxy, and a couple of HTTPS end points. Thanks. I'll look into this and will likely integrate something like this into the next release. Cheers, Edi. From edi at agharta.de Fri May 23 01:55:28 2008 From: edi at agharta.de (Edi Weitz) Date: Fri, 23 May 2008 03:55:28 +0200 Subject: [drakma-devel] New Chunga release 0.4.3 Message-ID: ChangeLog: Version 0.4.3 2008-05-23 Cleanup, reduce some consing Download: http://weitz.de/files/chunga.tar.gz From edi at agharta.de Mon May 26 14:39:07 2008 From: edi at agharta.de (Edi Weitz) Date: Mon, 26 May 2008 16:39:07 +0200 Subject: [drakma-devel] Current development Message-ID: As common-lisp.net has been pretty unreliable in the last days, we've moved development over to the bknr.net server maintained by Hans. If you're interested in trying out the soon-to-be-released Hunchentoot version, get all of Hunchentoot, Chunga, and FLEXI-STREAMS from here: http://bknr.net/trac/browser/trunk/thirdparty If you're using Drakma, you'll have to grab it from bknr.net as well, as there are incompatible changes in Chunga. The previous development history can be seen at http://trac.common-lisp.net/tbnl/browser/branches whenever the server is up. Edi. From lispercat at gmail.com Thu May 29 21:33:05 2008 From: lispercat at gmail.com (Andrei Stebakov) Date: Thu, 29 May 2008 17:33:05 -0400 Subject: [drakma-devel] http-request returns nil In-Reply-To: References: Message-ID: The problem was that sometimes the site responded with just headers so that content-length was 0. In this case the http-request returned request equal to nil and that confused me (I was expecting a zero length string). Now I look at the (header-value :content-length headers) and if it's zero, I get all the information I need from (header-value :location headers). I wonder if response should be nil when content-length is 0? On Fri, May 9, 2008 at 3:44 PM, Edi Weitz wrote: > On Fri, 9 May 2008 15:37:47 -0400, "Andrei Stebakov" > wrote: > > > I am using drakma-0.11.5 and chunga-0.4.2 on Ubuntu with > > sbcl-1.0.15. I am going to use *header-stream* to see if it gives > > some clue. > > If that doesn't help, try to use something like Wireshark. Or, > alternatively, try with another Lisp and/or from another OS. > Sometimes that makes a difference. (Which would be a bug in Drakma > because it shouldn't.) > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edi at agharta.de Thu May 29 22:09:32 2008 From: edi at agharta.de (Edi Weitz) Date: Fri, 30 May 2008 00:09:32 +0200 Subject: [drakma-devel] http-request returns nil In-Reply-To: (Andrei Stebakov's message of "Thu, 29 May 2008 17:33:05 -0400") References: Message-ID: On Thu, 29 May 2008 17:33:05 -0400, "Andrei Stebakov" wrote: > I wonder if response should be nil when content-length is 0? The first return value of HTTP-REQUEST isn't necessarily a string, it can also be an array of octets or a stream, so there's no particular reason to assume that you'll get a string of length zero if the content length is zero. But you can, no matter if it's a string, an octet array, or NIL always check with LENGTH first. One could argue that a zero-length array would be more appropriate, but I think returning NIL in this case is more Lisp-y. Having said that, I realize that NIL isn't even mentioned as a potential return value, so I should probably add this to the documentation.