From edi at agharta.de Mon Sep 4 22:38:22 2006 From: edi at agharta.de (Edi Weitz) Date: Tue, 05 Sep 2006 00:38:22 +0200 Subject: [drakma-devel] New versions of Chunga (0.1.2) and Drakma (0.4.0) Message-ID: Chunga: ------- ChangeLog: Version 0.1.2 2006-09-05 Exported CHUNKED-STREAM-STREAM Mentioned Gentoo port in docs Added info about mailing lists Download: http://weitz.de/files/chunga.tar.gz Drakma: ChangeLog: Version 0.4.0 2006-09-05 Added file uploads Added multipart/form-data Added enforced computation of request bodies in RAM Use LF line endings in default external format Download: http://weitz.de/files/drakma.tar.gz Cheers, Edi. From rm at seid-online.de Wed Sep 6 14:41:35 2006 From: rm at seid-online.de (Ralf Mattes) Date: Wed, 06 Sep 2006 16:41:35 +0200 Subject: [drakma-devel] Supported HTTP methods Message-ID: <1157553695.12284.20.camel@localhost.localdomain> Hello list, first, to Edi: thanks for such a nice library! I just stumbled across an error in the http-request function: DRAKMA-USER <- (http-request "http://www.zeit.de" :method :trace) Don't know how to handle method :TRACE. [Condition of type SIMPLE-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) triggered by this code in request.lisp: (unless (member method '(:get :head :post :put) :test #'eq) (error "Don't know how to handle method ~S." method)) shouldn't the library allow all methods from HTTP/1.1? Changing the code into: (unless (member method '(:get :head :post :put :delete :options :trace) :test #'eq) (error "Don't know how to handle method ~S." method)) fixes this bugglet. BTW, it might be a good idea to add the methods needed for WebDAV as well -- at least there seems to be no reason for not adding them. Some of them would need to be added to case dispatch on method that sets the request contents (request.lisp, line 344 ff.). One more remark about the 'OPTIONS' method. The standard introduces a 'magic' request URI to query server options (as opposed to options for a particular resource) - the request line should look like this: OPTIONS * HTTP/1.1 Host: my.cool.server I couldn't find a way to have puri:parse-uri generate a uri whose uri-path is "*" so i modified http-request to accept an method :options* (see attached patch). Cheers, Ralf Mattes -------------- next part -------------- A non-text attachment was scrubbed... Name: request.patch Type: text/x-patch Size: 1194 bytes Desc: not available URL: From edi at agharta.de Wed Sep 6 19:17:59 2006 From: edi at agharta.de (Edi Weitz) Date: Wed, 06 Sep 2006 21:17:59 +0200 Subject: [drakma-devel] Supported HTTP methods In-Reply-To: <1157553695.12284.20.camel@localhost.localdomain> (Ralf Mattes's message of "Wed, 06 Sep 2006 16:41:35 +0200") References: <1157553695.12284.20.camel@localhost.localdomain> Message-ID: Hi Ralf! On Wed, 06 Sep 2006 16:41:35 +0200, Ralf Mattes wrote: > first, to Edi: thanks for such a nice library! You're welcome... :) > I just stumbled across an error in the http-request function: > > DRAKMA-USER <- (http-request "http://www.zeit.de" :method :trace) > > Don't know how to handle method :TRACE. > [Condition of type SIMPLE-ERROR] > > Restarts: > 0: [ABORT-REQUEST] Abort handling SLIME request. > 1: [TERMINATE-THREAD] Terminate this thread (# "repl-thread" {AC7D291}>) > > triggered by this code in request.lisp: > > (unless (member method '(:get :head :post :put) :test #'eq) > (error "Don't know how to handle method ~S." method)) > > shouldn't the library allow all methods from HTTP/1.1? It's OK with me, I just didn't need them until now. > Changing the code into: > > (unless (member method > '(:get :head :post :put :delete :options :trace) :test #'eq) > (error "Don't know how to handle method ~S." method)) > > fixes this bugglet. BTW, it might be a good idea to add the methods > needed for WebDAV as well -- at least there seems to be no reason > for not adding them. Some of them would need to be added to case > dispatch on method that sets the request contents (request.lisp, > line 344 ff.). > > One more remark about the 'OPTIONS' method. The standard introduces > a 'magic' request URI to query server options (as opposed to options > for a particular resource) - the request line should look like this: > > OPTIONS * HTTP/1.1 > Host: my.cool.server > > I couldn't find a way to have puri:parse-uri generate a uri whose > uri-path is "*" so i modified http-request to accept an method > :options* (see attached patch). The patch is OK as far as I can see, but as with 99% of the patches I get, the patch for the doc strings and the documentation is missing. Adding half a dozen lines of code is the easy part... :) I'll add that if I find the time, but probably not today. Thanks, Edi. From edi at agharta.de Thu Sep 7 10:26:30 2006 From: edi at agharta.de (Edi Weitz) Date: Thu, 07 Sep 2006 12:26:30 +0200 Subject: [drakma-devel] New version 0.4.1 (Was: Supported HTTP methods) In-Reply-To: <1157553695.12284.20.camel@localhost.localdomain> (Ralf Mattes's message of "Wed, 06 Sep 2006 16:41:35 +0200") References: <1157553695.12284.20.camel@localhost.localdomain> Message-ID: On Wed, 06 Sep 2006 16:41:35 +0200, Ralf Mattes wrote: > shouldn't the library allow all methods from HTTP/1.1? [...] BTW, > it might be a good idea to add the methods needed for WebDAV as well OK, done. There's a new release 0.4.1 online. Please test. Cheers, Edi. From edi at agharta.de Thu Sep 7 10:52:24 2006 From: edi at agharta.de (Edi Weitz) Date: Thu, 07 Sep 2006 12:52:24 +0200 Subject: [drakma-devel] Version 0.4.2 (Was: Supported HTTP methods) In-Reply-To: <1157553695.12284.20.camel@localhost.localdomain> (Ralf Mattes's message of "Wed, 06 Sep 2006 16:41:35 +0200") References: <1157553695.12284.20.camel@localhost.localdomain> Message-ID: On Wed, 06 Sep 2006 16:41:35 +0200, Ralf Mattes wrote: > + ;; create a 'magic' options request > + (when (eq method :options*) > + (setf (uri-path uri) "*" > + (uri-query uri) "" > + method :options)) Woops, that was wrong. The query part must be set to NIL, not "". See release 0.4.2. Sorry, Edi. From rsynnott at gmail.com Sat Sep 23 19:04:15 2006 From: rsynnott at gmail.com (Robert Synnott) Date: Sat, 23 Sep 2006 20:04:15 +0100 Subject: [drakma-devel] Compile-time bug on OpenMCL Message-ID: <24f203480609231204k58a4d326na3f243a6e6ea651a@mail.gmail.com> Compiling on OpenMCL, (both drakma itself and chunga) an error is given: "No MAKE-LOAD-FORM method is defined for [external-format stuff]". Apparently, this is something to do with OpenMCL's handling of constants: http://www.clozure.com/pipermail/openmcl-devel/2002-December/000823.html . The constant in question is +latin-1+. I found replacing all references to the constant with its value to work, but I'm sure there's a neater way. Rob From edi at agharta.de Sat Sep 23 20:53:04 2006 From: edi at agharta.de (Edi Weitz) Date: Sat, 23 Sep 2006 22:53:04 +0200 Subject: [drakma-devel] Compile-time bug on OpenMCL In-Reply-To: <24f203480609231204k58a4d326na3f243a6e6ea651a@mail.gmail.com> (Robert Synnott's message of "Sat, 23 Sep 2006 20:04:15 +0100") References: <24f203480609231204k58a4d326na3f243a6e6ea651a@mail.gmail.com> Message-ID: Hi! On Sat, 23 Sep 2006 20:04:15 +0100, "Robert Synnott" wrote: > Compiling on OpenMCL, (both drakma itself and chunga) an error is > given: "No MAKE-LOAD-FORM method is defined for [external-format > stuff]". > > Apparently, this is something to do with OpenMCL's handling of > constants: > http://www.clozure.com/pipermail/openmcl-devel/2002-December/000823.html > The constant in question is +latin-1+. > > I found replacing all references to the constant with its value to > work, but I'm sure there's a neater way. IIUC, the best way to fix this is to add a method for MAKE-LOAD-FORM in FLEXI-STREAMS - which I just did. Please try the latest version (0.6.2), recompile Drakma, try it, and report to the list - I don't have a Mac to test myself. Thanks for the info, Edi. From rsynnott at gmail.com Sat Sep 23 21:31:05 2006 From: rsynnott at gmail.com (Robert Synnott) Date: Sat, 23 Sep 2006 22:31:05 +0100 Subject: [drakma-devel] Compile-time bug on OpenMCL In-Reply-To: References: <24f203480609231204k58a4d326na3f243a6e6ea651a@mail.gmail.com> Message-ID: <24f203480609231431o405dcb50i901382eb08199c57@mail.gmail.com> > > > Compiling on OpenMCL, (both drakma itself and chunga) an error is > > given: "No MAKE-LOAD-FORM method is defined for [external-format > > stuff]". > > > > Apparently, this is something to do with OpenMCL's handling of > > constants: > > http://www.clozure.com/pipermail/openmcl-devel/2002-December/000823.html > > The constant in question is +latin-1+. > > > > I found replacing all references to the constant with its value to > > work, but I'm sure there's a neater way. > > IIUC, the best way to fix this is to add a method for MAKE-LOAD-FORM > in FLEXI-STREAMS - which I just did. Please try the latest version > (0.6.2), recompile Drakma, try it, and report to the list - I don't > have a Mac to test myself. > > Thanks for the info, > Edi. > Yes, that works perfectly now. Thanks Rob From edi at agharta.de Sat Sep 23 23:38:02 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 24 Sep 2006 01:38:02 +0200 Subject: [drakma-devel] New version 0.4.3 Message-ID: ChangeLog: Version 0.4.3 2006-09-24 Circumvent CL+SSL for AllegroCL (suggested by David Lichteblau) Download: http://weitz.de/files/drakma.tar.gz Cheers, Edi. From ivan at toshkov.org Sun Sep 24 06:41:04 2006 From: ivan at toshkov.org (Ivan Toshkov) Date: Sun, 24 Sep 2006 09:41:04 +0300 Subject: [drakma-devel] Couple of problems Message-ID: <7c23adaa0609232341n3549e03ct8ab7b53773d71f13@mail.gmail.com> Hi, First of all, thank you Edi Weitz for your excellent libraries. I find them simple to use, well thought out, and with excellent documentation. Again, a big thank you. And thanks to whoever is working on porting Drakma. I found a couple of problems: Drakma cannot receive cookies from localhost. After a short investigation I found that the problem was with the method VALID-COOKIE-DOMAIN-P, which insist to have at least 2 dots in the normalized domain name. As a workaround I used 127.0.0.1 instead of localhost. The other problem is with an incomplete implementation of flexi-streams under CMUCL. The call: > (drakma:http-request "http://www.google.com/" :parameters '(("q" . "something"))) works normally, but > (drakma:http-request "http://www.google.com/search" :parameters '(("q" . "something"))) gives: No matching method for the generic function #, when called with arguments (#). [Condition of type PCL::NO-APPLICABLE-METHOD-ERROR] 0: ("DEFMETHOD NO-APPLICABLE-METHOD (T)" #<#1=unused-arg> #<#1#> # (#)) 1: (FLEXI-STREAMS:MAKE-FLEXI-STREAM # :EXTERNAL-FORMAT :LATIN-1) 2: (FLEXI-STREAMS:STRING-TO-OCTETS "q" :EXTERNAL-FORMAT :LATIN-1 :START ...) 3: (DRAKMA::URL-ENCODE "q" :LATIN-1) 4: (DRAKMA::ALIST-TO-URL-ENCODED-STRING (("q" . "something")) :LATIN-1) 5: (DRAKMA:HTTP-REQUEST # :PARAMETERS (("q" . "something"))) I don't have time at the moment to see if anything else is missing and to give patches. For this case though the workaround seems to be as simple as: (defmethod input-stream-p ((s flexi-streams::vector-output-stream)) nil) Cheers, Ivan P.S. I did a clean ASDF-install on Drakma and it's dependencies. I can give the precise versions if needed. From edi at agharta.de Sun Sep 24 20:50:47 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 24 Sep 2006 22:50:47 +0200 Subject: [drakma-devel] New version 0.4.4 (Was: Couple of problems) In-Reply-To: <7c23adaa0609232341n3549e03ct8ab7b53773d71f13@mail.gmail.com> (Ivan Toshkov's message of "Sun, 24 Sep 2006 09:41:04 +0300") References: <7c23adaa0609232341n3549e03ct8ab7b53773d71f13@mail.gmail.com> Message-ID: Hi! On Sun, 24 Sep 2006 09:41:04 +0300, "Ivan Toshkov" wrote: > First of all, thank you Edi Weitz for your excellent libraries. I > find them simple to use, well thought out, and with excellent > documentation. Again, a big thank you. You're welcome... :) > And thanks to whoever is working on porting Drakma. I think that's me... :) > I found a couple of problems: > > Drakma cannot receive cookies from localhost. After a short > investigation I found that the problem was with the method > VALID-COOKIE-DOMAIN-P, which insist to have at least 2 dots in the > normalized domain name. As a workaround I used 127.0.0.1 instead of > localhost. Thanks for the report, I've released a new version (0.4.4) which fixes this. > The other problem is with an incomplete implementation of > flexi-streams under CMUCL. The call: > >> (drakma:http-request "http://www.google.com/" :parameters '(("q" . "something"))) > > works normally, but > >> (drakma:http-request "http://www.google.com/search" :parameters '(("q" . "something"))) > > gives: > > No matching method for the generic function > #, when called with > arguments (#). > [Condition of type PCL::NO-APPLICABLE-METHOD-ERROR] > > 0: ("DEFMETHOD NO-APPLICABLE-METHOD (T)" #<#1=unused-arg> #<#1#> > # > (#)) > 1: (FLEXI-STREAMS:MAKE-FLEXI-STREAM > # :EXTERNAL-FORMAT > :LATIN-1) > 2: (FLEXI-STREAMS:STRING-TO-OCTETS "q" :EXTERNAL-FORMAT :LATIN-1 :START ...) > 3: (DRAKMA::URL-ENCODE "q" :LATIN-1) > 4: (DRAKMA::ALIST-TO-URL-ENCODED-STRING (("q" . "something")) :LATIN-1) > 5: (DRAKMA:HTTP-REQUEST # > :PARAMETERS (("q" . "something"))) > > I don't have time at the moment to see if anything else is missing > and to give patches. For this case though the workaround seems to > be as simple as: > > (defmethod input-stream-p ((s flexi-streams::vector-output-stream)) nil) If I understand the Gray streams proposal correctly, this isn't an incompleteness in FLEXI-STREAMS, but a problem in CMUCL's Gray streams implementation. Are you using the newest version of CMUCL? Have you asked on their mailing list? (It is also not clear to me why the problem only surfaces with the longer URL because URL-ENCODE has to be called in both cases.) Cheers, Edi. From ivan at toshkov.org Sun Sep 24 22:12:35 2006 From: ivan at toshkov.org (Ivan Toshkov) Date: Mon, 25 Sep 2006 01:12:35 +0300 Subject: [drakma-devel] Re: New version 0.4.4 (Was: Couple of problems) In-Reply-To: References: <7c23adaa0609232341n3549e03ct8ab7b53773d71f13@mail.gmail.com> Message-ID: <7c23adaa0609241512m2d957c19p1b1bd4526ecbf227@mail.gmail.com> On 9/24/06, Edi Weitz wrote: > Hi! > > On Sun, 24 Sep 2006 09:41:04 +0300, "Ivan Toshkov" wrote: > > > First of all, thank you Edi Weitz for your excellent libraries. I > > find them simple to use, well thought out, and with excellent > > documentation. Again, a big thank you. > > You're welcome... :) > > > And thanks to whoever is working on porting Drakma. > > I think that's me... :) Right. I was under the wrong impression that the port was done by Zach Beane, because I read about it on his blog. > > > I found a couple of problems: > > > > Drakma cannot receive cookies from localhost. After a short > > investigation I found that the problem was with the method > > VALID-COOKIE-DOMAIN-P, which insist to have at least 2 dots in the > > normalized domain name. As a workaround I used 127.0.0.1 instead of > > localhost. > > Thanks for the report, I've released a new version (0.4.4) which fixes > this. > Great. And I already have another one. The funcall (drakma:http-request "http://www.openrdf.org/sesame/servlets/listRepositories") returns: NIL 200 ((:CACHE-CONTROL . "no-cache") (:EXPIRES . "0") (:PRAGMA . "no-cache") (:CONTENT-TYPE . "text/xml") (:DATE . "Sun, 24 Sep 2006 21:52:34 GMT") (:SERVER . "Apache-Coyote/1.1") (:CONNECTION . "close")) # # T while the browser shows a small XML. I traced the problem to the READ-BODY function, which assumes (perhaps rightly so) that if the content length header is missing the stream must be a chunked stream. In this case it's not and so READ-BODY returns NIL. (please check my excuse for not providing any patches at the bottom of this email.) > > The other problem is with an incomplete implementation of > > flexi-streams under CMUCL. The call: > > > >> (drakma:http-request "http://www.google.com/" :parameters '(("q" . "something"))) > > > > works normally, but > > > >> (drakma:http-request "http://www.google.com/search" :parameters '(("q" . "something"))) > > > > gives: > > > > No matching method for the generic function > > #, when called with > > arguments (#). > > [Condition of type PCL::NO-APPLICABLE-METHOD-ERROR] > > > > 0: ("DEFMETHOD NO-APPLICABLE-METHOD (T)" #<#1=unused-arg> #<#1#> > > # > > (#)) > > 1: (FLEXI-STREAMS:MAKE-FLEXI-STREAM > > # :EXTERNAL-FORMAT > > :LATIN-1) > > 2: (FLEXI-STREAMS:STRING-TO-OCTETS "q" :EXTERNAL-FORMAT :LATIN-1 :START ...) > > 3: (DRAKMA::URL-ENCODE "q" :LATIN-1) > > 4: (DRAKMA::ALIST-TO-URL-ENCODED-STRING (("q" . "something")) :LATIN-1) > > 5: (DRAKMA:HTTP-REQUEST # > > :PARAMETERS (("q" . "something"))) > > > > I don't have time at the moment to see if anything else is missing > > and to give patches. For this case though the workaround seems to > > be as simple as: > > > > (defmethod input-stream-p ((s flexi-streams::vector-output-stream)) nil) > > If I understand the Gray streams proposal correctly, this isn't an > incompleteness in FLEXI-STREAMS, but a problem in CMUCL's Gray streams > implementation. Are you using the newest version of CMUCL? CMUCL 19c. > Have you asked on their mailing list? I haven't. Please check the excuse again :) > (It is also not clear to me why the problem only surfaces with the longer URL because URL-ENCODE has to be > called in both cases.) I thought it was caused by the :parameters keyword, but I actually haven't looked at the code. > > Cheers, > Edi. > EXCUSE: I have to make this legacy Lisp app to talk to a Java web app and I have to do it by Tuesday for a demo. So I don't have the time to do "the right thing" right now and I'm doing the minimal fixes required. Cheers, Ivan -- "...Please don't assume Lisp is only useful for Animation and Graphics, AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor applications, Expert Systems, Finance, Intelligent Agents, Knowledge Management, Mechanical CAD, Modeling and Simulation, Natural Language, Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web Authoring just because these are the only things they happened to list." -- Kent Pitman From colin_simmonds at Mac.lover.org Mon Sep 25 03:00:32 2006 From: colin_simmonds at Mac.lover.org (Colin Simmonds) Date: Sun, 24 Sep 2006 23:00:32 -0400 Subject: [drakma-devel] Patch for autoreferer support Message-ID: <9333274A-220E-4576-90A0-397411F481D7@Mac.lover.org> I'd previously been using Curl through FFI bindings for a simple Lisp web client project of mine, but have now switched to Drakma as it's much easier to use. However, there was one feature in Curl that I needed which was missing, and was easily added in the attached patch. Curl has an autoreferer option, which when following redirects, automatically adds a Referer header containing the URI of the page that is being redirected from. I've added an equivalent option to the http-request function. -------------- next part -------------- A non-text attachment was scrubbed... Name: autoreferer.patch Type: application/octet-stream Size: 2851 bytes Desc: not available URL: -------------- next part -------------- Colin Simmonds From edi at agharta.de Mon Sep 25 07:59:29 2006 From: edi at agharta.de (Edi Weitz) Date: Mon, 25 Sep 2006 09:59:29 +0200 Subject: [drakma-devel] New version 0.5.0 (Was: Patch for autoreferer support) In-Reply-To: <9333274A-220E-4576-90A0-397411F481D7@Mac.lover.org> (Colin Simmonds's message of "Sun, 24 Sep 2006 23:00:32 -0400") References: <9333274A-220E-4576-90A0-397411F481D7@Mac.lover.org> Message-ID: On Sun, 24 Sep 2006 23:00:32 -0400, Colin Simmonds wrote: > I'd previously been using Curl through FFI bindings for a simple > Lisp web client project of mine, but have now switched to Drakma as > it's much easier to use. > > However, there was one feature in Curl that I needed which was > missing, and was easily added in the attached patch. Curl has an > autoreferer option, which when following redirects, automatically > adds a Referer header containing the URI of the page that is being > redirected from. I've added an equivalent option to the http-request > function. Thanks for the patch - I've added the new functionality to the 0.5.0 release. Note, however, that I've called it AUTO-REFERER (note the hyphen). Cheers, Edi. From edi at agharta.de Mon Sep 25 08:03:23 2006 From: edi at agharta.de (Edi Weitz) Date: Mon, 25 Sep 2006 10:03:23 +0200 Subject: [drakma-devel] Re: New version 0.4.4 In-Reply-To: <7c23adaa0609241512m2d957c19p1b1bd4526ecbf227@mail.gmail.com> (Ivan Toshkov's message of "Mon, 25 Sep 2006 01:12:35 +0300") References: <7c23adaa0609232341n3549e03ct8ab7b53773d71f13@mail.gmail.com> <7c23adaa0609241512m2d957c19p1b1bd4526ecbf227@mail.gmail.com> Message-ID: On Mon, 25 Sep 2006 01:12:35 +0300, "Ivan Toshkov" wrote: > Great. And I already have another one. The funcall > > (drakma:http-request "http://www.openrdf.org/sesame/servlets/listRepositories") > > returns: > > NIL > 200 > ((:CACHE-CONTROL . "no-cache") (:EXPIRES . "0") (:PRAGMA . "no-cache") > (:CONTENT-TYPE . "text/xml") (:DATE . "Sun, 24 Sep 2006 21:52:34 GMT") > (:SERVER . "Apache-Coyote/1.1") (:CONNECTION . "close")) > # > # > T > > while the browser shows a small XML. I traced the problem to the > READ-BODY function, which assumes (perhaps rightly so) that if the > content length header is missing the stream must be a chunked > stream. In this case it's not and so READ-BODY returns NIL. Yeah, that's a mistake due to a thinko of mine which must have crept in around version 0.4.0. Should be fixed in 0.5.0 which was just released. Thanks for catching that. >> Have you asked on their mailing list? > > I haven't. Please check the excuse again :) OK, once you have more time you should probably ask there. If it's really an oversight in CMUCL, it should be fixed there and not in FLEXI-STREAMS. > EXCUSE: I have to make this legacy Lisp app to talk to a Java web > app and I have to do it by Tuesday for a demo. So I don't have the > time to do "the right thing" right now and I'm doing the minimal > fixes required. No problem. Good luck with the demo! Cheers, Edi.