From robert.brown at gmail.com Fri Mar 9 15:24:44 2012 From: robert.brown at gmail.com (Robert Brown) Date: Fri, 9 Mar 2012 10:24:44 -0500 Subject: [drakma-devel] drakma and non-ASCII content Message-ID: A co-worker of mine had some problems today using Drakma to POST a STRING containing non-ASCII characters encoded as UTF-8. He was doing something equivalent to: (http-request "http://zappa.com/favicon.ico" :method :post :content (concatenate 'string "hello" (string #\white_square) "world") :external-format-out :utf-8) which ends up setting the Content-Length header value to 11, which is the LENGTH in characters of the string being sent. The documentation says that "if content is a sequence, Drakma will use LENGTH to determine its length and will use the result for the 'Content-Length' header sent to the server," so Drakma is working as documented. Also, I think I understand why this behavior is the default. You don't want to scan a content string in order to determine what its length will be in octets once it has been encoded. My co-worker could have used a vector of type (unsigned-byte 8) to hold his UTF-8 encoded data. However, I think the current Drakma behavior may be a mistake. People who want high performance are probably manipulating encoded strings as vectors of (unsigned-byte 8). It's the casual users, those sending strings, who are the ones most likely to be bitten by the default behavior, but only when their strings contain non-ASCII characters. bob From austin at pettomato.com Fri Mar 9 15:41:40 2012 From: austin at pettomato.com (Austin Haas) Date: Fri, 9 Mar 2012 07:41:40 -0800 Subject: [drakma-devel] drakma and non-ASCII content In-Reply-To: References: Message-ID: <20120309154139.GA19866@mars.chicago> You can also explicitly supply the value of the content-length as a keyword argument to http-request. I use a wrapper that adds this: :content-length (if content (babel:string-size-in-octets content :encoding :utf-8) 0) I agree, though, that the default behavior can be surprising. It has bitten me, too, and I've seen at least one other library built-on Drakma that didn't account for it, either. -austin -- Austin Haas Pet Tomato, Inc. http://pettomato.com On Fri Mar 09 10:24 , Robert Brown wrote: > A co-worker of mine had some problems today using Drakma to POST a STRING > containing non-ASCII characters encoded as UTF-8. He was doing something > equivalent to: > > (http-request "http://zappa.com/favicon.ico" > :method :post > :content (concatenate 'string > "hello" (string #\white_square) "world") > :external-format-out :utf-8) > > which ends up setting the Content-Length header value to 11, which is the > LENGTH in characters of the string being sent. > > The documentation says that "if content is a sequence, Drakma will use LENGTH > to determine its length and will use the result for the 'Content-Length' header > sent to the server," so Drakma is working as documented. > > Also, I think I understand why this behavior is the default. You don't want to > scan a content string in order to determine what its length will be in octets > once it has been encoded. My co-worker could have used a vector of type > (unsigned-byte 8) to hold his UTF-8 encoded data. > > However, I think the current Drakma behavior may be a mistake. People who want > high performance are probably manipulating encoded strings as vectors of > (unsigned-byte 8). It's the casual users, those sending strings, who are the > ones most likely to be bitten by the default behavior, but only when their > strings contain non-ASCII characters. > > bob > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > From whalliburton at gmail.com Sat Mar 10 00:08:22 2012 From: whalliburton at gmail.com (William Halliburton) Date: Fri, 9 Mar 2012 16:08:22 -0800 Subject: [drakma-devel] Unwanted url-encoding of GET parameters. Message-ID: Hello folks, I'm trying to use drakma to fetch urls that contain utf8 characters but HTTP-REQUEST automatically url encodes any non latin-1 ascii characters. On my cursory reading of the RFCs, this seems conforming behavor, but in this case it is definitely unwanted. For example, the following url http://translate.google.com/translate_tts?tl=ru&q=?? if entered directly into the browser correctly returns the text-to-speech audo file but when attempting to use HTTP-REQUEST, the url is being url encoded into http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B of which google does not url-decode and fails to return the correct data. So, for this case, the url-encoding is unwanted. I am willing to submit patch an additional argument into HTTP-REQUEST to disallow the encoding. Thoughts? Thank you, William -------------- next part -------------- An HTML attachment was scrubbed... URL: From edi at agharta.de Sat Mar 10 10:46:49 2012 From: edi at agharta.de (Edi Weitz) Date: Sat, 10 Mar 2012 11:46:49 +0100 Subject: [drakma-devel] Unwanted url-encoding of GET parameters. In-Reply-To: References: Message-ID: You are not allowed to send arbitrary characters in the request line. FWIW, I just tried your example with Firefox and this is what the browser sends according to LiveHttpHeaders: http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B And Google returns the requested audio file. Edi. On Sat, Mar 10, 2012 at 1:08 AM, William Halliburton wrote: > > Hello folks, > > I'm trying to use drakma to fetch urls that contain utf8 characters but > HTTP-REQUEST automatically url encodes any non latin-1 ascii characters. > > On my cursory reading of the RFCs, this seems conforming behavor, but in > this case it is definitely unwanted. > > For example, the following url > > ??http://translate.google.com/translate_tts?tl=ru&q=?? > > if entered directly into the browser correctly returns the text-to-speech > audo file but > > when attempting to use HTTP-REQUEST, the url is being url encoded into > > ??http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B > > of which google does not url-decode and fails to return the correct data. > > So, for this case, the url-encoding is unwanted. > > I am willing to submit patch an additional argument into HTTP-REQUEST to > disallow the encoding. > > Thoughts? > > Thank you, > William > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > From whalliburton at gmail.com Sun Mar 11 05:50:22 2012 From: whalliburton at gmail.com (William Halliburton) Date: Sat, 10 Mar 2012 21:50:22 -0800 Subject: [drakma-devel] Unwanted url-encoding of GET parameters. In-Reply-To: References: Message-ID: Thanks much. After some wiresharking, I found that google doesn't like the drakma user-agent and changing it to firefox did the trick. On Sat, Mar 10, 2012 at 2:46 AM, Edi Weitz wrote: > You are not allowed to send arbitrary characters in the request line. > FWIW, I just tried your example with Firefox and this is what the > browser sends according to LiveHttpHeaders: > > http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B > > And Google returns the requested audio file. > > Edi. > > On Sat, Mar 10, 2012 at 1:08 AM, William Halliburton > wrote: > > > > Hello folks, > > > > I'm trying to use drakma to fetch urls that contain utf8 characters but > > HTTP-REQUEST automatically url encodes any non latin-1 ascii characters. > > > > On my cursory reading of the RFCs, this seems conforming behavor, but in > > this case it is definitely unwanted. > > > > For example, the following url > > > > http://translate.google.com/translate_tts?tl=ru&q=?? > > > > if entered directly into the browser correctly returns the text-to-speech > > audo file but > > > > when attempting to use HTTP-REQUEST, the url is being url encoded into > > > > http://translate.google.com/translate_tts?tl=ru&q=%D0%B2%D1%8B > > > > of which google does not url-decode and fails to return the correct data. > > > > So, for this case, the url-encoding is unwanted. > > > > I am willing to submit patch an additional argument into HTTP-REQUEST to > > disallow the encoding. > > > > Thoughts? > > > > Thank you, > > William > > > > _______________________________________________ > > drakma-devel mailing list > > drakma-devel at common-lisp.net > > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > > > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edi at weitz.de Sun Mar 11 14:57:53 2012 From: edi at weitz.de (Edi Weitz) Date: Sun, 11 Mar 2012 15:57:53 +0100 Subject: [drakma-devel] Unwanted url-encoding of GET parameters. In-Reply-To: References: Message-ID: On Sun, Mar 11, 2012 at 6:50 AM, William Halliburton wrote: > After some wiresharking, I found that google doesn't like the > drakma user-agent Bastards... :) From minerva at agentsheets.com Thu Mar 15 17:53:16 2012 From: minerva at agentsheets.com (Michael Minerva) Date: Thu, 15 Mar 2012 11:53:16 -0600 Subject: [drakma-devel] Changing content-type seems to have no effect Message-ID: <55C01673-6364-4ED5-AE24-AA391DDAC8A3@agentsheets.com> I have been trying to get my lisp application to interface and upload a file to my drupal 7 website using Drakma. So far I have been able to to connect to my drupal site and authenticate and store the session cookie (thank you Ryan Davis, I just found your response the other day). Here is the code I use to do that: (with-output-to-string (stream) (setf drakma:*header-stream* stream) (let ((cookie-jar (make-instance 'drakma:cookie-jar))) (drakma:http-request "http://localhost:8888/?q=rest/user/login" :method :post :cookie-jar cookie-jar :parameters '(("username" . "login") ("password" . "password")) ) (drakma:http-request "http://localhost:8888/?q=rest/file" :method :post :cookie-jar cookie-jar :content-length t :parameters '(("file1" . #p"/Users/Mike/hello.png"));("file1" #p"/Users/Mike/Desktop/mario-gif.gif" :content-type "image/gif" :filename "mario-gif.gif")) :content-type "application/x-www-form-urlencoded" ;:parameters ))) This code successfully sends both posts but the second post does not actually upload the file (but it does return 200 OK, no errors). I have successfully uploaded to by drupal site using PHP Curl and in that code, I was only able to get it to work by using a content type of application/x-www-form-urlencoded (if I use a multipart/form-data request on my php page I get similar behavior to what I am now seeing with Drakma). This is why I added the line :content-type "application/x-www-form-urlencoded" to my second request (posted above) but it seems to have no effect. Here is a copy of the posts I am making (as you can see in the second post it still uses multipart/form-data. Is there anyway that I can upload a file using Drakma using a content-type of application/x-www-form-urlencoded (I know this is not the recommended way of making this type of request but I think it may be the only way that my drupal site will accept the file). "POST /?q=rest/user/login HTTP/1.1 Host: localhost:8888 User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) Accept: */* Connection: close Content-Type: application/x-www-form-urlencoded Content-Length: 34 HTTP/1.1 200 OK Date: Thu, 15 Mar 2012 17:49:51 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Expires: Sun, 19 Nov 1978 05:00:00 GMT Last-Modified: Thu, 15 Mar 2012 17:49:51 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: \"1331833791\" Vary: Accept Set-Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ; expires=Sat, 07-Apr-2012 21:23:12 GMT; path=/; HttpOnly Content-Length: 115 Connection: close Content-Type: text/yaml POST /?q=rest/file HTTP/1.1 Host: localhost:8888 User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) Accept: */* Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ Connection: close Content-Type: multipart/form-data; boundary=----------ndnZ9xjDrDpzStPEQo97xwqPHEKXAhAOd8Ho6C8P3jtKbvNINm Content-Length: 244958 HTTP/1.1 200 OK Date: Thu, 15 Mar 2012 17:49:52 GMT Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 X-Powered-By: PHP/5.3.6 Expires: Sun, 19 Nov 1978 05:00:00 GMT Last-Modified: Thu, 15 Mar 2012 17:49:52 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: \"1331833792\" Vary: Accept Content-Length: 4 Connection: close Content-Type: text/yaml " Thanks, --Michael From edi at agharta.de Thu Mar 15 21:08:21 2012 From: edi at agharta.de (Edi Weitz) Date: Thu, 15 Mar 2012 22:08:21 +0100 Subject: [drakma-devel] Changing content-type seems to have no effect In-Reply-To: <55C01673-6364-4ED5-AE24-AA391DDAC8A3@agentsheets.com> References: <55C01673-6364-4ED5-AE24-AA391DDAC8A3@agentsheets.com> Message-ID: If you want to use the content type "application/x-www-form-urlencoded", you can only use strings as values in the parameter list. You could, if you insist, read your file into a string if it's not too big. I have to say that this all sounds wrong, though. Edi. On Thu, Mar 15, 2012 at 6:53 PM, Michael Minerva wrote: > I have been trying to get my lisp application to interface and upload a file to my drupal 7 website using Drakma. > > So far I have been able to to connect to my drupal site and authenticate and store the session cookie (thank you Ryan Davis, I just found your response the other day). ?Here is the code I use to do that: > > (with-output-to-string (stream) > ?(setf drakma:*header-stream* stream) > (let ((cookie-jar (make-instance 'drakma:cookie-jar))) > ?(drakma:http-request > ? "http://localhost:8888/?q=rest/user/login" > ? ?:method :post > ? ?:cookie-jar cookie-jar > ? ?:parameters > ? ?'(("username" . "login") > ? ? ?("password" . "password")) > ? ?) > ?(drakma:http-request > ? "http://localhost:8888/?q=rest/file" > ? ?:method :post > ? ?:cookie-jar cookie-jar > ? ?:content-length t > ? ?:parameters '(("file1" . #p"/Users/Mike/hello.png"));("file1" #p"/Users/Mike/Desktop/mario-gif.gif" :content-type "image/gif" :filename "mario-gif.gif")) > ? ?:content-type "application/x-www-form-urlencoded" > ? ?;:parameters > ? ?))) > > > This code successfully sends both posts but the second post does not actually upload the file (but it does return 200 OK, no errors). ?I have successfully uploaded to by drupal site using PHP Curl and in that code, I was only able to get it to work by using a content type of application/x-www-form-urlencoded (if I use a multipart/form-data request on my php page I get similar behavior to what I am now seeing with Drakma). ?This is why I added the line :content-type "application/x-www-form-urlencoded" ?to my second request (posted above) but it seems to have no effect. > > Here is a copy of the posts I am making (as you can see in the second post it still uses multipart/form-data. ?Is there anyway that I can upload a file using Drakma using a content-type of application/x-www-form-urlencoded (I know this is not the recommended way of making this type of request but I think it may be the only way that my drupal site will accept the file). > > "POST /?q=rest/user/login HTTP/1.1 > Host: localhost:8888 > User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M ?(DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) > Accept: */* > Connection: close > Content-Type: application/x-www-form-urlencoded > Content-Length: 34 > > HTTP/1.1 200 OK > Date: Thu, 15 Mar 2012 17:49:51 GMT > Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 > X-Powered-By: PHP/5.3.6 > Expires: Sun, 19 Nov 1978 05:00:00 GMT > Last-Modified: Thu, 15 Mar 2012 17:49:51 +0000 > Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 > ETag: \"1331833791\" > Vary: Accept > Set-Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ; expires=Sat, 07-Apr-2012 21:23:12 GMT; path=/; HttpOnly > Content-Length: 115 > Connection: close > Content-Type: text/yaml > > POST /?q=rest/file HTTP/1.1 > Host: localhost:8888 > User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M ?(DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) > Accept: */* > Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ > Connection: close > Content-Type: multipart/form-data; boundary=----------ndnZ9xjDrDpzStPEQo97xwqPHEKXAhAOd8Ho6C8P3jtKbvNINm > Content-Length: 244958 > > HTTP/1.1 200 OK > Date: Thu, 15 Mar 2012 17:49:52 GMT > Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 > X-Powered-By: PHP/5.3.6 > Expires: Sun, 19 Nov 1978 05:00:00 GMT > Last-Modified: Thu, 15 Mar 2012 17:49:52 +0000 > Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 > ETag: \"1331833792\" > Vary: Accept > Content-Length: 4 > Connection: close > Content-Type: text/yaml > > " > > Thanks, > > --Michael > > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel From minerva at agentsheets.com Thu Mar 15 23:24:37 2012 From: minerva at agentsheets.com (Michael Minerva) Date: Thu, 15 Mar 2012 17:24:37 -0600 Subject: [drakma-devel] Changing content-type seems to have no effect In-Reply-To: References: <55C01673-6364-4ED5-AE24-AA391DDAC8A3@agentsheets.com> Message-ID: <45EB3083-C680-47B2-87D7-0E8D2BA8BBB6@agentsheets.com> Hey Edi, Thank you so much for your response. I am now thinking my problem may have something to do with the encoding will the file be encoded using base64 (with multipart/form-data)? Is this something you can set? Is this what external-format-out is for? Thanks again, --Mike On Mar 15, 2012, at 3:08 PM, Edi Weitz wrote: > If you want to use the content type > "application/x-www-form-urlencoded", you can only use strings as > values in the parameter list. You could, if you insist, read your > file into a string if it's not too big. I have to say that this all > sounds wrong, though. > > Edi. > > On Thu, Mar 15, 2012 at 6:53 PM, Michael Minerva > wrote: >> I have been trying to get my lisp application to interface and upload a file to my drupal 7 website using Drakma. >> >> So far I have been able to to connect to my drupal site and authenticate and store the session cookie (thank you Ryan Davis, I just found your response the other day). Here is the code I use to do that: >> >> (with-output-to-string (stream) >> (setf drakma:*header-stream* stream) >> (let ((cookie-jar (make-instance 'drakma:cookie-jar))) >> (drakma:http-request >> "http://localhost:8888/?q=rest/user/login" >> :method :post >> :cookie-jar cookie-jar >> :parameters >> '(("username" . "login") >> ("password" . "password")) >> ) >> (drakma:http-request >> "http://localhost:8888/?q=rest/file" >> :method :post >> :cookie-jar cookie-jar >> :content-length t >> :parameters '(("file1" . #p"/Users/Mike/hello.png"));("file1" #p"/Users/Mike/Desktop/mario-gif.gif" :content-type "image/gif" :filename "mario-gif.gif")) >> :content-type "application/x-www-form-urlencoded" >> ;:parameters >> ))) >> >> >> This code successfully sends both posts but the second post does not actually upload the file (but it does return 200 OK, no errors). I have successfully uploaded to by drupal site using PHP Curl and in that code, I was only able to get it to work by using a content type of application/x-www-form-urlencoded (if I use a multipart/form-data request on my php page I get similar behavior to what I am now seeing with Drakma). This is why I added the line :content-type "application/x-www-form-urlencoded" to my second request (posted above) but it seems to have no effect. >> >> Here is a copy of the posts I am making (as you can see in the second post it still uses multipart/form-data. Is there anyway that I can upload a file using Drakma using a content-type of application/x-www-form-urlencoded (I know this is not the recommended way of making this type of request but I think it may be the only way that my drupal site will accept the file). >> >> "POST /?q=rest/user/login HTTP/1.1 >> Host: localhost:8888 >> User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) >> Accept: */* >> Connection: close >> Content-Type: application/x-www-form-urlencoded >> Content-Length: 34 >> >> HTTP/1.1 200 OK >> Date: Thu, 15 Mar 2012 17:49:51 GMT >> Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 >> X-Powered-By: PHP/5.3.6 >> Expires: Sun, 19 Nov 1978 05:00:00 GMT >> Last-Modified: Thu, 15 Mar 2012 17:49:51 +0000 >> Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 >> ETag: \"1331833791\" >> Vary: Accept >> Set-Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ; expires=Sat, 07-Apr-2012 21:23:12 GMT; path=/; HttpOnly >> Content-Length: 115 >> Connection: close >> Content-Type: text/yaml >> >> POST /?q=rest/file HTTP/1.1 >> Host: localhost:8888 >> User-Agent: Drakma/1.2.3 (Clozure Common Lisp Version 1.7-r14927M (DarwinX8664); Darwin; 10.8.0; http://weitz.de/drakma/) >> Accept: */* >> Cookie: SESSdc0685ed01f285dab628a3700259e6bc=oIJFlVrfoXi-PO7bdwKtauKfTeRSEK1iIucj0tl12UQ >> Connection: close >> Content-Type: multipart/form-data; boundary=----------ndnZ9xjDrDpzStPEQo97xwqPHEKXAhAOd8Ho6C8P3jtKbvNINm >> Content-Length: 244958 >> >> HTTP/1.1 200 OK >> Date: Thu, 15 Mar 2012 17:49:52 GMT >> Server: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.6 >> X-Powered-By: PHP/5.3.6 >> Expires: Sun, 19 Nov 1978 05:00:00 GMT >> Last-Modified: Thu, 15 Mar 2012 17:49:52 +0000 >> Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 >> ETag: \"1331833792\" >> Vary: Accept >> Content-Length: 4 >> Connection: close >> Content-Type: text/yaml >> >> " >> >> Thanks, >> >> --Michael >> >> >> _______________________________________________ >> drakma-devel mailing list >> drakma-devel at common-lisp.net >> http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel >