From eric_a_benson at yahoo.com  Mon Jul  7 14:28:13 2008
From: eric_a_benson at yahoo.com (Eric Benson)
Date: Mon, 7 Jul 2008 07:28:13 -0700 (PDT)
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
Message-ID: <810040.38094.qm@web65403.mail.ac4.yahoo.com>

It appears that drakma is encoding plus signs (+) in query parameters, turning them into %2B when they should be left alone. This confuses the server in this case. You can see that if I use the :parameters keyword the request fails, but if I include the same parameters as part of the URL the request succeeds.



* (setq drakma:*header-stream* *standard-output*)

#<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {91260B9}>
* (drakma:http-request "http://freedb.org/~cddb/cddb.cgi" :parameters '(("cmd" . "cddb+lscat") ("hello" . "user+host+freedb+1.0") ("proto" . "6")))
GET /~cddb/cddb.cgi?cmd=cddb%2Blscat&hello=user%2Bhost%2Bfreedb%2B1.0&proto=6 HTTP/1.1
Host: freedb.org
User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/)
Accept: */*
Connection: close

HTTP/1.1 200 OK
Date: Mon, 07 Jul 2008 14:17:36 GMT
Server: Apache/2.0.54 (Debian GNU/Linux)
Expires: Mon Jul  7 16:17:37 2008
Connection: close
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8

"500 Command syntax error: incorrect arg count forhandshake. 
"
200
((:DATE . "Mon, 07 Jul 2008 14:17:36 GMT")
 (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)")
 (:EXPIRES . "Mon Jul  7 16:17:37 2008") (:CONNECTION . "close")
 (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8"))
#<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb%2Blscat&hello=user%2Bhost%2Bfreedb%2B1.0&proto=6>
#<FLEXI-STREAMS:FLEXI-IO-STREAM {BA65021}>
T
"OK"
* (drakma:http-request "http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6")
GET /~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6 HTTP/1.1
Host: freedb.org
User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/)
Accept: */*
Connection: close

HTTP/1.1 200 OK
Date: Mon, 07 Jul 2008 14:17:42 GMT
Server: Apache/2.0.54 (Debian GNU/Linux)
Expires: Mon Jul  7 16:17:42 2008
Connection: close
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8

"210 OK, category list follows (until terminating `.') 
data 
folk 
jazz 
misc 
rock 
country 
blues 
newage 
reggae 
classical 
soundtrack 
. 
"
200
((:DATE . "Mon, 07 Jul 2008 14:17:42 GMT")
 (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)")
 (:EXPIRES . "Mon Jul  7 16:17:42 2008") (:CONNECTION . "close")
 (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8"))
#<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6>
#<FLEXI-STREAMS:FLEXI-IO-STREAM {BAAA461}>
T
"OK"
* 



From edi at agharta.de  Mon Jul  7 14:41:47 2008
From: edi at agharta.de (Edi Weitz)
Date: Mon, 07 Jul 2008 16:41:47 +0200
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
In-Reply-To: <810040.38094.qm@web65403.mail.ac4.yahoo.com> (Eric Benson's
	message of "Mon, 7 Jul 2008 07:28:13 -0700 (PDT)")
References: <810040.38094.qm@web65403.mail.ac4.yahoo.com>
Message-ID: <uvdzh3ges.fsf@agharta.de>

On Mon, 7 Jul 2008 07:28:13 -0700 (PDT), Eric Benson <eric_a_benson at yahoo.com> wrote:

> It appears that drakma is encoding plus signs (+) in query
> parameters, turning them into %2B when they should be left alone.

According to 3.4 of RFC 2396 the character "+" is reserved in query
components.  Encoding plus signs in GET requests is also something
Firefox and Internet Explorer do.

> This confuses the server in this case.

Maybe the server is confused anyway?

Edi.


From eric_a_benson at yahoo.com  Mon Jul  7 16:08:18 2008
From: eric_a_benson at yahoo.com (Eric Benson)
Date: Mon, 7 Jul 2008 09:08:18 -0700 (PDT)
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
Message-ID: <196213.55091.qm@web65409.mail.ac4.yahoo.com>

RFC 2396 has been superseded by RFC 3986, although that does not affect this case.

The freedb server is using plus signs in HTTP requests to substitute for spaces, which are used in CDDBP requests. That appears to be supported by this recommendation from W3:

http://www.w3.org/Addressing/URL/4_URI_Recommentations.html

"Within the query string,
the plus sign is reserved as shorthand notation for a space.
Therefore, real plus signs must be encoded.
This method was used to make query URIs easier to pass in systems which did
not allow spaces."

For the record, neither IE nor Firefox encodes plus signs when they are entered as part of the query. However, they probably should! Spaces are encoded as plus signs. If I left the spaces in the CDDB query alone when passing them to http-request, they would be encoded as plus signs.

It seems to me that http-request should make encoding of the parameters
optional, since a client program may need to deal with encoded as well
as unencoded strings. Otherwise, a caller of http-request that already
has encoded parameters must explicitly decode them before calling. A
keyword argument that specifies whether parameters are already encoded,
defaulting to nil, would be a useful addition. Most HTTP client APIs separate the encoding of URLs and query strings from the network request, leaving it up to the caller to handle the encoding explicitly.


* (drakma:http-request "http://freedb.org/~cddb/cddb.cgi" :parameters '(("cmd" . "cddb lscat") ("hello" . "user host freedb 1.0") ("proto" . "6")))
GET /~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6 HTTP/1.1
Host: freedb.org
User-Agent: Drakma/0.11.5 (SBCL 1.0.11.debian; Linux; 2.6.24-19-generic; http://weitz.de/drakma/)
Accept: */*
Connection: close

HTTP/1.1 200 OK
Date: Mon, 07 Jul 2008 16:02:17 GMT
Server: Apache/2.0.54 (Debian GNU/Linux)
Expires: Mon Jul  7 18:02:17 2008
Connection: close
Transfer-Encoding: chunked
Content-Type: text/plain; charset=UTF-8

"210 OK, category list follows (until terminating `.') 
data 
folk 
jazz 
misc 
rock 
country 
blues 
newage 
reggae 
classical 
soundtrack 
. 
"
200
((:DATE . "Mon, 07 Jul 2008 16:02:17 GMT")
 (:SERVER . "Apache/2.0.54 (Debian GNU/Linux)")
 (:EXPIRES . "Mon Jul  7 18:02:17 2008") (:CONNECTION . "close")
 (:TRANSFER-ENCODING . "chunked") (:CONTENT-TYPE . "text/plain; charset=UTF-8"))
#<PURI:URI http://freedb.org/~cddb/cddb.cgi?cmd=cddb+lscat&hello=user+host+freedb+1.0&proto=6>
#<FLEXI-STREAMS:FLEXI-IO-STREAM {BAE7859}>
T
"OK"
* 





----- Original Message ----
From: Edi Weitz <edi at agharta.de>
To: General interest list for Drakma and Chunga <drakma-devel at common-lisp.net>
Sent: Monday, July 7, 2008 7:41:47 AM
Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters

On Mon, 7 Jul 2008 07:28:13 -0700 (PDT), Eric Benson <eric_a_benson at yahoo.com> wrote:

> It appears that drakma is encoding plus signs (+) in query
> parameters, turning them into %2B when they should be left alone.

According to 3.4 of RFC 2396 the character "+" is reserved in query
components.  Encoding plus signs in GET requests is also something
Firefox and Internet Explorer do.

> This confuses the server in this case.

Maybe the server is confused anyway?

Edi.
_______________________________________________
drakma-devel mailing list
drakma-devel at common-lisp.net
http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel



From edi at agharta.de  Mon Jul  7 16:41:29 2008
From: edi at agharta.de (Edi Weitz)
Date: Mon, 07 Jul 2008 18:41:29 +0200
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
In-Reply-To: <196213.55091.qm@web65409.mail.ac4.yahoo.com> (Eric Benson's
	message of "Mon, 7 Jul 2008 09:08:18 -0700 (PDT)")
References: <196213.55091.qm@web65409.mail.ac4.yahoo.com>
Message-ID: <uskul4pfq.fsf@agharta.de>

On Mon, 7 Jul 2008 09:08:18 -0700 (PDT), Eric Benson <eric_a_benson at yahoo.com> wrote:

> RFC 2396 has been superseded by RFC 3986, although that does not
> affect this case.

RFC 2396 is what is quoted in RFC 2616, so that's why I was referring
to it.

> http://www.w3.org/Addressing/URL/4_URI_Recommentations.html
>
> "Within the query string, the plus sign is reserved as shorthand
> notation for a space.  Therefore, real plus signs must be encoded.

That's what Drakma does.

> For the record, neither IE nor Firefox encodes plus signs when they
> are entered as part of the query.

They encode them if they are GET parameters, like Drakma.  Open the
attached HTML page in IE or Firefox and enter "a+b", then press the
Return key.

> It seems to me that http-request should make encoding of the
> parameters optional, since a client program may need to deal with
> encoded as well as unencoded strings. Otherwise, a caller of
> http-request that already has encoded parameters must explicitly
> decode them before calling.

No, you are free to build the URL for Drakma yourself.

Edi.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20080707/04ce47d3/attachment.html>

From eric_a_benson at yahoo.com  Mon Jul  7 17:07:15 2008
From: eric_a_benson at yahoo.com (Eric Benson)
Date: Mon, 7 Jul 2008 10:07:15 -0700 (PDT)
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
Message-ID: <278162.37896.qm@web65402.mail.ac4.yahoo.com>

Building your own URL only works for GET requests, not for POST requests.

I don't need to stop drakma from encoding query strings for my application, I can pass it the string with spaces and let it encode them to plus signs. I'm suggesting a useful, minor, upward-compatible addition to drakma's API. Perhaps drakma's interface is too high-level for the kind of application that might need more direct control of encoding and decoding. I can understand not wanting to add another keyword to http-request and another few lines to the documentation. That's your judgement call. Trivial-http and cl-curl are also available.

Anyway, thanks for listening, and thanks for the code...



----- Original Message ----
From: Edi Weitz <edi at agharta.de>
To: General interest list for Drakma and Chunga <drakma-devel at common-lisp.net>
Sent: Monday, July 7, 2008 9:41:29 AM
Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters

On Mon, 7 Jul 2008 09:08:18 -0700 (PDT), Eric Benson <eric_a_benson at yahoo.com> wrote:

> RFC 2396 has been superseded by RFC 3986, although that does not
> affect this case.

RFC 2396 is what is quoted in RFC 2616, so that's why I was referring
to it.

> http://www.w3.org/Addressing/URL/4_URI_Recommentations.html
>
> "Within the query string, the plus sign is reserved as shorthand
> notation for a space.  Therefore, real plus signs must be encoded.

That's what Drakma does.

> For the record, neither IE nor Firefox encodes plus signs when they
> are entered as part of the query.

They encode them if they are GET parameters, like Drakma.  Open the
attached HTML page in IE or Firefox and enter "a+b", then press the
Return key.

> It seems to me that http-request should make encoding of the
> parameters optional, since a client program may need to deal with
> encoded as well as unencoded strings. Otherwise, a caller of
> http-request that already has encoded parameters must explicitly
> decode them before calling.

No, you are free to build the URL for Drakma yourself.

Edi.




-----Inline Attachment Follows-----


From edi at agharta.de  Mon Jul  7 18:55:14 2008
From: edi at agharta.de (Edi Weitz)
Date: Mon, 07 Jul 2008 20:55:14 +0200
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
In-Reply-To: <278162.37896.qm@web65402.mail.ac4.yahoo.com> (Eric Benson's
	message of "Mon, 7 Jul 2008 10:07:15 -0700 (PDT)")
References: <278162.37896.qm@web65402.mail.ac4.yahoo.com>
Message-ID: <uod594j8t.fsf@agharta.de>

On Mon, 7 Jul 2008 10:07:15 -0700 (PDT), Eric Benson <eric_a_benson at yahoo.com> wrote:

> Building your own URL only works for GET requests, not for POST
> requests.

http://weitz.de/drakma/#content


From eric_a_benson at yahoo.com  Mon Jul  7 19:54:41 2008
From: eric_a_benson at yahoo.com (Eric Benson)
Date: Mon, 7 Jul 2008 12:54:41 -0700 (PDT)
Subject: [drakma-devel] Bug report: Overeager encoding of parameters
Message-ID: <65227.52980.qm@web65413.mail.ac4.yahoo.com>

Ah, I overlooked the :CONTENT keyword. There's a strong argument for not adding another keyword!

So, there is a way to avoid re-encoding or de-encoding strings: build a URL for a GET request, or build a CONTENT block for a POST request. Not as convenient, but doable. And of course, it turned out I didn't need it anyway.

Thanks again.



----- Original Message ----
From: Edi Weitz <edi at agharta.de>
To: General interest list for Drakma and Chunga <drakma-devel at common-lisp.net>
Sent: Monday, July 7, 2008 11:55:14 AM
Subject: Re: [drakma-devel] Bug report: Overeager encoding of parameters

On Mon, 7 Jul 2008 10:07:15 -0700 (PDT), Eric Benson <eric_a_benson at yahoo.com> wrote:

> Building your own URL only works for GET requests, not for POST
> requests.

http://weitz.de/drakma/#content
_______________________________________________
drakma-devel mailing list
drakma-devel at common-lisp.net
http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel



From daniel at sentivision.com  Mon Jul 14 16:26:40 2008
From: daniel at sentivision.com (Daniel Janus)
Date: Mon, 14 Jul 2008 18:26:40 +0200
Subject: [drakma-devel] dotless cookie domains
Message-ID: <1216052800.10287.19.camel@daniel>

Hello list,

The current version of Drakma contains a check of cookie domains'
validity.  A (textual) domain is considered to be valid iff either it
contains a dot or is exactly "localhost".  I'm sure there is a reason
for this (perhaps some part of a relevant RFC?), but I got bitten by
it in a real-world scenario.  I was using Drakma for testing of a
Java EE-based web application at my company, accessible at the internal
address "http://someserver:9090/gui/app".  The relevant headers returned
are:

DRAKMA> (http-request "http://someserver:9090/gui/app")
==> 
"blah blah blah, some HTML"
200
((:X-POWERED-BY . "Servlet/2.5")
 (:SERVER . "Sun Java System Application Server 9.1")
 (:SET-COOKIE . "JSESSIONID=41165991cdf70add1d9e0a103cfc; Path=/gui")
 (:CONTENT-TYPE . "text/html;charset=UTF-8")
 (:DATE . "Tue, 15 Jul 2008 00:14:24 GMT") (:CONNECTION . "close"))
#<URI http://someserver:9090/gui/app>
#<FLEXI-STREAMS::FLEXI-UTF-8-IO-STREAM {BB78A59}>
T
"OK"

and if I pass a cookie jar to such HTTP-REQUEST, the returned cookie is
not included in the jar due to the checking machinery.  FWIW, Firefox 3
does include it (and so the app works on Firefox 3, but fails the
Drakma tests).

I quickly hacked up a patch (attached) to Drakma which adds a special
variable *ALLOW-DOTLESS-COOKIE-DOMAINS-P*, which, when set to non-NIL,
causes the domains like SOMESERVER to be accepted.

Might it be useful to include in the official distribution?

Best regards,
-- 
?Daniel Janus     <daniel at sentivision.com>
Quality Assurance Specialist, SENTIVISION
Mb: +48 501 180 396 | Ph: +48 22 640 0860
-------------- next part --------------
A non-text attachment was scrubbed...
Name: drakma.patch
Type: text/x-patch
Size: 1749 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20080714/26aa5a37/attachment.bin>

From edi at agharta.de  Wed Jul 16 07:29:34 2008
From: edi at agharta.de (Edi Weitz)
Date: Wed, 16 Jul 2008 09:29:34 +0200
Subject: [drakma-devel] dotless cookie domains
In-Reply-To: <1216052800.10287.19.camel@daniel> (Daniel Janus's message of
	"Mon, 14 Jul 2008 18:26:40 +0200")
References: <1216052800.10287.19.camel@daniel>
Message-ID: <uod4ye17l.fsf@agharta.de>

Hi Daniel,

On Mon, 14 Jul 2008 18:26:40 +0200, Daniel Janus <daniel at sentivision.com> wrote:

> The current version of Drakma contains a check of cookie domains'
> validity.  A (textual) domain is considered to be valid iff either
> it contains a dot or is exactly "localhost".  I'm sure there is a
> reason for this (perhaps some part of a relevant RFC?), but I got
> bitten by it in a real-world scenario.  I was using Drakma for
> testing of a Java EE-based web application at my company, accessible
> at the internal address "http://someserver:9090/gui/app".

I wouldn't call this "real-world" as in the real "real world" domain
names without a dot don't exist... :)

But, yes, this is obviously useful for in-house testing.

> I quickly hacked up a patch (attached) to Drakma which adds a
> special variable *ALLOW-DOTLESS-COOKIE-DOMAINS-P*, which, when set
> to non-NIL, causes the domains like SOMESERVER to be accepted.
>
> Might it be useful to include in the official distribution?

Looks fine to me except that it's lacking the HTML documentation
patch.

  http://weitz.de/patches.html

Could you add this and send a diff against the current dev version?

  http://bknr.net/trac/browser/trunk/thirdparty/drakma

Thanks a lot,
Edi.


From daniel at sentivision.com  Wed Jul 16 09:11:32 2008
From: daniel at sentivision.com (Daniel Janus)
Date: Wed, 16 Jul 2008 11:11:32 +0200
Subject: [drakma-devel] dotless cookie domains
In-Reply-To: <uod4ye17l.fsf@agharta.de>
References: <1216052800.10287.19.camel@daniel>  <uod4ye17l.fsf@agharta.de>
Message-ID: <1216199492.29287.5.camel@daniel>

Dnia 2008-07-16, ?ro o godzinie 09:29 +0200, Edi Weitz pisze:

> Looks fine to me except that it's lacking the HTML documentation
> patch.
> 
>   http://weitz.de/patches.html

Whoops, I'm sorry.  I omitted it, thinking that DOCUMENTATION-TEMPLATE
would somehow automatically include the doc for new variable.  Do you
maintain the Drakma doc by hand?

In any case, the attached patch is a second attempt.  I changed the
docstring of *ALLOW-DOTLESS-COOKIE-DOMAINS-P* to be a little more
elaborate and moved its definition to specials.lisp.  Is that OK?

-- 
?Daniel Janus     <daniel at sentivision.com>
Quality Assurance Specialist, SENTIVISION
Mb: +48 501 180 396 | Ph: +48 22 640 0860
-------------- next part --------------
A non-text attachment was scrubbed...
Name: drakma.patch
Type: text/x-patch
Size: 3243 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20080716/52580843/attachment.bin>

From edi at agharta.de  Wed Jul 16 09:14:52 2008
From: edi at agharta.de (Edi Weitz)
Date: Wed, 16 Jul 2008 11:14:52 +0200
Subject: [drakma-devel] dotless cookie domains
In-Reply-To: <1216199492.29287.5.camel@daniel> (Daniel Janus's message of
	"Wed, 16 Jul 2008 11:11:32 +0200")
References: <1216052800.10287.19.camel@daniel> <uod4ye17l.fsf@agharta.de>
	<1216199492.29287.5.camel@daniel>
Message-ID: <uhcaqchrn.fsf@agharta.de>

On Wed, 16 Jul 2008 11:11:32 +0200, Daniel Janus <daniel at sentivision.com> wrote:

> Whoops, I'm sorry.  I omitted it, thinking that
> DOCUMENTATION-TEMPLATE would somehow automatically include the doc
> for new variable.  Do you maintain the Drakma doc by hand?

Yes, unfortunately.  DOCUMENTATION-TEMPLATE is currently only used to
generate the initial page, but updates are done manually.  I'm working
on automating this process, but it's one of my projects that's laying
dormant due to lack of time... :(

> In any case, the attached patch is a second attempt.  I changed the
> docstring of *ALLOW-DOTLESS-COOKIE-DOMAINS-P* to be a little more
> elaborate and moved its definition to specials.lisp.  Is that OK?

Thanks, I'll add this to the dev version.