From victor.kryukov at gmail.com Sun Mar 16 03:59:09 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Sat, 15 Mar 2008 22:59:09 -0500 Subject: [drakma-devel] Parsing quoted cookie values (bug?) Message-ID: <87zlsztj3m.fsf@esculap.gateway.2wire.net> Hello, I believe the following behaviour is a bug: (drakma::parse-set-cookie "session=\"1,2,3\"; domain=example.com; path=/") generates an error: While parsing cookie header "session=\"1,2,3\"; domain=example.com; path=/": Read character #\,, but expected #\=. [Condition of type SIMPLE-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (CHUNGA::SIGNAL-UNEXPECTED-CHARS #\, #\=) 1: (CHUNGA:ASSERT-CHAR # #\=) 2: (CHUNGA:READ-NAME-VALUE-PAIR #)[:EXTERNAL] 3: (DRAKMA::PARSE-SET-COOKIE "session=\"1,2,3\"; domain=example.com; path=/") According to RFC2965[1], av-pairs = av-pair *(";" av-pair) av-pair = attr ["=" value] ; optional value attr = token value = token | quoted-string value is either a token or a quoted string. According to RFC2616[2], quoted-string = ( <"> *(qdtext | quoted-pair ) <"> ) qdtext = > quoted string can contain any text except \", including commas. I know a particular web-site (reddit.com) that sets cookies of that form, and Firefox parses them without any problem. The problem seem to be in read-name-value-pair from CHUNGA: DRAKMA> (with-input-from-string (in "session=\"1,2,3\"; domain=example.com; path=/") (read-name-value-pair in :cookie-syntax t)) ("session" . "\"1") Best Regards, Victor. [1] http://www.faqs.org/rfcs/rfc2965.html [2] http://www.faqs.org/rfcs/rfc2616.html -- http://macrodefinition.blogspot.com From edi at agharta.de Mon Mar 17 11:36:06 2008 From: edi at agharta.de (Edi Weitz) Date: Mon, 17 Mar 2008 12:36:06 +0100 Subject: [drakma-devel] Parsing quoted cookie values (bug?) In-Reply-To: <87zlsztj3m.fsf@esculap.gateway.2wire.net> (Victor Kryukov's message of "Sat, 15 Mar 2008 22:59:09 -0500") References: <87zlsztj3m.fsf@esculap.gateway.2wire.net> Message-ID: On Sat, 15 Mar 2008 22:59:09 -0500, Victor Kryukov wrote: > According to RFC2965[1] Unfortunately, it has been my experience that you can't rely on RFCs when parsing cookies as there are too many deviations out there. Having said that, I'd be happy to accept a patch which fixed this particular issue without breaking "compatibility" with other widespread ways of setting cookies. Edi. From victor.kryukov at gmail.com Mon Mar 17 23:04:18 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Mon, 17 Mar 2008 18:04:18 -0500 Subject: [drakma-devel] Re: Parsing quoted cookie values (bug?) References: <87zlsztj3m.fsf@esculap.gateway.2wire.net> Message-ID: Edi Weitz writes: > On Sat, 15 Mar 2008 22:59:09 -0500, Victor Kryukov wrote: > >> According to RFC2965[1] > > Unfortunately, it has been my experience that you can't rely on RFCs > when parsing cookies as there are too many deviations out there. First of all, let me correct myself: not only RFC 2965 (which is a new standard), but also RFC 2109 (which is the old standard for cookies that seem to be widespreadly used) requires cookie values to be either terms or quoted strings. I understand why you may want to be less strict about following the standard and allow some deviations from it, but current DRAKMA behaviour is broken for the sites that _do follow the standard_, which I think is not acceptable, as least not as a default behaviour. As a reference point, I've checked that python >= 2.4 follows RFC 2109 more or less (you can check lib/Cookie.py, _CookiePattern is their regexp for parsing name/value pairs), and Perl also parses quoted cookie values without any problems (I've used WWW::Mechanize which in turn uses HTTP::Cookie for cookie handling). > Having said that, I'd be happy to accept a patch which fixed this > particular issue without breaking "compatibility" with other > widespread ways of setting cookies. I'd be happy to provide a patch once I better understand what are the "compatibility" requirements. Could you please point me in the right direction? Do you mean old-style Netscape cookies[1]? Also, there are mutliple strategies to fix that. One is to follow RFC2109 by default, but allow for Netscape-style cookies after setting a certain parameter (I'd prefer this one). Another is to try to follow RFC 2109, but then rollback to Netscape-style cookies if parsing error occurs. There are probably other possible approaches. Which one would you prefer? Regards, Victor. [1] http://wp.netscape.com/newsref/std/cookie_spec.html -- http://macrodefinition.blogspot.com From edi at agharta.de Mon Mar 17 23:24:19 2008 From: edi at agharta.de (Edi Weitz) Date: Tue, 18 Mar 2008 00:24:19 +0100 Subject: [drakma-devel] Re: Parsing quoted cookie values (bug?) In-Reply-To: (Victor Kryukov's message of "Mon, 17 Mar 2008 18:04:18 -0500") References: <87zlsztj3m.fsf@esculap.gateway.2wire.net> Message-ID: On Mon, 17 Mar 2008 18:04:18 -0500, Victor Kryukov wrote: > I understand why you may want to be less strict about following the > standard and allow some deviations from it, but current DRAKMA > behaviour is broken for the sites that _do follow the standard_, > which I think is not acceptable, as least not as a default > behaviour. Agreed. > I'd be happy to provide a patch once I better understand what are > the "compatibility" requirements. Could you please point me in the > right direction? I wanted to provide some examples this morning already, but unfortunately, I can't come up with any right now. I /do/ know that at one time I tested a lot of "real-life" cookies with Drakma, though, and quite a few weren't RFC-compliant. > Do you mean old-style Netscape cookies[1]? Allowing old-style Netscape cookies - at least optionally - would be a good idea, yes. > Also, there are mutliple strategies to fix that. One is to follow > RFC2109 by default, but allow for Netscape-style cookies after > setting a certain parameter (I'd prefer this one). Another is to try > to follow RFC 2109, but then rollback to Netscape-style cookies if > parsing error occurs. There are probably other possible approaches. > Which one would you prefer? The ideal solution, I think, would be a combination of the first two strategies you mentioned. Follow RFC2109 by default, but provide one or more sensible restarts in case a parsing error occurs. Additionally, have some parameter (probably a global special variable) which if set automatically invokes the "use old Netscape-style" restart. Does that sound reasonable? From victor.kryukov at gmail.com Tue Mar 18 00:35:55 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Mon, 17 Mar 2008 19:35:55 -0500 Subject: [drakma-devel] Re: Parsing quoted cookie values (bug?) References: <87zlsztj3m.fsf@esculap.gateway.2wire.net> Message-ID: Edi Weitz writes: > On Mon, 17 Mar 2008 18:04:18 -0500, Victor Kryukov wrote: > >> I understand why you may want to be less strict about following the >> standard and allow some deviations from it, but current DRAKMA >> behaviour is broken for the sites that _do follow the standard_, >> which I think is not acceptable, as least not as a default >> behaviour. > > Agreed. > >> Do you mean old-style Netscape cookies[1]? > > Allowing old-style Netscape cookies - at least optionally - would be a > good idea, yes. It seems to me that this is exactly what read-cookie-value is doing. So we just need to wrap it up nicely. >> Also, there are mutliple strategies to fix that. One is to follow >> RFC2109 by default, but allow for Netscape-style cookies after >> setting a certain parameter (I'd prefer this one). Another is to try >> to follow RFC 2109, but then rollback to Netscape-style cookies if >> parsing error occurs. There are probably other possible approaches. >> Which one would you prefer? > > The ideal solution, I think, would be a combination of the first two > strategies you mentioned. Follow RFC2109 by default, but provide one > or more sensible restarts in case a parsing error occurs. > Additionally, have some parameter (probably a global special variable) > which if set automatically invokes the "use old Netscape-style" > restart. > > Does that sound reasonable? Yes, that must be the optimal solution. I'll try to implement it as a patch soon, most likely early next week. Best Regards, Victor -- http://macrodefinition.blogspot.com From avodonosov at yandex.ru Fri Mar 21 22:12:26 2008 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sat, 22 Mar 2008 00:12:26 +0200 Subject: [drakma-devel] CLISP problem with READ-SEQUENCE; patch Message-ID: <1778416840.20080322001226@yandex.ru> Hello. When using HTTP-REQUEST for non text content types, we have following error in CLISP: SYSTEM::STORE: #\< does not fit into #(0 0 0 0 0 0 0 0 0 0 ...), bad type Backtrace: 0: INVOKE-DEBUGGER 1: SYSTEM::STORE 2: EXT:READ-CHAR-SEQUENCE 3: READ-SEQUENCE 4: DRAKMA::READ-BODY 5: DRAKMA::HTTP-REQUEST-FINISH-REQUEST 6: unwind-protect frame 7: compiled block frame for HTTP-REQUEST 8: HTTP-REQUEST This is because CLISP doesn't support READ-SEQUENCE on a stream having STREAM-ELEMENT-TYPE == 'CHARACTER. The patch attached fixes this problem by explicit change of stream element type in the DRAKMA::READ-BODY function. Test URLs: http://www.google.com/calendar/feeds/pm55j8kg30dnm54ib2if9fuocc at group.calendar.google.com/public/basic http://www.lisperati.com/lisplogo_alien_128.png Best regards, -- Anton -------------- next part -------------- A non-text attachment was scrubbed... Name: clisp-read-sequence-patch.diff Type: application/octet-stream Size: 1172 bytes Desc: not available URL: From edi at agharta.de Fri Mar 21 22:42:37 2008 From: edi at agharta.de (Edi Weitz) Date: Fri, 21 Mar 2008 23:42:37 +0100 Subject: [drakma-devel] New release 0.11.5 (Was: CLISP problem with READ-SEQUENCE; patch) In-Reply-To: <1778416840.20080322001226@yandex.ru> (Anton Vodonosov's message of "Sat, 22 Mar 2008 00:12:26 +0200") References: <1778416840.20080322001226@yandex.ru> Message-ID: On Sat, 22 Mar 2008 00:12:26 +0200, Anton Vodonosov wrote: > This is because CLISP doesn't support READ-SEQUENCE on a stream > having STREAM-ELEMENT-TYPE == 'CHARACTER. Isn't that a bug which should be reported to the CLISP maintainers? > The patch attached fixes this problem by explicit change of stream > element type in the DRAKMA::READ-BODY function. Thanks. Applied. From avodonosov at yandex.ru Fri Mar 21 23:31:55 2008 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sat, 22 Mar 2008 01:31:55 +0200 Subject: [drakma-devel] Re: New release 0.11.5 (Was: CLISP problem with READ-SEQUENCE; patch) In-Reply-To: References: <1778416840.20080322001226@yandex.ru> Message-ID: <331559694.20080322013155@yandex.ru> On Sat, 22 Mar 2008 0:42:37 Edi Weitz wrote: > On Sat, 22 Mar 2008 00:12:26 +0200, Anton Vodonosov wrote: >> This is because CLISP doesn't support READ-SEQUENCE on a stream >> having STREAM-ELEMENT-TYPE == 'CHARACTER. > Isn't that a bug which should be reported to the CLISP maintainers? Strictly speaking it's not a bug. CLHS even says that READ-SEQUENCE "Might signal an error of type type-error if an element read from the stream is not a member of the element type of the sequence." For WRITE-STREAM event worse, it explicitly mention stream element type: "Might signal an error of type type-error if an element of the bounded sequence is not a member of the stream element type of the stream." Drakma relies on FLEXI-STREAMS in hope that FLEXI-STREAMS:STREAM-READ-SEQUENCE will handle it gracefully, but the problem is that it is not called at all. Function STREAM-READ-SEQUENCE is not included into Gray streams proposal (I guess because READ-SEQUENCE itself wasn't part of the standard draft at the time when the proposal was made, at least it's absent in CLTL2). trivial-gray-streams tries to emulate STREAM-READ-SEQUENCE based on more limited CLISP's STREAM-READ-BYTE-SEQUENCE, but this doesn't work in all situations. >From my personal point of view it would be better to just implement generic STREAM-READ-SEQUENCE in CLISP, as it is done in most of other CL implementations; I event suggested a patch (which is trivial). But CLISP maintainers weren't too enthusiastic about that. Maybe I must pressurize them, to the coming generations good. Regards, -- Anton From edi at agharta.de Sat Mar 22 00:45:47 2008 From: edi at agharta.de (Edi Weitz) Date: Sat, 22 Mar 2008 01:45:47 +0100 Subject: [drakma-devel] Re: New release 0.11.5 In-Reply-To: <331559694.20080322013155@yandex.ru> (Anton Vodonosov's message of "Sat, 22 Mar 2008 01:31:55 +0200") References: <1778416840.20080322001226@yandex.ru> <331559694.20080322013155@yandex.ru> Message-ID: On Sat, 22 Mar 2008 01:31:55 +0200, Anton Vodonosov wrote: > From my personal point of view it would be better to just implement > generic STREAM-READ-SEQUENCE in CLISP, as it is done in most of > other CL implementations; Right, I agree. > I event suggested a patch (which is trivial). But CLISP maintainers > weren't too enthusiastic about that. Hmmm... From pinterface at gmail.com Sat Mar 22 03:02:32 2008 From: pinterface at gmail.com (Pixel // pinterface) Date: Sat, 22 Mar 2008 03:02:32 -0000 Subject: [drakma-devel] Re: CLISP problem with READ-SEQUENCE; patch References: <1778416840.20080322001226@yandex.ru> Message-ID: "Anton Vodonosov" wrote in message news:1778416840.20080322001226 at yandex.ru... > Hello. > > When using HTTP-REQUEST for non text content types, we > have following error in CLISP: > > SYSTEM::STORE: #\< does not fit into #(0 0 0 0 0 0 0 0 0 0 ...), bad type > > [snipped Backtrace] > > This is because CLISP doesn't support READ-SEQUENCE on > a stream having STREAM-ELEMENT-TYPE == 'CHARACTER. > > The patch attached fixes this problem by explicit change > of stream element type in the DRAKMA::READ-BODY function. That's odd, I've been using clisp + drakma for months on streams which are a mixture of ASCII and binary without problems. /me checks something. Aha! I took a somewhat longer approach and mucked around with system::read-sequence and system::write-sequence to prefer byte-sequences over character-sequences (see attached). Never made a patch to CLISP because I figured it wasn't the right solution so much as something that happens to work for my particular application. I like your way better! :) begin 666 clisp-fix.lisp M.SL at 26X@8VQI6]U(')E86QL>2!W86YT(&EN=&5G97)S+B!";&5H(2!&:7@@=&AI M2!S=V%P<&EN9PH[.R!S;VUE(&-O;F1I=&EO;G, at 87)O=6YD+B!%2P@2!W:&%T M('=E('=A;G1E9"!A;GEW87DN"CL[("A!(&QI='1L92!A<'!L:6-A=&EO;BUS M<&5C:69I8R!K;F]W;&5D9V4 at 9V]E7-T96TI"@HH97AT.G=I=&AO=70M<&%C:V%G92UL;V-K("@I"B @ M*&1E9G5N('-Y2 H7!E("AI9B H=F5C=&]R<"!S97%U96YC92D@*&%R7!E("=.24PI("AE<2!S96QT>7!E("=#2$%204-415(I"B @(" @ M(" @(" @(" @(" @*&5Q('9E;'1Y<&4@)T-(05)!0U1%4BDI"B @(" @(" @ M(" @(" H87!P;'D@(R=R96%D+6-H87(M2 H7!E("AS>7-T96TZ.G-T7!E('-T7!E("AI9B H=F5C=&]R<"!S M97%U96YC92D@*&%R7!E<"!V96QT>7!E("=)3E1%1T52*2D*(" @(" @(" @(" @.SL at 8'=R M:71E+6)Y=&4M2 C)W-Y7!E("=#2$%204-415(I*0H@(" @(" @(" @(" H M87!P;'D@(R=S>7-T96TZ.G=R:71E+6-H87(M71E 0+7-E<75E;F-E*2DI*2DI"@`` ` end From avodonosov at yandex.ru Sat Mar 22 03:36:00 2008 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sat, 22 Mar 2008 05:36:00 +0200 Subject: [drakma-devel] Re: CLISP problem with READ-SEQUENCE; patch In-Reply-To: References: <1778416840.20080322001226@yandex.ru> Message-ID: <724752539.20080322053600@yandex.ru> Pixel wrote: > Aha! I took a somewhat longer approach and mucked around with > system::read-sequence and system::write-sequence to prefer byte-sequences > over character-sequences (see attached). Good idea. I wasn't aware that it is possible to redefine system functions. At first sight this way it may be possible to fix trivial-gray-stream so that trivial-gray-streams:stream-read-sequence will always be called by cl:read-sequence and therefore eliminate all #+clisp in all trivial-gray-streams depended libraries. From avodonosov at yandex.ru Sun Mar 23 21:53:19 2008 From: avodonosov at yandex.ru (Anton Vodonosov) Date: Sun, 23 Mar 2008 23:53:19 +0200 Subject: [drakma-devel] Re: CLISP problem with READ-SEQUENCE; patch In-Reply-To: References: <1778416840.20080322001226@yandex.ru> Message-ID: <11110070519.20080323235319@yandex.ru> Hello. Good news, CLISP CVS now has GRAY:STREAM-READ-SEQUENCE and GRAY:STREAM-WRITE-SEQUENCE. http://sourceforge.net/mailarchive/forum.php?thread_name=47E6C701.6030004%40gnu.org&forum_name=clisp-devel Only small update of trivial-gray-streams remains to be done. -Anton From edi at agharta.de Sun Mar 23 22:03:08 2008 From: edi at agharta.de (Edi Weitz) Date: Sun, 23 Mar 2008 23:03:08 +0100 Subject: [drakma-devel] Re: CLISP problem with READ-SEQUENCE; patch In-Reply-To: <11110070519.20080323235319@yandex.ru> (Anton Vodonosov's message of "Sun, 23 Mar 2008 23:53:19 +0200") References: <1778416840.20080322001226@yandex.ru> <11110070519.20080323235319@yandex.ru> Message-ID: On Sun, 23 Mar 2008 23:53:19 +0200, Anton Vodonosov wrote: > and GRAY:STREAM-WRITE-SEQUENCE. Nice, thanks.