From philogb at gmail.com  Thu Jun  5 00:43:48 2008
From: philogb at gmail.com (Nico Garcia Belmonte)
Date: Thu, 5 Jun 2008 02:43:48 +0200
Subject: [drakma-devel] Handling chunked content
Message-ID: <4e2f5dc00806041743p7b2e6d29kc36dd8e84d4630ff@mail.gmail.com>

Hi,

...I'm kind of a newbie, so please forgive me if I make stupid questions.
I've been using drakma to request html pages and everything seems to work
fine.

I'm now using drakma to make a request to the new youtube json api, and I
get chunked content:

(http-request "
http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
")

...chunked content...

((:CONTENT-TYPE . "application/json; charset=UTF-8")
 (:CACHE-CONTROL . "max-age=0, must-revalidate, no-transform, private")
 (:GDATA-VERSION . "1.0") (:LAST-MODIFIED . "Thu, 05 Jun 2008 00:20:27 GMT")
 (:TRANSFER-ENCODING . "chunked") (:DATE . "Thu, 05 Jun 2008 00:20:27 GMT")
 (:SERVER . "GFE/1.3") (:CONNECTION . "Close"))
#<PURI:URI
http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
>
#<FLEXI-STREAMS:FLEXI-IO-STREAM {B90E249}>

The thing is that I need it to be a string in order to decode to json format
using the cl-json lisp package.
I tried changing the external-output parameter without any good results.

Do you have some advice on how I could serialize this content?

Thanks at advanced!

Your library is great!

Nico.

-- 
I would never die for my beliefs because I might be wrong.

Bertrand Russell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20080605/47a1bee6/attachment.html>

From edi at agharta.de  Thu Jun  5 01:26:45 2008
From: edi at agharta.de (Edi Weitz)
Date: Wed, 04 Jun 2008 21:26:45 -0400
Subject: [drakma-devel] Handling chunked content
In-Reply-To: <4e2f5dc00806041743p7b2e6d29kc36dd8e84d4630ff@mail.gmail.com>
	(Nico
	Garcia Belmonte's message of "Thu, 5 Jun 2008 02:43:48 +0200")
References: <4e2f5dc00806041743p7b2e6d29kc36dd8e84d4630ff@mail.gmail.com>
Message-ID: <u63sowu22.fsf@agharta.de>

On Thu, 5 Jun 2008 02:43:48 +0200, "Nico Garcia Belmonte" <philogb at gmail.com> wrote:

> ...I'm kind of a newbie, so please forgive me if I make stupid
> questions.  I've been using drakma to request html pages and
> everything seems to work fine.
>
> I'm now using drakma to make a request to the new youtube json api,
> and I get chunked content:
>
> (http-request "
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
> ")
>
> ...chunked content...
>
> ((:CONTENT-TYPE . "application/json; charset=UTF-8")
>  (:CACHE-CONTROL . "max-age=0, must-revalidate, no-transform, private")
>  (:GDATA-VERSION . "1.0") (:LAST-MODIFIED . "Thu, 05 Jun 2008 00:20:27 GMT")
>  (:TRANSFER-ENCODING . "chunked") (:DATE . "Thu, 05 Jun 2008 00:20:27 GMT")
>  (:SERVER . "GFE/1.3") (:CONNECTION . "Close"))
> #<PURI:URI
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
>>
> #<FLEXI-STREAMS:FLEXI-IO-STREAM {B90E249}>
>
> The thing is that I need it to be a string in order to decode to
> json format using the cl-json lisp package.  I tried changing the
> external-output parameter without any good results.
>
> Do you have some advice on how I could serialize this content?

Your problem is not chunked content (Drakma takes care of that
automatically) but that the server returns a content type which is by
default not recognized as text.  See for example here:

  http://weitz.de/drakma/#*text-content-types*
  http://weitz.de/drakma/#*body-format-function*

In your particular case, something like this will probably do the
trick:

  (let ((*text-content-types* (cons '("application" . "json")
                                    *text-content-types*)))
    (http-request "http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5"))

Cheers,
Edi.


From philogb at gmail.com  Thu Jun  5 17:28:43 2008
From: philogb at gmail.com (Nico Garcia Belmonte)
Date: Thu, 5 Jun 2008 19:28:43 +0200
Subject: [drakma-devel] Re: drakma-devel Digest, Vol 20, Issue 1
In-Reply-To: <20080605160132.290442804F@common-lisp.net>
References: <20080605160132.290442804F@common-lisp.net>
Message-ID: <4e2f5dc00806051028q2e0e389br391bcc5beccffe62@mail.gmail.com>

It works!

Thanks a lot!

On 6/5/08, drakma-devel-request at common-lisp.net <
drakma-devel-request at common-lisp.net> wrote:
>
> Send drakma-devel mailing list submissions to
>         drakma-devel at common-lisp.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel
> or, via email, send a message with subject or body 'help' to
>         drakma-devel-request at common-lisp.net
>
> You can reach the person managing the list at
>         drakma-devel-owner at common-lisp.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of drakma-devel digest..."
>
>
> Today's Topics:
>
>    1. Handling chunked content (Nico Garcia Belmonte)
>    2. Re: Handling chunked content (Edi Weitz)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 5 Jun 2008 02:43:48 +0200
> From: "Nico Garcia Belmonte" <philogb at gmail.com>
> Subject: [drakma-devel] Handling chunked content
> To: drakma-devel at common-lisp.net
> Message-ID:
>         <4e2f5dc00806041743p7b2e6d29kc36dd8e84d4630ff at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi,
>
> ...I'm kind of a newbie, so please forgive me if I make stupid questions.
> I've been using drakma to request html pages and everything seems to work
> fine.
>
> I'm now using drakma to make a request to the new youtube json api, and I
> get chunked content:
>
> (http-request "
>
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
> ")
>
> ...chunked content...
>
> ((:CONTENT-TYPE . "application/json; charset=UTF-8")
>   (:CACHE-CONTROL . "max-age=0, must-revalidate, no-transform, private")
>   (:GDATA-VERSION . "1.0") (:LAST-MODIFIED . "Thu, 05 Jun 2008 00:20:27
> GMT")
>   (:TRANSFER-ENCODING . "chunked") (:DATE . "Thu, 05 Jun 2008 00:20:27
> GMT")
>   (:SERVER . "GFE/1.3") (:CONNECTION . "Close"))
> #<PURI:URI
>
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
> >
> #<FLEXI-STREAMS:FLEXI-IO-STREAM {B90E249}>
>
> The thing is that I need it to be a string in order to decode to json
> format
> using the cl-json lisp package.
> I tried changing the external-output parameter without any good results.
>
> Do you have some advice on how I could serialize this content?
>
> Thanks at advanced!
>
> Your library is great!
>
> Nico.
>
> --
> I would never die for my beliefs because I might be wrong.
>
> Bertrand Russell
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://common-lisp.net/pipermail/drakma-devel/attachments/20080605/47a1bee6/attachment.htm
>
> ------------------------------
>
> Message: 2
> Date: Wed, 04 Jun 2008 21:26:45 -0400
> From: Edi Weitz <edi at agharta.de>
> Subject: Re: [drakma-devel] Handling chunked content
> To: General interest list for Drakma and Chunga
>         <drakma-devel at common-lisp.net>
> Message-ID: <u63sowu22.fsf at agharta.de>
> Content-Type: text/plain; charset=us-ascii
>
> On Thu, 5 Jun 2008 02:43:48 +0200, "Nico Garcia Belmonte" <
> philogb at gmail.com> wrote:
>
> > ...I'm kind of a newbie, so please forgive me if I make stupid
> > questions.  I've been using drakma to request html pages and
> > everything seems to work fine.
> >
> > I'm now using drakma to make a request to the new youtube json api,
> > and I get chunked content:
> >
> > (http-request "
> >
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
> > ")
> >
> > ...chunked content...
> >
> > ((:CONTENT-TYPE . "application/json; charset=UTF-8")
> >  (:CACHE-CONTROL . "max-age=0, must-revalidate, no-transform, private")
> >  (:GDATA-VERSION . "1.0") (:LAST-MODIFIED . "Thu, 05 Jun 2008 00:20:27
> GMT")
> >  (:TRANSFER-ENCODING . "chunked") (:DATE . "Thu, 05 Jun 2008 00:20:27
> GMT")
> >  (:SERVER . "GFE/1.3") (:CONNECTION . "Close"))
> > #<PURI:URI
> >
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
> >>
> > #<FLEXI-STREAMS:FLEXI-IO-STREAM {B90E249}>
> >
> > The thing is that I need it to be a string in order to decode to
> > json format using the cl-json lisp package.  I tried changing the
> > external-output parameter without any good results.
> >
> > Do you have some advice on how I could serialize this content?
>
> Your problem is not chunked content (Drakma takes care of that
> automatically) but that the server returns a content type which is by
> default not recognized as text.  See for example here:
>
>   http://weitz.de/drakma/#*text-content-types*
>   http://weitz.de/drakma/#*body-format-function*
>
> In your particular case, something like this will probably do the
> trick:
>
>   (let ((*text-content-types* (cons '("application" . "json")
>                                     *text-content-types*)))
>     (http-request "
> http://gdata.youtube.com/feeds/api/videos/-/Music/?max-results=10&vq=bush&alt=json&format=5
> "))
>
> Cheers,
> Edi.
>
>
> ------------------------------
>
> _______________________________________________
> drakma-devel mailing list
> drakma-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel
>
>
> End of drakma-devel Digest, Vol 20, Issue 1
> *******************************************
>



-- 
I would never die for my beliefs because I might be wrong.

Bertrand Russell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20080605/5a6e0ffd/attachment.html>

From wade.humeniuk at gmail.com  Tue Jun 10 04:20:50 2008
From: wade.humeniuk at gmail.com (Wade Humeniuk)
Date: Mon, 9 Jun 2008 22:20:50 -0600
Subject: [drakma-devel] Clozure CL Page Specific Error
Message-ID: <4a8bfaea0806092120x7fe6fcd1g5227d5f8b06341c7@mail.gmail.com>

I am having trouble with Clozure CL.  I am not sure this is the right
mailing list as it
seems to be a CCL problem (unexpected end-of-file). This same request works
with LW 5.0.1
and SBCL 1.0.15 on the same machine.  CCL also works with other urls like "
https://bmo.com".
The failing url is being served by an apache web server.  I have noticed
similiar problems have
been posted to this mailing list.  (setting chunga:*accept-bogus-eols* does
not help).

CL-USER> (lisp-implementation-version)
"Version 1.2-r9226-RC1  (LinuxX8664)"
CL-USER>
CL-USER> (drakma:http-request "https://esqa.moneris.com")

Unexpected end of file on #<SSL-STREAM for 4>
   [Condition of type END-OF-FILE]

Restarts:
 0: [ABORT] Return to SLIME's top level.
 1: [ABORT-BREAK] Reset this process
 2: [ABORT] Kill this process

Backtrace:
  0: (READ-BYTE #<SSL-STREAM for 4> 'T 'NIL)
  1: (#<CCL::STANDARD-KERNEL-METHOD STREAM-READ-VECTOR
(CCL::BINARY-INPUT-STREAM
                                                       T
                                                       T
                                                       T)> #<SSL-STREAM for
4> #(60 72 84 77 76 62 10 60 72 69 ...) 0 2355)
  2: (READ-SEQUENCE #(60 72 84 77 76 62 10 60 72 69 ...) #<SSL-STREAM for 4>
':START 0 ':END 2355)
  3: (READ-SEQUENCE #(60 72 84 77 76 62 10 60 72 69 ...)
#<CHUNGA:CHUNKED-IO-STREAM #x30004141A8FD> ':START 0 ':END 2355)
  4: ((:INTERNAL FLEXI-STREAMS::FILL-BUFFER
      (TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE
        (FLEXI-STREAMS:FLEXI-INPUT-STREAM T T T))) 2355)
  5: (#<STANDARD-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE
(FLEXI-STREAMS:FLEXI-INPUT-STREAM
                                                                  T
                                                                  T
                                                                  T)>
#<FLEXI-STREAMS:FLEXI-IO-STREAM #x30004129C14D>
  6: (CCL::%%STANDARD-COMBINED-METHOD-DCODE '((#<STANDARD-METHOD
TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE :BEFORE
(FLEXI-STREAMS:FLEXI-IO-STREAM

T

T

T)>)
       NIL
       #<STANDARD-METHOD TRIVIAL-GRAY-STREAMS:STREAM-READ-SEQUENCE
(FLEXI-STREAMS:FLEXI-INPUT-STREAM
                                                                    T
                                                                    T
                                                                    T)>)
17575205018776)
  7: (READ-SEQUENCE <edited>
  8: (DRAKMA::READ-BODY #<FLEXI-STREAMS:FLEXI-IO-STREAM #x30004129C14D>
'((:DATE . "Tue, 10 Jun 2008 04:15:25 GMT") (:SERVER . "Apache")
       (:LAST-MODIFIED . "Thu, 08 Nov 2007 15:41:34 GMT")
       (:ETAG . "\"42951-931-b3656780\"") (:ACCEPT-RANGES . "bytes")
       (:CONTENT-LENGTH . "2353") (:CONNECTION . "close")
       (:CONTENT-TYPE . "text/html; charset=UTF-8")) 'T #<FLEXI-UTF-8-FORMAT
(:UTF-8 :EOL-STYLE :LF) #x30004129BCCD>)
  9: ((:INTERNAL DRAKMA::FINISH-REQUEST DRAKMA:HTTP-REQUEST) 'NIL 'NIL)
 10: (DRAKMA:HTTP-REQUEST #<URI https://esqa.moneris.com>)
 11: (CCL::CALL-CHECK-REGS 'DRAKMA:HTTP-REQUEST "https://esqa.moneris.com")
 12: (CCL::RUN-PROCESS-INITIAL-FORM #<PROCESS worker(7) [Active]
#x30004130E5FD> '(#<COMPILED-LEXICAL-CLOSURE (:INTERNAL
CCL::%PROCESS-RUN-FUNCTION) #x30004130E48F>))
 13: ((:INTERNAL CCL::%PROCESS-PRESET-INTERNAL) #<PROCESS worker(7) [Active]
#x30004130E5FD> '(#<COMPILED-LEXICAL-CLOSURE (:INTERNAL
CCL::%PROCESS-RUN-FUNCTION) #x30004130E48F>))
 14: ((:INTERNAL CCL::THREAD-MAKE-STARTUP-FUNCTION))
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20080609/e943625e/attachment.html>

From edi at agharta.de  Tue Jun 10 06:52:46 2008
From: edi at agharta.de (Edi Weitz)
Date: Tue, 10 Jun 2008 08:52:46 +0200
Subject: [drakma-devel] Clozure CL Page Specific Error
In-Reply-To: <4a8bfaea0806092120x7fe6fcd1g5227d5f8b06341c7@mail.gmail.com>
	(Wade Humeniuk's message of "Mon, 9 Jun 2008 22:20:50 -0600")
References: <4a8bfaea0806092120x7fe6fcd1g5227d5f8b06341c7@mail.gmail.com>
Message-ID: <u8wxdn5mp.fsf@agharta.de>

On Mon, 9 Jun 2008 22:20:50 -0600, "Wade Humeniuk" <wade.humeniuk at gmail.com> wrote:

> I am having trouble with Clozure CL.  I am not sure this is the
> right mailing list as it seems to be a CCL problem (unexpected
> end-of-file). This same request works with LW 5.0.1 and SBCL 1.0.15
> on the same machine.  CCL also works with other urls like "
> https://bmo.com".  The failing url is being served by an apache web
> server.  I have noticed similiar problems have been posted to this
> mailing list.

I can confirm that this fails for me with ClozureCL and works with
LispWorks.  A short analysis of what I think happens:

1. We're reading from a text stream with UTF-8 encoding.

2. The server tells us the length of the body is 2353 octets.

3. Drakma creates a string of length 2353.  With UTF-8, the string
   could be shorter, but you can't be sure before you've read it.

4. It then calls READ-SEQUENCE on this string which is implemented by
   FLEXI-STREAMS.

5. FLEXI-STREAMS, knowing that this is UTF-8 and /not/ knowing that
   there can be at most 2353 octets (2353 /characters/ are asked for)
   tries to read a bit more.  It tries to read 2355 octets from the
   underlying binary stream.

6. The underlying stream is a Chunga stream, but as there's no chunked
   encoding involved, READ-SEQUENCE is directly invoked on the SSL
   stream.

7. That's where the error happens.

My understanding is that if you're trying to READ-SEQUENCE 2355 octets
and there's an EOF after 2353 octets, then READ-SEQUENCE should read
these 2353 octets and return the position as described by the ANSI
standard.  I don't think it is correct that ClozureCL signals an error
here.  You should probably ask on their mailing list.  (Does this also
happen with HTTP instead of HTTPS?  Maybe it's a problem with the
CL+SSL library?)

A workaround would be to send the request with :FORCE-BINARY T and to
convert the content yourself which is admittedly a kludge.

Having said that, this is actually due to the way READ-SEQUENCE is
implemented in newer versions of FLEXI-STREAMS.  Now that
FLEXI-STREAMS has an efficient version of OCTETS-TO-STRING, I'm
wondering if it would make sense if Drakma always read the sequence as
binary data and then called OCTETS-TO-STRING on the result if needed.
Comments?

Cheers,
Edi.


From wade.humeniuk at gmail.com  Wed Jun 11 00:07:31 2008
From: wade.humeniuk at gmail.com (Wade Humeniuk)
Date: Tue, 10 Jun 2008 18:07:31 -0600
Subject: [drakma-devel] Clozure CL Page Specific Error
In-Reply-To: <u8wxdn5mp.fsf@agharta.de>
References: <4a8bfaea0806092120x7fe6fcd1g5227d5f8b06341c7@mail.gmail.com>
	<u8wxdn5mp.fsf@agharta.de>
Message-ID: <4a8bfaea0806101707t56080148s294c6bc57a2d969d@mail.gmail.com>

After reading the Hyperspec I do think it is problem with CCL's read-sequence.
I modifed CCL's source from (the code looks obviously in error)

(defmethod stream-read-vector ((stream binary-input-stream)
			       vector start end)
  (declare (fixnum start end))
  (do* ((i start (1+ i)))
       ((= i end) end)
    (declare (fixnum i))
    (let* ((b (read-byte stream))
      (if (eq b :eof)
	(return i)
	(setf (uvref vector i) b)))))

to

(defmethod stream-read-vector ((stream binary-input-stream)
			       vector start end)
  (declare (fixnum start end))
  (do* ((i start (1+ i)))
       ((= i end) end)
    (declare (fixnum i))
    (let* ((b (read-byte stream nil :eof)))
      (if (eq b :eof)
	(return i)
	(setf (uvref vector i) b)))))

and it now works for "https://esqa.moneris.com"

Just for your info Drakma is being used for Moneris credit card purchase
validatons.

Example:

CL-USER> (moneris-test::test-purchase)
(:RECEIPT (:RECEIPTID "ORDER-3422131578-72") (:REFERENCENUM
"660021630017911950") (:RESPONSECODE "027") (:ISO "01") (:AUTHCODE
"337410") (:TRANSTIME "20:06:19") (:TRANSDATE "2008-06-10")
(:TRANSTYPE "00") (:COMPLETE T) (:MESSAGE "APPROVED           *
            =") (:TRANSAMOUNT "10.00") (:CARDTYPE "V") (:TRANSID
"420704-0_7") (:TIMEDOUT NIL) (:BANKTOTALS NIL) (:TICKET NIL))
27
"APPROVED           *                    ="
"420704-0_7"
"<?xml version=\"1.0\"
standalone=\"yes\"?><response><receipt><ReceiptId>ORDER-3422131578-72</ReceiptId><ReferenceNum>660021630017911950</ReferenceNum><ResponseCode>027</ResponseCode><ISO>01</ISO><AuthCode>337410</AuthCode><TransTime>20:06:19</TransTime><TransDate>2008-06-10</TransDate><TransType>00</TransType><Complete>true</Complete><Message>APPROVED
          *
=</Message><TransAmount>10.00</TransAmount><CardType>V</CardType><TransID>420704-0_7</TransID><TimedOut>false</TimedOut><BankTotals>null</BankTotals><Ticket>null</Ticket></receipt></response>"
CL-USER>

I will submit the info to the  ClozureCL devel list.

Thanks, Wade


From wade.humeniuk at gmail.com  Thu Jun 12 01:51:10 2008
From: wade.humeniuk at gmail.com (Wade Humeniuk)
Date: Wed, 11 Jun 2008 19:51:10 -0600
Subject: [drakma-devel] Clozure CL Page Specific Error
In-Reply-To: <u8wxdn5mp.fsf@agharta.de>
References: <4a8bfaea0806092120x7fe6fcd1g5227d5f8b06341c7@mail.gmail.com>
	<u8wxdn5mp.fsf@agharta.de>
Message-ID: <4a8bfaea0806111851h1f328fcl1ffc30acaf29d9b4@mail.gmail.com>

Gary Byers at Clozure has made the fix (as well as a couple of others).
You can update with svn update to at least r9730.

Wade

 Clozure Common Lisp Version 1.2-r9736M-RC1  (LinuxX8664)