[drakma-devel] Clozure CL Page Specific Error

[Edi Weitz]

On Mon, 9 Jun 2008 22:20:50 -0600, "Wade Humeniuk" <wade.humeniuk at gmail.com> wrote:

> I am having trouble with Clozure CL.  I am not sure this is the
> right mailing list as it seems to be a CCL problem (unexpected
> end-of-file). This same request works with LW 5.0.1 and SBCL 1.0.15
> on the same machine.  CCL also works with other urls like "
> https://bmo.com".  The failing url is being served by an apache web
> server.  I have noticed similiar problems have been posted to this
> mailing list.

I can confirm that this fails for me with ClozureCL and works with
LispWorks.  A short analysis of what I think happens:

1. We're reading from a text stream with UTF-8 encoding.

2. The server tells us the length of the body is 2353 octets.

3. Drakma creates a string of length 2353.  With UTF-8, the string
   could be shorter, but you can't be sure before you've read it.

4. It then calls READ-SEQUENCE on this string which is implemented by
   FLEXI-STREAMS.

5. FLEXI-STREAMS, knowing that this is UTF-8 and /not/ knowing that
   there can be at most 2353 octets (2353 /characters/ are asked for)
   tries to read a bit more.  It tries to read 2355 octets from the
   underlying binary stream.

6. The underlying stream is a Chunga stream, but as there's no chunked
   encoding involved, READ-SEQUENCE is directly invoked on the SSL
   stream.

7. That's where the error happens.

My understanding is that if you're trying to READ-SEQUENCE 2355 octets
and there's an EOF after 2353 octets, then READ-SEQUENCE should read
these 2353 octets and return the position as described by the ANSI
standard.  I don't think it is correct that ClozureCL signals an error
here.  You should probably ask on their mailing list.  (Does this also
happen with HTTP instead of HTTPS?  Maybe it's a problem with the
CL+SSL library?)

A workaround would be to send the request with :FORCE-BINARY T and to
convert the content yourself which is admittedly a kludge.

Having said that, this is actually due to the way READ-SEQUENCE is
implemented in newer versions of FLEXI-STREAMS.  Now that
FLEXI-STREAMS has an efficient version of OCTETS-TO-STRING, I'm
wondering if it would make sense if Drakma always read the sequence as
binary data and then called OCTETS-TO-STRING on the result if needed.
Comments?

Cheers,
Edi.