From ndj at hivsa.com  Wed Dec 12 11:34:14 2007
From: ndj at hivsa.com (Nico de Jager)
Date: Wed, 12 Dec 2007 13:34:14 +0200
Subject: [drakma-devel] CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE
Message-ID: <87ir346tah.fsf@davinci.hivsa.com>

Hi

I have a fingerprint scanning application, that uploads fingerprint images using Drakma to a Hunchentoot server. Everything works great as long as I don't use a secure connection. With https, I get a CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the server, and "Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are successful with https, though.

The server runs LispWorks 5.0.2 on Debian Etch. The client runs on LispWorks 5.0.2 on Windows XP SP2. Part of the application pulls in the trivial-ldap module which depends on cl+ssl - not sure if this can be an issue as cl+ssl is not needed on LispWorks for Drakma and Hunchentoot.

I am sure I am doing something stupid. The handler that does the actual upload on the client (which also uses Hunchentoot) is:
(define-easy-handler (upload-fp-image.action :uri "/hunchentoot/patient-study-db/upload-fp-image.action") ()
  (let ((cookie-jar (make-instance 'drakma:cookie-jar
                                   :cookies (list (make-instance 'drakma:cookie :name (session-value :hs-name) :value (session-value :hs-id)
                                                                 :path (session-value :hs-path) :domain (session-value :hs-domain))))))
    (handler-case
        (let ((html-result (drakma:http-request (session-value :upload-url)
                                                :method :post
                                                :parameters (list (cons "finger" (session-value :finger))
                                                                  (cons "img-file" (pathname *scanned-file*))
                                                                  (cons "hs-id" (session-value :hs-id)))
                                                :user-agent (session-value :user-agent)
                                                :cookie-jar cookie-jar)))
          (delete-file *scanned-file*)
          (with-html-output-to-string (*standard-output*)
            (str html-result)))
      (condition (con)
        (delete-file *scanned-file*)
        (show-condition-html con (session-value :back-caption) (session-value :back-url))))))

The Hunchentoot server log spits out:
[2007-12-12 11:35:39 [ERROR]] While parsing multipart/form-data parameters: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 200B000C> occurred
[2007-12-12 11:35:39] 10.38.1.45 - "POST /hunchentoot/patient-study-db/identify-patient.action HTTP/1.1" 200 1058 "-" "Opera/9.10 (Windows NT 5.1; U; en)"
[2007-12-12 11:35:39 [ERROR]] Error while processing connection: Not all chunks from #<CHUNGA:CHUNKED-IO-STREAM 223A7893> have been read completely.

Backtraces on the server and client are included below.

Thanks.
Nico

Backtrace on server:
Error: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 223AB36C> occurred
  1 (abort) Quit process.

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

CL-USER 1 : 1 > :b
Call to SIGNAL
Call to (METHOD CHUNGA::FILL-BUFFER (CHUNGA:CHUNKED-INPUT-STREAM))
Call to (METHOD STREAM:STREAM-READ-BYTE (CHUNGA:CHUNKED-INPUT-STREAM))
Call to READ-BYTE
Call to (METHOD FLEXI-STREAMS::READ-BYTE* (FLEXI-STREAMS::FLEXI-BINARY-INPUT-STREAM))
Call to (METHOD STREAM:STREAM-READ-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-INPUT-STREAM))
Call to READ-CHAR
Call to (SUBFUNCTION RFC2388::RUN RFC2388::READ-UNTIL-NEXT-BOUNDARY)
Call to RFC2388::READ-UNTIL-NEXT-BOUNDARY
Call to (METHOD RFC2388:PARSE-MIME (STREAM T))
Call to HUNCHENTOOT::PARSE-RFC2388-FORM-DATA
Call to (METHOD INITIALIZE-INSTANCE :AFTER (HUNCHENTOOT::REQUEST))
Call to CLOS::METHOD-COMBINATION-TEMPLATE
Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1
Call to HUNCHENTOOT::PROCESS-REQUEST
Call to HUNCHENTOOT::PROCESS-CONNECTION
Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)


Backtrace on client:
Error: Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM 2009C0E7>.
  1 (abort) Quit process.

Type :b for backtrace, :c <option number> to proceed,  or :? for other options

CL-USER 1 : 1 > :b
Call to SIGNAL
Call to (METHOD STREAM:STREAM-FLUSH-BUFFER (STREAM:BUFFERED-STREAM))
Call to (METHOD STREAM:STREAM-WRITE-SEQUENCE (STREAM:BUFFERED-STREAM T T T))
Call to CLOS::METHOD-COMBINATION-TEMPLATE
Call to WRITE-SEQUENCE
Call to WRITE-SEQUENCE
Call to (METHOD CHUNGA::WRITE-CHUNK (CHUNGA:CHUNKED-OUTPUT-STREAM T))
Call to (METHOD CHUNGA::FLUSH-BUFFER (CHUNGA:CHUNKED-OUTPUT-STREAM))
Call to (METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (CHUNGA:CHUNKED-OUTPUT-STREAM T T T))
Call to WRITE-SEQUENCE
Call to WRITE-SEQUENCE
Call to DRAKMA::SEND-CONTENT
Call to DRAKMA::SEND-CONTENT
Call to (SUBFUNCTION 1 DRAKMA::MAKE-FORM-DATA-FUNCTION)
Call to (SUBFUNCTION (LABELS DRAKMA::FINISH-REQUEST) DRAKMA:HTTP-REQUEST)
Call to DRAKMA:HTTP-REQUEST
Interpreted call to PATIENT-STUDY-DB-CLIENT::UPLOAD-FP-IMAGE.ACTION
Call to HUNCHENTOOT::PROCESS-REQUEST
Call to HUNCHENTOOT::PROCESS-CONNECTION
Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)


From ndj at hivsa.com  Wed Dec 12 13:59:20 2007
From: ndj at hivsa.com (Nico de Jager)
Date: Wed, 12 Dec 2007 15:59:20 +0200
Subject: [drakma-devel] CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE
In-Reply-To: <87ir346tah.fsf@davinci.hivsa.com> (Nico de Jager's message of
	"Wed, 12 Dec 2007 13:34:14 +0200")
References: <87ir346tah.fsf@davinci.hivsa.com>
Message-ID: <87ejds6mkn.fsf@davinci.hivsa.com>

Nico de Jager <ndj at hivsa.com> writes:

> Hi
>
> I have a fingerprint scanning application, that uploads fingerprint images using Drakma to a Hunchentoot server. Everything works great as long as I don't use a secure connection. With https, I get a CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the server, and "Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are successful with https, though.
>
> The server runs LispWorks 5.0.2 on Debian Etch. The client runs on LispWorks 5.0.2 on Windows XP SP2. Part of the application pulls in the trivial-ldap module which depends on cl+ssl - not sure if this can be an issue as cl+ssl is not needed on LispWorks for Drakma and Hunchentoot.

Trying to isolate the problem, I set up a general upload easy handlers in a separate application. All https uploads work fine from Linux clients but not from Windows, so I guess it is my version of OpenSSL on the Windows box. I use a binary version which I downloaded here (Win32 OpenSSL v0.9.8g): http://www.slproweb.com/products/Win32OpenSSL.html. I followed this link from the FAQ on www.openssl.org.

Does anyone have advice on easily getting a reliable/working copy of OpenSSL on Windows (preferably a binary package), before I compile my own? Or is it something else?

Thanks.
Nico

>
> I am sure I am doing something stupid. The handler that does the actual upload on the client (which also uses Hunchentoot) is:
> (define-easy-handler (upload-fp-image.action :uri "/hunchentoot/patient-study-db/upload-fp-image.action") ()
>   (let ((cookie-jar (make-instance 'drakma:cookie-jar
>                                    :cookies (list (make-instance 'drakma:cookie :name (session-value :hs-name) :value (session-value :hs-id)
>                                                                  :path (session-value :hs-path) :domain (session-value :hs-domain))))))
>     (handler-case
>         (let ((html-result (drakma:http-request (session-value :upload-url)
>                                                 :method :post
>                                                 :parameters (list (cons "finger" (session-value :finger))
>                                                                   (cons "img-file" (pathname *scanned-file*))
>                                                                   (cons "hs-id" (session-value :hs-id)))
>                                                 :user-agent (session-value :user-agent)
>                                                 :cookie-jar cookie-jar)))
>           (delete-file *scanned-file*)
>           (with-html-output-to-string (*standard-output*)
>             (str html-result)))
>       (condition (con)
>         (delete-file *scanned-file*)
>         (show-condition-html con (session-value :back-caption) (session-value :back-url))))))
>
> The Hunchentoot server log spits out:
> [2007-12-12 11:35:39 [ERROR]] While parsing multipart/form-data parameters: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 200B000C> occurred
> [2007-12-12 11:35:39] 10.38.1.45 - "POST /hunchentoot/patient-study-db/identify-patient.action HTTP/1.1" 200 1058 "-" "Opera/9.10 (Windows NT 5.1; U; en)"
> [2007-12-12 11:35:39 [ERROR]] Error while processing connection: Not all chunks from #<CHUNGA:CHUNKED-IO-STREAM 223A7893> have been read completely.
>
> Backtraces on the server and client are included below.
>
> Thanks.
> Nico
>
> Backtrace on server:
> Error: The condition #<CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE 223AB36C> occurred
>   1 (abort) Quit process.
>
> Type :b for backtrace, :c <option number> to proceed,  or :? for other options
>
> CL-USER 1 : 1 > :b
> Call to SIGNAL
> Call to (METHOD CHUNGA::FILL-BUFFER (CHUNGA:CHUNKED-INPUT-STREAM))
> Call to (METHOD STREAM:STREAM-READ-BYTE (CHUNGA:CHUNKED-INPUT-STREAM))
> Call to READ-BYTE
> Call to (METHOD FLEXI-STREAMS::READ-BYTE* (FLEXI-STREAMS::FLEXI-BINARY-INPUT-STREAM))
> Call to (METHOD STREAM:STREAM-READ-CHAR (FLEXI-STREAMS::FLEXI-LATIN-1-INPUT-STREAM))
> Call to READ-CHAR
> Call to (SUBFUNCTION RFC2388::RUN RFC2388::READ-UNTIL-NEXT-BOUNDARY)
> Call to RFC2388::READ-UNTIL-NEXT-BOUNDARY
> Call to (METHOD RFC2388:PARSE-MIME (STREAM T))
> Call to HUNCHENTOOT::PARSE-RFC2388-FORM-DATA
> Call to (METHOD INITIALIZE-INSTANCE :AFTER (HUNCHENTOOT::REQUEST))
> Call to CLOS::METHOD-COMBINATION-TEMPLATE
> Call to CLOS::MAKE-INSTANCE-FROM-CLASS-1
> Call to HUNCHENTOOT::PROCESS-REQUEST
> Call to HUNCHENTOOT::PROCESS-CONNECTION
> Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)
>
>
> Backtrace on client:
> Error: Timeout while writing 8192 bytes to #<COMM:SOCKET-STREAM 2009C0E7>.
>   1 (abort) Quit process.
>
> Type :b for backtrace, :c <option number> to proceed,  or :? for other options
>
> CL-USER 1 : 1 > :b
> Call to SIGNAL
> Call to (METHOD STREAM:STREAM-FLUSH-BUFFER (STREAM:BUFFERED-STREAM))
> Call to (METHOD STREAM:STREAM-WRITE-SEQUENCE (STREAM:BUFFERED-STREAM T T T))
> Call to CLOS::METHOD-COMBINATION-TEMPLATE
> Call to WRITE-SEQUENCE
> Call to WRITE-SEQUENCE
> Call to (METHOD CHUNGA::WRITE-CHUNK (CHUNGA:CHUNKED-OUTPUT-STREAM T))
> Call to (METHOD CHUNGA::FLUSH-BUFFER (CHUNGA:CHUNKED-OUTPUT-STREAM))
> Call to (METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (CHUNGA:CHUNKED-OUTPUT-STREAM T T T))
> Call to WRITE-SEQUENCE
> Call to WRITE-SEQUENCE
> Call to DRAKMA::SEND-CONTENT
> Call to DRAKMA::SEND-CONTENT
> Call to (SUBFUNCTION 1 DRAKMA::MAKE-FORM-DATA-FUNCTION)
> Call to (SUBFUNCTION (LABELS DRAKMA::FINISH-REQUEST) DRAKMA:HTTP-REQUEST)
> Call to DRAKMA:HTTP-REQUEST
> Interpreted call to PATIENT-STUDY-DB-CLIENT::UPLOAD-FP-IMAGE.ACTION
> Call to HUNCHENTOOT::PROCESS-REQUEST
> Call to HUNCHENTOOT::PROCESS-CONNECTION
> Call to (SUBFUNCTION MP::PROCESS-SG-FUNCTION MP::INITIALIZE-PROCESS-STACK)
> _______________________________________________
> drakma-devel mailing list
> drakma-devel at common-lisp.net
> http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel


From edi at agharta.de  Thu Dec 13 20:36:22 2007
From: edi at agharta.de (Edi Weitz)
Date: Thu, 13 Dec 2007 21:36:22 +0100
Subject: [drakma-devel] CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE
In-Reply-To: <87ir346tah.fsf@davinci.hivsa.com> (Nico de Jager's message of
	"Wed, 12 Dec 2007 13:34:14 +0200")
References: <87ir346tah.fsf@davinci.hivsa.com>
Message-ID: <uwsrigwmx.fsf@agharta.de>

On Wed, 12 Dec 2007 13:34:14 +0200, Nico de Jager <ndj at hivsa.com> wrote:

> I have a fingerprint scanning application, that uploads fingerprint
> images using Drakma to a Hunchentoot server. Everything works great
> as long as I don't use a secure connection. With https, I get a
> CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE condition on the
> server, and "Timeout while writing 8192 bytes to
> #<COMM:SOCKET-STREAM >" on the client. Sometimes the uploads are
> successful with https, though.

It's hard to say something about this from here.  Are you able to
generate a small, self-contained example that exposes the problem?
What did you do to isolate the error?  Does it work without the
trivial-ldap module being loaded?  Does it work with another Lisp
implementation on the client or on the server side?  Does it work if
you use :CONTENT-LENGTH T in your Drakma request?

Edi.


From edi at agharta.de  Thu Dec 13 20:37:46 2007
From: edi at agharta.de (Edi Weitz)
Date: Thu, 13 Dec 2007 21:37:46 +0100
Subject: [drakma-devel] CHUNGA:INPUT-CHUNKING-UNEXPECTED-END-OF-FILE
In-Reply-To: <87ejds6mkn.fsf@davinci.hivsa.com> (Nico de Jager's message of
	"Wed, 12 Dec 2007 15:59:20 +0200")
References: <87ir346tah.fsf@davinci.hivsa.com>
	<87ejds6mkn.fsf@davinci.hivsa.com>
Message-ID: <usl26gwkl.fsf@agharta.de>

On Wed, 12 Dec 2007 15:59:20 +0200, Nico de Jager <ndj at hivsa.com> wrote:

> Trying to isolate the problem, I set up a general upload easy
> handlers in a separate application. All https uploads work fine from
> Linux clients but not from Windows, so I guess it is my version of
> OpenSSL on the Windows box. I use a binary version which I
> downloaded here (Win32 OpenSSL v0.9.8g):
> http://www.slproweb.com/products/Win32OpenSSL.html. I followed this
> link from the FAQ on www.openssl.org.
>
> Does anyone have advice on easily getting a reliable/working copy of
> OpenSSL on Windows (preferably a binary package), before I compile
> my own? Or is it something else?

I use the one you also have.  They are the ones recommended by
LispWorks:

  http://www.lispworks.com/documentation/lw50/LWUG/html/lwuser-344.htm


From peter.eddy at gmail.com  Mon Dec 17 17:45:51 2007
From: peter.eddy at gmail.com (Peter Eddy)
Date: Mon, 17 Dec 2007 12:45:51 -0500
Subject: [drakma-devel] question about encodings and content length
	calculation
Message-ID: <5aea26870712170945s52bcb6abvcc37d7f7cf6992b0@mail.gmail.com>

Hi,

I'm trying resolve a problem I'm having trying to send utf-8 data and
the calculation of the length of that data. If I make the following
post:

(defconstant +utf-8+ (make-external-format :utf-8 :eol-style :lf)

(http-request "http://odeon:5984"
                       :content-type "application/xml"
                       :external-format-out +utf-8+
                       :content "e"
                       :method :put)

The resulting request looks like this:

PUT / HTTP/1.1
Host: odeon:5984
User-Agent: Drakma/0.11.1 (OpenMCL Version 1.1-pre-070722
(DarwinX8664); Darwin; 9.1.0; http://weitz.de/drakma/)
Accept: */*
Connection: close
Content-Type: application/xml
Content-Length: 1

e

Which is right. But if I replace the "e" content with "?"  (e accent
acute), the request looks like this:

PUT / HTTP/1.1
Host: odeon:5984
User-Agent: Drakma/0.11.1 (OpenMCL Version 1.1-pre-070722
(DarwinX8664); Darwin; 9.1.0; http://weitz.de/drakma/)
Accept: */*
Connection: close
Content-Type: application/xml
Content-Length: 1

..

The hex values of the ".." are C3 A9, which is the utf-8 value for ?.
The problem is, of course, that the content length is really 2.

I assume I'm doing something wrong, but I don't know what exactly that
would be. Should I be calculating the length myself?

thanks for any pointers or suggestions.

- Peter


From edi at agharta.de  Mon Dec 17 20:25:54 2007
From: edi at agharta.de (Edi Weitz)
Date: Mon, 17 Dec 2007 21:25:54 +0100
Subject: [drakma-devel] question about encodings and content length
	calculation
In-Reply-To: <5aea26870712170945s52bcb6abvcc37d7f7cf6992b0@mail.gmail.com>
	(Peter Eddy's message of "Mon, 17 Dec 2007 12:45:51 -0500")
References: <5aea26870712170945s52bcb6abvcc37d7f7cf6992b0@mail.gmail.com>
Message-ID: <ur6hlt6el.fsf@agharta.de>

On Mon, 17 Dec 2007 12:45:51 -0500, "Peter Eddy" <peter.eddy at gmail.com> wrote:

> Should I be calculating the length myself?

Yes.  See here:

  http://weitz.de/drakma/#content-length

One could argue that Drakma should be able to compute the right value
itself, but this is more trickier than one first thinks.  Of course,
if you think you know how to do this in a way that covers all corner
cases, I'd be happy to accept a patch.  The general design philosophy
of Drakma is that it should be as easy to use as possible.

  http://weitz.de/patches.html

Thanks,
Edi.


From peter.eddy at gmail.com  Mon Dec 17 21:53:42 2007
From: peter.eddy at gmail.com (Peter Eddy)
Date: Mon, 17 Dec 2007 16:53:42 -0500
Subject: [drakma-devel] question about encodings and content length
	calculation
In-Reply-To: <ur6hlt6el.fsf@agharta.de>
References: <5aea26870712170945s52bcb6abvcc37d7f7cf6992b0@mail.gmail.com>
	<ur6hlt6el.fsf@agharta.de>
Message-ID: <5aea26870712171353w36727ae0mc2ea64af7189074b@mail.gmail.com>

On Dec 17, 2007 3:25 PM, Edi Weitz <edi at agharta.de> wrote:
> On Mon, 17 Dec 2007 12:45:51 -0500, "Peter Eddy" <peter.eddy at gmail.com> wrote:
>
> > Should I be calculating the length myself?
>
> Yes.  See here:
>
>   http://weitz.de/drakma/#content-length
>
> One could argue that Drakma should be able to compute the right value
> itself, but this is more trickier than one first thinks.  Of course,
> if you think you know how to do this in a way that covers all corner
> cases, I'd be happy to accept a patch.  The general design philosophy
> of Drakma is that it should be as easy to use as possible.

It turns out that in my case setting content-length to nil (and
therefore using chunked encoding) works perfectly. I was sure I'd
tried that before, but must not have. Anyway, I think that qualifies
as easy.

Thanks!

- Peter


From edi at agharta.de  Sat Dec 22 02:29:55 2007
From: edi at agharta.de (Edi Weitz)
Date: Sat, 22 Dec 2007 03:29:55 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
In-Reply-To: <20071221125840.GA26461@tehran.lain.pl> (Stanislaw Halik's
	message of "Fri, 21 Dec 2007 13:58:40 +0100")
References: <20071221125840.GA26461@tehran.lain.pl>
Message-ID: <utzmbsbq4.fsf@agharta.de>

On Fri, 21 Dec 2007 13:58:40 +0100, Stanislaw Halik <sthalik at tehran.lain.pl> wrote:

> I attach the patch for drakma to support the `connection-timeout'
> argument for `http-request' on SBCL.  Hope you like it.

Hi Stanislaw,

I'm copying my reply to the Drakma mailing list where I think this
discussion should take place.  Thank you for the patches you sent, but
I think that this functionality should rather be in usocket and not in
Drakma.  If it's offered by usocket, then Drakma will use it.  Have
you considered sending a patch to the usocket maintainers?

Thanks again,
Edi.


From sthalik at tehran.lain.pl  Sun Dec 23 04:34:02 2007
From: sthalik at tehran.lain.pl (Stanislaw Halik)
Date: Sun, 23 Dec 2007 05:34:02 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
Message-ID: <20071223043402.GA10963@tehran.lain.pl>

I've updated the patch to support the IO-TIMEOUT keyword argument. It's
not possible to non-intrusively support read- and write-timeouts
separately to bring the API in congruency with LispWorks.

The resolver error gets changed to SIMPLE-ERROR because it's not an
ERROR, but a CONDITION (signalled with ERROR, though).

I've stress-tested the patch, running 30 threads doing HTTP-REQUEST.

My rationale for including the patch - it doesn't change the program
logic much, and except for the function definition and socket connection
there's no special-casing for SBCL.

The timeout option is crucial for me: my typical usage is running
concurrent threads accessing URLs posted in spam to make the senders
think I'm a used accessing the page, in turn sending more spam,
addresses of which land in my spamtrap. Some of the URLs point to broken
httpds, leaving an open connection without sending any replies. This
leads to unfinished requests using threads and hanging for days.

If you feel like maintaing a piece of code for SBCL, by all means please
do so. I think other SBCL users might benefit from it.

-- 
   /\   
    /   Jabber ID       :: sthalik at jabber.autocom.pl
  \ \/  Unix stuff      :: http://tehran.lain.pl
 \/\    Yet Another RBL :: http://rbl.lain.pl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: drakma-sbcl-connection-timeout.diff
Type: text/x-diff
Size: 3385 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20071223/ccd5cdde/attachment.diff>

From sthalik at tehran.lain.pl  Sun Dec 23 05:13:43 2007
From: sthalik at tehran.lain.pl (Stanislaw Halik)
Date: Sun, 23 Dec 2007 06:13:43 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
In-Reply-To: <20071223043402.GA10963@tehran.lain.pl>
References: <20071223043402.GA10963@tehran.lain.pl>
Message-ID: <20071223051343.GA13614@tehran.lain.pl>

On Sun, Dec 23, 2007, Stanislaw Halik wrote:
> [...]

As for the reason I haven't submitted the patch to usocket itself:
usocket is a compatibility layer, comprising of features readily
available in most of CL implementations. With socket timeouts supported
so far by only for LispWorks and SBCL, I believe the patch would have no
chances of getting accepted.

-- 
   /\   
    /   Jabber ID       :: sthalik at jabber.autocom.pl
  \ \/  Unix stuff      :: http://tehran.lain.pl
 \/\    Yet Another RBL :: http://rbl.lain.pl


From edi at agharta.de  Sun Dec 23 22:09:33 2007
From: edi at agharta.de (Edi Weitz)
Date: Sun, 23 Dec 2007 23:09:33 +0100
Subject: [drakma-devel] Re: [patch] drakma, support for
	connection-timeout on SBCL
In-Reply-To: <20071223051343.GA13614@tehran.lain.pl> (Stanislaw Halik's
	message of "Sun, 23 Dec 2007 06:13:43 +0100")
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl>
Message-ID: <ulk7lf4gy.fsf@agharta.de>

On Sun, 23 Dec 2007 06:13:43 +0100, Stanislaw Halik <sthalik at tehran.lain.pl> wrote:

> As for the reason I haven't submitted the patch to usocket itself:
> usocket is a compatibility layer, comprising of features readily
> available in most of CL implementations. With socket timeouts
> supported so far by only for LispWorks and SBCL, I believe the patch
> would have no chances of getting accepted.

Let's do it like this:

1. Ask them.  That doesn't cost anything.  They can still offer to
   support timeouts and make that a no-op for implementations which
   don't have them (like Drakam does).  (Also, are you sure it's only
   LispWorks and SBCL?  Have you checked AllegroCL for example?)

2. If they say yes, that's fine and I'll update Drakma to work with
   the latest usocket release.

3. If they say no, let's talk about integrating your patch into
   Drakma.

Actually, I hope they say yes.  Several people tried to convince me to
switch Drakma from trivial-sockets to usocket because there's active
development over there...


From ehuels at gmail.com  Wed Dec 26 14:25:54 2007
From: ehuels at gmail.com (Erik Huelsmann)
Date: Wed, 26 Dec 2007 15:25:54 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
In-Reply-To: <ulk7lf4gy.fsf@agharta.de>
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
Message-ID: <aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>

On Dec 23, 2007 11:09 PM, Edi Weitz <edi at agharta.de> wrote:
> On Sun, 23 Dec 2007 06:13:43 +0100, Stanislaw Halik <sthalik at tehran.lain.pl> wrote:
>
> > As for the reason I haven't submitted the patch to usocket itself:
> > usocket is a compatibility layer, comprising of features readily
> > available in most of CL implementations. With socket timeouts
> > supported so far by only for LispWorks and SBCL, I believe the patch
> > would have no chances of getting accepted.
>
> Let's do it like this:
>
> 1. Ask them.  That doesn't cost anything.  They can still offer to
>    support timeouts and make that a no-op for implementations which
>    don't have them (like Drakam does).  (Also, are you sure it's only
>    LispWorks and SBCL?  Have you checked AllegroCL for example?)
>
> 2. If they say yes, that's fine and I'll update Drakma to work with
>    the latest usocket release.

Well, I think they'll accept (a patch which fits into the usocket
framework), especially since timeouts come for free in Allegro and
CLISP (through a with-timeout macro and a :timeout parameter
respectively). Research should be able to turn up ways to achieve the
same thing in other lisps.

In a portability library you generally can't be expected to implement
a feature for all supported platforms. However: the more you submit
with the original patch, the bigger the chances for integration.

bye,


Erik.


From edi at agharta.de  Wed Dec 26 17:00:14 2007
From: edi at agharta.de (Edi Weitz)
Date: Wed, 26 Dec 2007 18:00:14 +0100
Subject: [drakma-devel] Re: [patch] drakma, support for
	connection-timeout on SBCL
In-Reply-To: <aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	(Erik Huelsmann's message of "Wed, 26 Dec 2007 15:25:54 +0100")
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
Message-ID: <uir2lgzmp.fsf@agharta.de>

On Wed, 26 Dec 2007 15:25:54 +0100, "Erik Huelsmann" <ehuels at gmail.com> wrote:

> Well, I think they'll accept (a patch which fits into the usocket
> framework),

To the OP: In case that wasn't clear - Erik is the maintainer of
usocket.  So, if /he/ says so...

> especially since timeouts come for free in Allegro and CLISP

And LispWorks.  See :TIMEOUT, :READ-TIMEOUT, and :WRITE-TIMEOUT here:

  http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-35.htm

Edi.


From ehuels at gmail.com  Wed Dec 26 17:29:51 2007
From: ehuels at gmail.com (Erik Huelsmann)
Date: Wed, 26 Dec 2007 18:29:51 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
In-Reply-To: <uir2lgzmp.fsf@agharta.de>
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
Message-ID: <aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>

On Dec 26, 2007 6:00 PM, Edi Weitz <edi at agharta.de> wrote:
> On Wed, 26 Dec 2007 15:25:54 +0100, "Erik Huelsmann" <ehuels at gmail.com> wrote:
>
> > Well, I think they'll accept (a patch which fits into the usocket
> > framework),
>
> To the OP: In case that wasn't clear - Erik is the maintainer of
> usocket.  So, if /he/ says so...
>
> > especially since timeouts come for free in Allegro and CLISP
>
> And LispWorks.

Right, I meant "in addition to the LispWorks and SBCL timeouts which
are implemented in the patch".

Actually, if I re-read the original posting and the reason for the
time-outs, I think I have it already implemented in the usocket trunk
(except for SBCL/Win32):

usocket trunk supports a select() like interface to determine which
sockets are available for reading. With that code, you could implement
the 10 socket readers in 1 thread instead of in 10. The interface
returns the sockets which are ready for reading. If the timeout
exceeds, an empty list is returned. The interface is called
WAIT-FOR-INPUT.

The idea is that you only start reading the response until you have
input on the socket available. I understand this is not exactly the
same solution as the route you have taken, but it's available today on
many more lisps than the 2 you implemented so far. Do you think this
approach might work for you?

The only problem is that I fixed the LispWorks/Win32 code last weekend
and haven't had the chance to commit it yet. (Meaning that I did
implement: ECL/Unix, ECL/Windows, Allegro, SBCL/Unix, CMUCL,
ArmedBear, CLISP, Clozure CL (aka OpenMCL). Douglas Crosher
implemented Scieneer CL.)

I could use someone with a real life application to test the library.


bye,


Erik.


From sthalik at tehran.lain.pl  Fri Dec 28 19:25:10 2007
From: sthalik at tehran.lain.pl (Stanislaw Halik)
Date: Fri, 28 Dec 2007 20:25:10 +0100
Subject: [drakma-devel] Re: [patch] drakma, support for
	connection-timeout on SBCL
In-Reply-To: <aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
	<aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
Message-ID: <20071228192510.GA9012@tehran.lain.pl>

On Wed, Dec 26, 2007, Erik Huelsmann wrote:
> usocket trunk supports a select() like interface to determine which
> sockets are available for reading. With that code, you could implement
> the 10 socket readers in 1 thread instead of in 10. The interface
> returns the sockets which are ready for reading. If the timeout
> exceeds, an empty list is returned. The interface is called
> WAIT-FOR-INPUT.

How about wrapping the socket in a gray stream when timeouts are used,
and doing WAIT-FOR-INPUT before read operations? That sounds better than
changing every place Drakma and Chunga read from sockets.

-- 
   /\   
    /   Jabber ID       :: sthalik at jabber.autocom.pl
  \ \/  Unix stuff      :: http://tehran.lain.pl
 \/\    Yet Another RBL :: http://rbl.lain.pl


From sthalik at tehran.lain.pl  Fri Dec 28 21:53:51 2007
From: sthalik at tehran.lain.pl (Stanislaw Halik)
Date: Fri, 28 Dec 2007 22:53:51 +0100
Subject: [drakma-devel] Re: [patch] drakma, support for
	connection-timeout on SBCL
In-Reply-To: <20071228192510.GA9012@tehran.lain.pl>
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
	<aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
	<20071228192510.GA9012@tehran.lain.pl>
Message-ID: <20071228215350.GA11092@tehran.lain.pl>

On Fri, Dec 28, 2007, Stanislaw Halik wrote:
> On Wed, Dec 26, 2007, Erik Huelsmann wrote:
>> usocket trunk supports a select() like interface to determine which
>> sockets are available for reading. With that code, you could implement
>> the 10 socket readers in 1 thread instead of in 10. The interface
>> returns the sockets which are ready for reading. If the timeout
>> exceeds, an empty list is returned. The interface is called
>> WAIT-FOR-INPUT.

> How about wrapping the socket in a gray stream when timeouts are used,
> and doing WAIT-FOR-INPUT before read operations? That sounds better than
> changing every place Drakma and Chunga read from sockets.

This can't be done this easily. WAIT-FOR-INPUT interacts badly with
buffering. Consider this:

CL-USER> (usocket:socket-connect "ananke" 25)
#<USOCKET:STREAM-USOCKET {B550781}>

CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3)
(#<USOCKET:STREAM-USOCKET {B550781}>)
3
CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3)
(#<USOCKET:STREAM-USOCKET {B550781}>)
3

CL-USER> (read-char (usocket:socket-stream (car '(#<USOCKET:STREAM-USOCKET {B550781}>))))
#\2

CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3)
NIL
NIL

The SMTP banner has already been slurped into the buffer and select(2)
returns 0. This could work with :buffering :none. usocket doesn't give
an option as whether to enable input buffering.

As for read timeout support for usocket, only SBCL and LispWorks support
it. CLISP has WITH-TIMEOUT, but that would require wrapping the socket
in a gray stream with WITH-TIMEOUT. Note that WITH-TIMEOUT in CLISP
depends on threads, not enabled by default nor in Debian package.

-- 
   /\   
    /   Jabber ID       :: sthalik at jabber.autocom.pl
  \ \/  Unix stuff      :: http://tehran.lain.pl
 \/\    Yet Another RBL :: http://rbl.lain.pl


From ehuels at gmail.com  Sat Dec 29 09:27:18 2007
From: ehuels at gmail.com (Erik Huelsmann)
Date: Sat, 29 Dec 2007 10:27:18 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
In-Reply-To: <20071228215350.GA11092@tehran.lain.pl>
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
	<aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
	<20071228192510.GA9012@tehran.lain.pl>
	<20071228215350.GA11092@tehran.lain.pl>
Message-ID: <aea328ab0712290127p3d2fe7e0yc09f70c1db3d1150@mail.gmail.com>

On Dec 28, 2007 10:53 PM, Stanislaw Halik <sthalik at tehran.lain.pl> wrote:
> On Fri, Dec 28, 2007, Stanislaw Halik wrote:
> > On Wed, Dec 26, 2007, Erik Huelsmann wrote:
> >> usocket trunk supports a select() like interface to determine which
> >> sockets are available for reading. With that code, you could implement
> >> the 10 socket readers in 1 thread instead of in 10. The interface
> >> returns the sockets which are ready for reading. If the timeout
> >> exceeds, an empty list is returned. The interface is called
> >> WAIT-FOR-INPUT.
>
> > How about wrapping the socket in a gray stream when timeouts are used,
> > and doing WAIT-FOR-INPUT before read operations? That sounds better than
> > changing every place Drakma and Chunga read from sockets.
>
> This can't be done this easily. WAIT-FOR-INPUT interacts badly with
> buffering. Consider this:

In the LispWorks variant (on Windows) I realised this problem and I've
implemented something that takes buffering into account. I'll need to
backport that. I think however that this subject is starting to be
more about usocket than about Drakma, so, possibly we need to switch
this discussion to usocket-devel@ ? I'd gladly work with you (and
others) to make WAIT-FOR-INPUT usable.

> CL-USER> (usocket:socket-connect "ananke" 25)
> #<USOCKET:STREAM-USOCKET {B550781}>
>
> CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3)
> (#<USOCKET:STREAM-USOCKET {B550781}>)
> 3
> CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3)
> (#<USOCKET:STREAM-USOCKET {B550781}>)
> 3
>
> CL-USER> (read-char (usocket:socket-stream (car '(#<USOCKET:STREAM-USOCKET {B550781}>))))
> #\2
>
> CL-USER> (usocket:wait-for-input '#<USOCKET:STREAM-USOCKET {B550781}> :timeout 3)
> NIL
> NIL
>
> The SMTP banner has already been slurped into the buffer and select(2)
> returns 0. This could work with :buffering :none. usocket doesn't give
> an option as whether to enable input buffering.

Right. I haven't decided what to do about access to buffering streams
(or at least setting the parameter), because in general this can only
be an advisory parameter (not all implementations support setting
'bufferedness' of a stream).

> As for read timeout support for usocket, only SBCL and LispWorks support
> it. CLISP has WITH-TIMEOUT, but that would require wrapping the socket
> in a gray stream with WITH-TIMEOUT. Note that WITH-TIMEOUT in CLISP
> depends on threads, not enabled by default nor in Debian package.

The UNIX socket FAQ
(http://www.developerweb.net/forum/archive/index.php/t-3439.html)
suggests that it's nearly impossible to correctly implement timeouts
using SO_RCVTIMEO and SO_SENDTIMEO in a portable manner. Too many
differences occur on different platforms...

That's why I rather use
select()/LISTEN/READ-CHAR/READ-CHAR-NO-HANG/etc. to implement time-out
behaviour which can be used on all lisps on all OSes. I hope you don't
mind (and that you - as I do - think that the result is what counts).


Thanks for taking the time to test this new usocket behaviour!

bye,


Erik.


From sthalik at tehran.lain.pl  Sat Dec 29 15:32:48 2007
From: sthalik at tehran.lain.pl (Stanislaw Halik)
Date: Sat, 29 Dec 2007 16:32:48 +0100
Subject: [drakma-devel] Re: [patch] drakma, support for
	connection-timeout on SBCL
In-Reply-To: <aea328ab0712290127p3d2fe7e0yc09f70c1db3d1150@mail.gmail.com>
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
	<aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
	<20071228192510.GA9012@tehran.lain.pl>
	<20071228215350.GA11092@tehran.lain.pl>
	<aea328ab0712290127p3d2fe7e0yc09f70c1db3d1150@mail.gmail.com>
Message-ID: <20071229153248.GA8556@tehran.lain.pl>

On Sat, Dec 29, 2007, Erik Huelsmann wrote:
> That's why I rather use
> select()/LISTEN/READ-CHAR/READ-CHAR-NO-HANG/etc. to implement time-out
> behaviour which can be used on all lisps on all OSes. I hope you don't
> mind (and that you - as I do - think that the result is what counts).

I have implemented timeouts for non-LispWorks lisps. Latest usocket
trunk and working usocket:wait-for-input required.

Patch attached.

-- 
   /\   
    /   Jabber ID       :: sthalik at jabber.autocom.pl
  \ \/  Unix stuff      :: http://tehran.lain.pl
 \/\    Yet Another RBL :: http://rbl.lain.pl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: drakma-timeout-gray-stream.diff
Type: text/x-diff
Size: 4947 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/drakma-devel/attachments/20071229/e857d16a/attachment.diff>

From edi at agharta.de  Sat Dec 29 17:49:08 2007
From: edi at agharta.de (Edi Weitz)
Date: Sat, 29 Dec 2007 18:49:08 +0100
Subject: [drakma-devel] Re: [patch] drakma, support for
	connection-timeout on SBCL
In-Reply-To: <20071229153248.GA8556@tehran.lain.pl> (Stanislaw Halik's
	message of "Sat, 29 Dec 2007 16:32:48 +0100")
References: <20071223043402.GA10963@tehran.lain.pl>
	<20071223051343.GA13614@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
	<aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
	<20071228192510.GA9012@tehran.lain.pl>
	<20071228215350.GA11092@tehran.lain.pl>
	<aea328ab0712290127p3d2fe7e0yc09f70c1db3d1150@mail.gmail.com>
	<20071229153248.GA8556@tehran.lain.pl>
Message-ID: <u3atlmlwr.fsf@agharta.de>

On Sat, 29 Dec 2007 16:32:48 +0100, Stanislaw Halik <sthalik at tehran.lain.pl> wrote:

> I have implemented timeouts for non-LispWorks lisps. Latest usocket
> trunk and working usocket:wait-for-input required.
>
> Patch attached.

Ugh, does that mean usocket fully supports timeouts now?  I must have
missed parts of the discussion.

Still, I'd rather wait until there's an official usocket release, so
people who install Drakma can use ASDF-INSTALL if they want.


From ehuels at gmail.com  Sun Dec 30 10:16:43 2007
From: ehuels at gmail.com (Erik Huelsmann)
Date: Sun, 30 Dec 2007 11:16:43 +0100
Subject: [drakma-devel] Re: [patch] drakma,
	support for connection-timeout on SBCL
In-Reply-To: <u3atlmlwr.fsf@agharta.de>
References: <20071223043402.GA10963@tehran.lain.pl> <ulk7lf4gy.fsf@agharta.de>
	<aea328ab0712260625v778e9d26o50e069b3eca3512a@mail.gmail.com>
	<uir2lgzmp.fsf@agharta.de>
	<aea328ab0712260929o6e9d78b0m102cf6c9ff245373@mail.gmail.com>
	<20071228192510.GA9012@tehran.lain.pl>
	<20071228215350.GA11092@tehran.lain.pl>
	<aea328ab0712290127p3d2fe7e0yc09f70c1db3d1150@mail.gmail.com>
	<20071229153248.GA8556@tehran.lain.pl> <u3atlmlwr.fsf@agharta.de>
Message-ID: <aea328ab0712300216v46db8092s554e32a043ec50ef@mail.gmail.com>

On Dec 29, 2007 6:49 PM, Edi Weitz <edi at agharta.de> wrote:
> On Sat, 29 Dec 2007 16:32:48 +0100, Stanislaw Halik <sthalik at tehran.lain.pl> wrote:
>
> > I have implemented timeouts for non-LispWorks lisps. Latest usocket
> > trunk and working usocket:wait-for-input required.
> >
> > Patch attached.
>
> Ugh, does that mean usocket fully supports timeouts now?  I must have
> missed parts of the discussion.

I don't think you missed parts of the discussion: usocket supports
time-outs only in a way supported by select(). This means we can only
wait for the signal "there's input". This is different than the first
patch submitted in this thread which waits a certain amount of time
for a specified minimum amount of data.

When reading about the reason *why* he wants to implement timeouts is
not because he's getting incomplete data, but some servers are broken
and send no response at all (but still hang on to the connection!).
This use-case can be solved without any send/receive time-outs.

In order to be able to use the select() approach with Drakma,
Stanislaw implemented a stream which waits for *some* data to become
available within the specified timeout period. If it does not happen,
it raises an error, if it does, it just reads the data from the
network and returns it to Drakma.

> Still, I'd rather wait until there's an official usocket release, so
> people who install Drakma can use ASDF-INSTALL if they want.

That'd be my preferred approach too. (Although testing trunk code is
highly appreciated, ofcourse!) I have found in the past it's hard to
manage a project which has dependencies on non-released code, if you
don't own the dependency project too.

bye,


Erik.
PS: I think Stanislaw can use his patch and usocket trunk without
problems, it's just that I'd prefer not to have projects depend on
unreleased usocket code - yet.