From edi at weitz.de Thu Mar 3 07:16:52 2011 From: edi at weitz.de (Edi Weitz) Date: Thu, 3 Mar 2011 08:16:52 +0100 Subject: [drakma-devel] http-request error In-Reply-To: References: <20110228155006.228780@gmx.net>

Message-ID: So, I don't have CLISP to test this, but this looks like an incompatibility between CLISP and usocket. If you're using the newest versions of both, I'd suggest that you ask on the usocket mailing list and send a full backtrace. Thanks, Edi. On Mon, Feb 28, 2011 at 7:53 PM, Andre Sch?tz wrote: > On Mon, 28 Feb 2011 18:06:19 +0100 > Edi Weitz wrote: > >> 2011/2/28 "Andr? Sch?tz" : >> >> > I loaded the drakma system successfully into my clisp environment and got the following error when I tried to make a http-request: >> > >> > (http-request "http://lisp.org/") >> > >> > Fehler: NODELAY in SOCKET-CONNECT is unsupported. >> > >> > Any ideas? >> >> This is the latest Drakma release or the development version? >> >> Also, are you using the newest version of usocket? ?Which CLISP >> version are you on? ?Which operating system? >> >> Thanks, >> Edi. > > My configuration is as follows: > > - clisp: GNU CLISP 2.44.1 (2008-02-23) (built 3476003035) > - usocket: 0.4.1 > - Operating System: Ubuntu > > Andre > > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > > From dlw at itasoftware.com Thu Mar 17 15:30:21 2011 From: dlw at itasoftware.com (Daniel Weinreb) Date: Thu, 17 Mar 2011 11:30:21 -0400 Subject: [drakma-devel] read-status-line is seeing the HTTP request line!? Message-ID: <4D82290D.10502@itasoftware.com> Has anybody seen a behavior in which read-status-line signals a syntax-error, and it turns out that the value of "line" looks like an HTTP request line rather than an HTTP status (response) line? The following is the stack trace. From my old C++ days (may they never return), I conjectured that some string buffer is being reused or something like that.; We are running on CCL (formerly openmcl), and nothing like that is going on at the CCL level.; I don't know much about flexi-streams or gray-streams. Anyway that's just a rather wild guess. Thanks! DRAKMA::LOG-STREAM: NIL Local bindings: CHUNGA:CURRENT-ERROR-MESSAGE "While reading status line:"" DRAKMA::LINE: ""POST /stat/ping HTTP/1.1"" DRAKMA::FIRST-SPACE-POS: 4 DRAKMA::SECOND-SPACE-POS: 15 Thanks! - Dan From edi at weitz.de Thu Mar 17 16:08:07 2011 From: edi at weitz.de (Edi Weitz) Date: Thu, 17 Mar 2011 17:08:07 +0100 Subject: [drakma-devel] read-status-line is seeing the HTTP request line!? In-Reply-To: <4D82290D.10502@itasoftware.com> References: <4D82290D.10502@itasoftware.com> Message-ID: Hi Dan, I never saw something like that. Can you reproduce it? FWIW, Chunga uses buffered streams, but there are different buffers for input and output. And, again, I have never seen something like this. I would think that there are enough users of Drakma, so that somebody else should have encountered such a behavior before. BTW, is the request line you saw the one you sent? Edi. On Thu, Mar 17, 2011 at 4:30 PM, Daniel Weinreb wrote: > Has anybody seen a behavior in which read-status-line > signals a syntax-error, and it turns out that the value > of "line" looks like an HTTP request line rather > than an HTTP status (response) line? ?The following > is the stack trace. > > ?From my old C++ days (may they never return), I > conjectured that some string buffer is being reused > or something like that.; We are running on CCL > (formerly openmcl), and nothing like that is > going on at the CCL level.; I don't know much > about flexi-streams or gray-streams. Anyway > that's just a rather wild guess. Thanks! > > ? ? Arguments: (STREAM &OPTIONAL DRAKMA::LOG-STREAM) > ? ? ? STREAM: > ? ? ? DRAKMA::LOG-STREAM: NIL > ? ? Local bindings: > ? ? ? CHUNGA:CURRENT-ERROR-MESSAGE "While reading status line:"" > ? ? ? DRAKMA::LINE: ""POST /stat/ping HTTP/1.1"" > ? ? ? DRAKMA::FIRST-SPACE-POS: 4 > ? ? ? DRAKMA::SECOND-SPACE-POS: 15 > > Thanks! > > - Dan > > > > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > From ryan at acceleration.net Thu Mar 17 16:04:37 2011 From: ryan at acceleration.net (Ryan Davis) Date: Thu, 17 Mar 2011 12:04:37 -0400 Subject: [drakma-devel] read-status-line is seeing the HTTP request line!? In-Reply-To: <4D82290D.10502@itasoftware.com> References: <4D82290D.10502@itasoftware.com> Message-ID: <4D823115.3000906@acceleration.net> On 3/17/2011 11:30 AM, Daniel Weinreb wrote: > Has anybody seen a behavior in which read-status-line > signals a syntax-error, and it turns out that the value > of "line" looks like an HTTP request line rather > than an HTTP status (response) line? The following > is the stack trace. I had a similar problem, and I concluded it was the server sending back a very malformed response. Did you validate the server's response outside of drakma? HTH, Ryan > From my old C++ days (may they never return), I > conjectured that some string buffer is being reused > or something like that.; We are running on CCL > (formerly openmcl), and nothing like that is > going on at the CCL level.; I don't know much > about flexi-streams or gray-streams. Anyway > that's just a rather wild guess. Thanks! > > Arguments: (STREAM &OPTIONAL DRAKMA::LOG-STREAM) > STREAM: > DRAKMA::LOG-STREAM: NIL > Local bindings: > CHUNGA:CURRENT-ERROR-MESSAGE "While reading status line:"" > DRAKMA::LINE: ""POST /stat/ping HTTP/1.1"" > DRAKMA::FIRST-SPACE-POS: 4 > DRAKMA::SECOND-SPACE-POS: 15 > > Thanks! > > - Dan > > > > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel -- Ryan Davis Acceleration.net Director of Programming Services 2831 NW 41st street, suite B Gainesville, FL 32606 Office: 352-335-6500 x 124 Fax: 352-335-6506 From dlw at itasoftware.com Thu Mar 17 19:08:27 2011 From: dlw at itasoftware.com (Daniel Weinreb) Date: Thu, 17 Mar 2011 15:08:27 -0400 Subject: [drakma-devel] read-status-line is seeing the HTTP request line!? In-Reply-To: References: <4D82290D.10502@itasoftware.com> Message-ID: <4D825C2B.3080600@itasoftware.com> Edi Weitz wrote: > Hi Dan, > > I never saw something like that. Can you reproduce it? > Ah, if I could do that, I wouldn't be asking. :( > FWIW, Chunga uses buffered streams, but there are different buffers > for input and output. And, again, I have never seen something like > this. I would think that there are enough users of Drakma, so that > somebody else should have encountered such a behavior before. > > BTW, is the request line you saw the one you sent? > Yes, it's the one we sent for this request. It's not clear which level of abstraction could be causing this. Matt Emerson and Gail Zacharais have checked out the CCL level (what's going on inside with-output-to-stream), but it doesn't seem to be happening there. In fact I just don't see any way that this could be happening. I was hoping against hope that someone might have already seen this. Thanks! -- Dan > Edi. > > > > On Thu, Mar 17, 2011 at 4:30 PM, Daniel Weinreb wrote: > >> Has anybody seen a behavior in which read-status-line >> signals a syntax-error, and it turns out that the value >> of "line" looks like an HTTP request line rather >> than an HTTP status (response) line? The following >> is the stack trace. >> >> From my old C++ days (may they never return), I >> conjectured that some string buffer is being reused >> or something like that.; We are running on CCL >> (formerly openmcl), and nothing like that is >> going on at the CCL level.; I don't know much >> about flexi-streams or gray-streams. Anyway >> that's just a rather wild guess. Thanks! >> >> > Arguments: (STREAM &OPTIONAL DRAKMA::LOG-STREAM) >> STREAM: >> DRAKMA::LOG-STREAM: NIL >> Local bindings: >> CHUNGA:CURRENT-ERROR-MESSAGE "While reading status line:"" >> DRAKMA::LINE: ""POST /stat/ping HTTP/1.1"" >> DRAKMA::FIRST-SPACE-POS: 4 >> DRAKMA::SECOND-SPACE-POS: 15 >> >> Thanks! >> >> - Dan >> >> >> >> >> _______________________________________________ >> drakma-devel mailing list >> drakma-devel at common-lisp.net >> http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel >> >> From lan at academsoft.ru Sat Mar 26 04:15:13 2011 From: lan at academsoft.ru (=?utf-8?B?0JDQu9C10LrRgdCw0L3QtNGAINCb0LjRgtCy0LjQvdC+0LI=?=) Date: Sat, 26 Mar 2011 10:15:13 +0600 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: Message-ID: <9c467ffba269c7569d5c50733794042b@work.ac-sw.com> A long time ago (Dec 2007) there was a patch [1] for drakma to implement timeouts with sbcl. It was rejected with suggestion to post similar patch into usocket library. Totday, I start to learn lisp programing and I need http-request for my small program. I took drakma for this. But it does not support timeouts on sbcl. I use debian and it seems sbcl - is the best choice for me. By googling I found a patch for drakma. I repplaied it for 1.2.3 drakma. It seems to me, it still forth appling into main line of drakma. I did not investigate what happen with usocket patch and such long time shows - this patch needs to be applied anyway :-) 1. http://common-lisp.net/pipermail/drakma-devel/2007-December/000601.html -- Alexander Litvinov -------------- next part -------------- A non-text attachment was scrubbed... Name: sbcl.patch Type: text/x-diff Size: 4694 bytes Desc: not available URL: From edi at weitz.de Sat Mar 26 10:44:20 2011 From: edi at weitz.de (Edi Weitz) Date: Sat, 26 Mar 2011 11:44:20 +0100 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <9c467ffba269c7569d5c50733794042b@work.ac-sw.com> References: <9c467ffba269c7569d5c50733794042b@work.ac-sw.com> Message-ID: I think this is something that belongs into usocket and not into Drakma. I'm not monitoring usocket, but if it had the ability to optionally specify timeouts and deadlines for those implementations supporting these features (and making them no-ops on others), I'd happily integrate that into Drakma. I don't want to litter Drakma with conditional expression for various Lisps, though. Thanks, Edi. On Sat, Mar 26, 2011 at 5:15 AM, ????????? ???????? wrote: > A long time ago (Dec 2007) there was a patch [1] for drakma to implement > timeouts with sbcl. It was rejected with suggestion to post similar > patch into usocket library. > > Totday, I start to learn lisp programing and I need http-request for my > small program. I took drakma for this. But it does not support timeouts on > sbcl. I use debian and it seems sbcl - is the best choice for me. > > By googling I found a patch for drakma. I repplaied it for 1.2.3 drakma. It > seems to me, it still forth appling into main line of drakma. I did not > investigate > what happen with usocket patch and such long time shows - this patch needs > to be applied anyway :-) > > 1. http://common-lisp.net/pipermail/drakma-devel/2007-December/000601.html > > -- > Alexander Litvinov > > > > _______________________________________________ > drakma-devel mailing list > drakma-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel > > From lan at academsoft.ru Sat Mar 26 17:17:33 2011 From: lan at academsoft.ru (=?utf-8?B?0JDQu9C10LrRgdCw0L3QtNGAINCb0LjRgtCy0LjQvdC+0LI=?=) Date: Sat, 26 Mar 2011 23:17:33 +0600 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: Message-ID: There was the same decision at Dec 2007. Now, three years later, you have not changed it. Anyway, this patch can be usefull for somebody else. ----------------???????? ?????????----------------- ??: "Edi Weitz" edi at weitz.de ????: "General interest list for Drakma and Chunga" drakma-devel at common-lisp.net CC(??????? ?????): "????????? ????????" lan at academsoft.ru ????: Sat, 26 Mar 2011 11:44:20 +0100 ------------------------------------------------- > I think this is something that belongs into usocket and not into > Drakma. I'm not monitoring usocket, but if it had the ability to > optionally specify timeouts and deadlines for those implementations > supporting these features (and making them no-ops on others), I'd > happily integrate that into Drakma. I don't want to litter Drakma > with conditional expression for various Lisps, though. > > Thanks, > Edi. > > > On Sat, Mar 26, 2011 at 5:15 AM, ????????? ???????? lan at academsoft.ru wrote: >> A long time ago (Dec 2007) there was a patch [1] for drakma to implement >> timeouts with sbcl. It was rejected with suggestion to post similar >> patch into usocket library. >> >> Totday, I start to learn lisp programing and I need http-request for my >> small program. I took drakma for this. But it does not support timeouts on >> sbcl. I use debian and it seems sbcl - is the best choice for me. >> >> By googling I found a patch for drakma. I repplaied it for 1.2.3 drakma. It >> seems to me, it still forth appling into main line of drakma. I did not >> investigate >> what happen with usocket patch and such long time shows - this patch needs >> to be applied anyway :-) >> >> 1. >> http://common-lisp.net/pipermail/drakma-devel/2007-December/000601 >> .html >> >> -- >> Alexander Litvinov >> >> >> >> _______________________________________________ >> drakma-devel mailing list >> drakma-devel at common-lisp.net >> http://common-lisp.net/cgi-bin/mailman/listinfo/drakma-devel >> >> > From edi at weitz.de Sat Mar 26 18:17:44 2011 From: edi at weitz.de (Edi Weitz) Date: Sat, 26 Mar 2011 19:17:44 +0100 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

Message-ID: On Sat, Mar 26, 2011 at 6:17 PM, ????????? ???????? wrote: > ?There was the same decision at Dec 2007. Now, three years later, you > have not changed it. Well, I don't think it's too bad not to change your mind every year... :) Actually, I had hoped that there had been some development in usocket since. From ehuels at gmail.com Sat Mar 26 19:11:37 2011 From: ehuels at gmail.com (Erik Huelsmann) Date: Sat, 26 Mar 2011 20:11:37 +0100 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

Message-ID: Hi Edi, 2011/3/26 Edi Weitz : > On Sat, Mar 26, 2011 at 6:17 PM, ????????? ???????? wrote: >> ?There was the same decision at Dec 2007. Now, three years later, you >> have not changed it. > > Well, I don't think it's too bad not to change your mind every year... :) > > Actually, I had hoped that there had been some development in usocket since. Well, there has been some development in usocket in the direction of supporting timeouts. On platforms where timeouts are not supported, an UNSUPPORTED-FEATURE condition is signalled. If that's ignored, everything should just continue without the timeout settings. If the above is not good enough, could you indicate what more needs to happen to make it so? Bye, Erik. From binghe.lisp at gmail.com Sun Mar 27 15:07:53 2011 From: binghe.lisp at gmail.com (Chun Tian (binghe)) Date: Sun, 27 Mar 2011 23:07:53 +0800 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

Message-ID: <0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> Hi, Erik Could you tell me what exact timeout feature they want? Setting Read/Write timeout when doing SOCKET-CONNECT? or Apply new timeout values to exist usocket object? --binghe ? 2011-3-27?03:11? Erik Huelsmann ??? > Hi Edi, > > 2011/3/26 Edi Weitz : >> On Sat, Mar 26, 2011 at 6:17 PM, ????????? ???????? wrote: >>> There was the same decision at Dec 2007. Now, three years later, you >>> have not changed it. >> >> Well, I don't think it's too bad not to change your mind every year... :) >> >> Actually, I had hoped that there had been some development in usocket since. > > Well, there has been some development in usocket in the direction of > supporting timeouts. On platforms where timeouts are not supported, an > UNSUPPORTED-FEATURE condition is signalled. If that's ignored, > everything should just continue without the timeout settings. > > If the above is not good enough, could you indicate what more needs to > happen to make it so? > > > Bye, > > > > Erik. From ehuels at gmail.com Sun Mar 27 15:20:10 2011 From: ehuels at gmail.com (Erik Huelsmann) Date: Sun, 27 Mar 2011 17:20:10 +0200 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> Message-ID: Hi Chun, 2011/3/27 Chun Tian (binghe) : > Hi, Erik > > Could you tell me what exact timeout feature they want? Setting Read/Write timeout when doing SOCKET-CONNECT? or Apply new timeout values to exist usocket object? As far as I understand, the request is to for SOCKET-CONNECT to be cancelled after a certain timeout period expires. I don't think that's a read/write timeout, but rather a way to limit the waiting time until a valid connection is established. Bye, Erik. > ? 2011-3-27?03:11? Erik Huelsmann ??? > >> Hi Edi, >> >> 2011/3/26 Edi Weitz : >>> On Sat, Mar 26, 2011 at 6:17 PM, ????????? ???????? wrote: >>>> ?There was the same decision at Dec 2007. Now, three years later, you >>>> have not changed it. >>> >>> Well, I don't think it's too bad not to change your mind every year... :) >>> >>> Actually, I had hoped that there had been some development in usocket since. >> >> Well, there has been some development in usocket in the direction of >> supporting timeouts. On platforms where timeouts are not supported, an >> UNSUPPORTED-FEATURE condition is signalled. If that's ignored, >> everything should just continue without the timeout settings. >> >> If the above is not good enough, could you indicate what more needs to >> happen to make it so? >> >> >> Bye, >> >> >> >> Erik. > > From edi at weitz.de Sun Mar 27 17:48:34 2011 From: edi at weitz.de (Edi Weitz) Date: Sun, 27 Mar 2011 19:48:34 +0200 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

Message-ID: Hi Erik, On Sat, Mar 26, 2011 at 8:11 PM, Erik Huelsmann wrote: > Well, there has been some development in usocket in the direction of > supporting timeouts. On platforms where timeouts are not supported, an > UNSUPPORTED-FEATURE condition is signalled. If that's ignored, > everything should just continue without the timeout settings. > > If the above is not good enough, could you indicate what more needs to > happen to make it so? I didn't know that usocket supports timeouts now. I was assuming it didn't because the OP resubmitted an old patch. >From your description it sounds as if this should be exactly what Drakma needs, so if someone wants to send a patch to integrate this usocket functionality (assuming this is in a released version) with Drakma, please go ahead. Thanks, Edi. From edi at weitz.de Sun Mar 27 17:50:43 2011 From: edi at weitz.de (Edi Weitz) Date: Sun, 27 Mar 2011 19:50:43 +0200 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> Message-ID: 2011/3/27 Chun Tian (binghe) : > Could you tell me what exact timeout feature they want? Setting Read/Write timeout when doing SOCKET-CONNECT? or Apply new timeout values to exist usocket object? The LW version of Drakma supports connection timeouts, read timeouts, and write timeouts. Existing usocket objects wouldn't be modified. Cheers, Edi. From binghe.lisp at gmail.com Sun Mar 27 18:24:58 2011 From: binghe.lisp at gmail.com (Chun Tian (binghe)) Date: Mon, 28 Mar 2011 02:24:58 +0800 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> Message-ID: <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings: 1. When doing TCP work, it means a "connection timeout"; 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP); The TIMEOUT keyword arguments was added from 0.4.0 (released on Oct 2008, more than two years ago), so I think Drakma should try to use it. However, it may not work on all supported platforms -- I remembered. But this should be considered as bug, I'll try to fix it after a small rewrite of our unit test framework. On the other hand, for "read timeout" and "write timeout", they're changeable properties (or options) for any socket objects. A new API called SOCKET-OPTION (learnt from CLISP) should be added in next USOCKET major version. Dynamically setting options like "broadcast" and "reuse address" should also be added under SOCKET-OPTION. I see hunchentoot has a function called SET-TIMEOUTS, which can be used to set "read timeout" and "write timeout" for any exist usocket, I want to merge it into usocket under the new SOCKET-OPTION API. --binghe ? 2011-3-28?01:50? Edi Weitz ??? > 2011/3/27 Chun Tian (binghe) : > >> Could you tell me what exact timeout feature they want? Setting Read/Write timeout when doing SOCKET-CONNECT? or Apply new timeout values to exist usocket object? > > The LW version of Drakma supports connection timeouts, read timeouts, > and write timeouts. Existing usocket objects wouldn't be modified. > > Cheers, > Edi. From nikodemus at random-state.net Sun Mar 27 20:26:45 2011 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Sun, 27 Mar 2011 23:26:45 +0300 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> Message-ID: 2011/3/27 Chun Tian (binghe) : > Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings: > > 1. When doing TCP work, it means a "connection timeout"; > 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP); For SBCL backend at least this is not correct. There :TIMEOUT is a read timeout for any single blocking read from the socket. Connections and writes are unaffected -- though SBCL changes may apply the timeouts to writes as well. Waiting for a connection is not affected -- but adding support for that is pretty easy. Sidenote: I haven't really looked at usocket's SBCL backend beyond SOCKET-CONNECT prior to this, but a quick peek shows that it's pretty cavalier about using SBCL internals. This is not good. Internals _will_ change, and then usocket will break. Specifically, SB-UNIX package is an internal implementation package -- using it is bad. SB-POSIX is the supported POSIX api. Some SB-FOO::BAR stuff in there as well. :/ Cheers, -- Nikodemus From binghe.lisp at gmail.com Sun Mar 27 21:47:48 2011 From: binghe.lisp at gmail.com (Chun Tian (binghe)) Date: Mon, 28 Mar 2011 05:47:48 +0800 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> Message-ID: <76FE19AC-6165-4045-AEA9-C38F79F06FAF@gmail.com> Hi, Nikodemus ? 2011-3-28?04:26? Nikodemus Siivola ??? > 2011/3/27 Chun Tian (binghe) : > >> Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings: >> >> 1. When doing TCP work, it means a "connection timeout"; >> 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP); > > For SBCL backend at least this is not correct. I see. But for other backends (at least LW and CCL), seems correct. Among all CL platforms, Clozure CL have the most feature-rich networking API. Its MAKE-SOCKET [1] provided three keyword arguments: INPUT-TIMEOUT, OUTPUT-TIMEOUT and CONNECT-TIMEOUT, and an additional DEADLINE. LispWorks only has TCP support, but its OPEN-TCP-STREAM [2] also provided three keyword arguments: READ-TIMEOUT, WRITE-TIMEOUT, and TIMEOUT. [1] http://ccl.clozure.com/manual/chapter7.2.html [2] http://www.lispworks.com/documentation/lw60/LW/html/lw-553.htm#pgfId-888355 > > There :TIMEOUT is a read timeout for any single blocking read from the > socket. Connections and writes are unaffected -- though SBCL changes > may apply the timeouts to writes as well. > > Waiting for a connection is not affected -- but adding support for > that is pretty easy. Then I suggest adding these missing part, at lest adjust the meaning of "TIMEOUT" argument of SOCKET-MAKE-STREAM into "connection timeout", and add READ-TIME / WRITE-TIMEOUT support. > > Sidenote: I haven't really looked at usocket's SBCL backend beyond > SOCKET-CONNECT prior to this, but a quick peek shows that it's pretty > cavalier about using SBCL internals. This is not good. Internals > _will_ change, and then usocket will break. Specifically, SB-UNIX > package is an internal implementation package -- using it is bad. > SB-POSIX is the supported POSIX api. Some SB-FOO::BAR stuff in there > as well. :/ Well, I think maybe the two projects (USOCKET and SBCL) didn't collaborate quite well in past years. USOCKET authors (at least me) want to support as most versions of SBCL as possible. This means, if we ask SBCL to add all necessary support to make things on usocket side become trivial, then we will loose support for older SBCL versions. Sometimes we have to do some dirty work (all around SBCL backend code) to detect if a "feature" exist in a SBCL version... I admit this is not good, but consider SBCL's current networking API lack of many features, and SBCL's release cycle is too fast to make all in-use SBCL versions quickly become old, I don't think a full rewrite of usocket's SBCL backend is possible at current stage. But if SBCL official can start to improve its networking part from now on, after maybe one year, or after at least 10 new SBCL releases, maybe we can force switch to a new backend with "good code" without using any internals. Glad to know your thoughts on this. Regards, --binghe > > Cheers, > > -- Nikodemus From binghe.lisp at gmail.com Mon Mar 28 17:28:05 2011 From: binghe.lisp at gmail.com (Chun Tian (binghe)) Date: Tue, 29 Mar 2011 01:28:05 +0800 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> Message-ID: <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> Hi, Nikodemus Today I think out another way to solve the SBCL connection timeout issue, I wrap a SB-EXT:WITH-TIMEOUT on SB-BSD-SOCKET:SOCKET-CONNNECT [1], and the result work seems working well: * (time (ignore-errors (usocket:socket-connect "8.8.8.8" 80 :timeout 3))) Evaluation took: 3.010 seconds of real time 0.010234 seconds of total run time (0.009223 user, 0.001011 system) 0.33% CPU 32 lambdas converted 6,608,582,602 processor cycles 916,672 bytes consed NIL # Do you think this is an acceptable solution? --binghe [1] http://trac.common-lisp.net/usocket/changeset/589 ? 2011-3-28?04:26? Nikodemus Siivola ??? > 2011/3/27 Chun Tian (binghe) : > >> Well, I think the TIMEOUT keyword arguments on SOCKET-CONNECT currently has following meanings: >> >> 1. When doing TCP work, it means a "connection timeout"; >> 2. When doing UCP work, it sets the default "read timeout" (because "connection timeout" is meaningless for UDP); > > For SBCL backend at least this is not correct. > > There :TIMEOUT is a read timeout for any single blocking read from the > socket. Connections and writes are unaffected -- though SBCL changes > may apply the timeouts to writes as well. > > Waiting for a connection is not affected -- but adding support for > that is pretty easy. > > Sidenote: I haven't really looked at usocket's SBCL backend beyond > SOCKET-CONNECT prior to this, but a quick peek shows that it's pretty > cavalier about using SBCL internals. This is not good. Internals > _will_ change, and then usocket will break. Specifically, SB-UNIX > package is an internal implementation package -- using it is bad. > SB-POSIX is the supported POSIX api. Some SB-FOO::BAR stuff in there > as well. :/ > > Cheers, > > -- Nikodemus From dlw at itasoftware.com Mon Mar 28 19:47:00 2011 From: dlw at itasoftware.com (Daniel Weinreb) Date: Mon, 28 Mar 2011 15:47:00 -0400 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

Message-ID: <4D90E5B4.6010304@itasoftware.com> Edi Weitz wrote: I think this is something that belongs into usocket and not into Drakma. I'm not monitoring usocket, but if it had the ability to optionally specify timeouts and deadlines for those implementations supporting these features (and making them no-ops on others), I'd happily integrate that into Drakma. I don't want to litter Drakma with conditional expression for various Lisps, though. and in later mail: > > I didn't know that usocket supports timeouts now. But in the latest Drakma, in http-request, there is a :deadline argument, which is passed through to usocket:socket-connect. Unfortunately, it is specific to CCL (OpenMCL). Drakma also knows, for CCL, to set the ccl:stream-deadline of the stream, too, so that the deadline applies not only to the connect work itself, but to the entire connection. This is quite useful for anyone writing a program that needs to have end-to-end timeouts. Servers usually need this more than clients, in my experience. We use this in our server for that reason: every request is given a deadline, and we have to make sure that anything that might block for input or output will honor the deadline. For Lispworks, I see that there is a different set of keyword arguments, that have "timeout" in their names. I completely sympathize with your not wanting to litter Drakma with implemenation-specific code. It would be good to get as much of this out of Drakma as possible. For Lispworks this may be difficult; I notice that Hunchentoot also needs to have a lot of Lispworks-specfic code. However, it looks to me as if usocket:socket-connect does support a :timeout argument for SBCL, as well as for CCL (OpenMCL). Also abcl and mcl, but not Allegro, cmucl, scl, nor clisp. Presumably this is because the underling implementation doens't support it; like many things, it's not part of the CL standard. Erik says, about usocket: Well, there has been some development in usocket in the direction of supporting timeouts. On platforms where timeouts are not supported, an UNSUPPORTED-FEATURE condition is signalled. If that's ignored, everything should just continue without the timeout settings. Indeed, if you look at socket-connect for SBCL,if the deadline argument is not null, it signals a condition to say that the feature is unsupported. socket-connect does, however, support the :timeout argument. I don't know SBCL can't accept :deadline, subtract from it the current time, and then use that as a :timeout, but maybe there's a good reason for that. The documentation of socket-connect, which is in the file usocket.lisp, doesn't document all of the arguments to socket-connect, so I might be missing something about the semantics. -- Dan From nikodemus at random-state.net Mon Mar 28 19:59:47 2011 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Mon, 28 Mar 2011 22:59:47 +0300 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <4D90E5B4.6010304@itasoftware.com> References:

<4D90E5B4.6010304@itasoftware.com> Message-ID: On 28 March 2011 22:47, Daniel Weinreb wrote: > I don't know SBCL can't accept :deadline, > subtract from it the current time, and then use that > as a :timeout, but maybe there's a good reason for Not really. The timeout becomes the stream timeout, which is the bound given to waiting for new input when a buffer needs to be refilled. Adding stream deadline support for SBCL needs to happen first, really, unless usocket wants to implement streams from scratch or wrap fd streams with a grey stream that check the deadline. Cheers, -- Nikodemus From edi at weitz.de Mon Mar 28 20:18:33 2011 From: edi at weitz.de (Edi Weitz) Date: Mon, 28 Mar 2011 22:18:33 +0200 Subject: [drakma-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <4D90E5B4.6010304@itasoftware.com> References:

<4D90E5B4.6010304@itasoftware.com> Message-ID: Hi Dan! On Mon, Mar 28, 2011 at 9:47 PM, Daniel Weinreb wrote: > For Lispworks this may > be difficult; I notice that Hunchentoot also needs > to have a lot of Lispworks-specfic code. That's easy to explain: I wrote all of these libraries for my own professional usage originally and I use LispWorks. In some cases I thought the code might be useful for others and so I added compatibility code for other Lisps. However, this usually entails adding a lot of compatibility libraries which means more things the original library depends on and more code you need to add if you deliver something to your clients. It would be easy to get rid of the LispWorks-specific code and have everything use usocket and such, but for the reasons mentioned I don't want that. Therefore, my policy usually is to keep a lean LispWorks kernel and if possible have others take care of other Lisps - in a way that doesn't make the code base unnecessarily complicated. The deadline stuff is special because ITA specifically asked for it and Hans added it after me grudgingly accepting this change. If this could be rolled into usocket as well (even if it's a no-op for all Lisps except ClozureCL), I'd appreciate that. Cheers, Edi. From nikodemus at random-state.net Mon Mar 28 20:35:14 2011 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Mon, 28 Mar 2011 23:35:14 +0300 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> Message-ID: 2011/3/28 Chun Tian (binghe) : > Today I think out another way to solve the SBCL connection timeout issue, I wrap a > SB-EXT:WITH-TIMEOUT on SB-BSD-SOCKET:SOCKET-CONNNECT [1], and the result work seems working well: That's along the lines I was thinking off, except that SB-EXT:WITH-TIMEOUT is a broken construct. (Soon to be deprecated, in all likelihood.) Consider this: (with-timeout 1.0 (handler-case (with-timeout 4.0 (sleep 2)) (sb-ext:timeout ()))) which is to say that you cannot distinguish an outer timeout from an inner one, which is bad. You need something like this, instead: (defmacro with-timeout-handler ((seconds timeout-form) &body body) "Runs BODY as an implicit PROGN with timeout of SECONDS. If timeout occurs before BODY has finished, BODY is unwound and TIMEOUT-FORM is executed with its values returned instead. Note that BODY is unwound asynchronously when a timeout occurs, so unless all code executed during it -- including anything down the call chain -- is asynch unwind safe, bad things will happen. Use with care." (alexandria:with-gensyms (exec unwind timer timeout block) `(block ,block (tagbody (flet ((,unwind () (go ,timeout)) (,exec () , at body)) (declare (dynamic-extent #',exec #',unwind)) (let ((,timer (sb-ext:make-timer #',unwind))) (declare (dynamic-extent ,timer)) (sb-sys:without-interrupts (unwind-protect (progn (sb-ext:schedule-timer ,timer ,seconds) (return-from ,block (sb-sys:with-local-interrupts (,exec)))) (sb-ext:unschedule-timer ,timer))))) ,timeout (return-from ,block ,timeout-form))))) with which (with-timeout-handler (1.0 :outer) (with-timeout-handler (4.0 :inner) (sleep 10.0) :ok)) does the right thing. Gods, I hate asynch timeouts. Is there a sane way to tell connect() to time out without needing SIGALRM? Cheers, -- Nikodemus From ehuels at gmail.com Mon Mar 28 20:42:12 2011 From: ehuels at gmail.com (Erik Huelsmann) Date: Mon, 28 Mar 2011 22:42:12 +0200 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> Message-ID: > Gods, I hate asynch timeouts. Is there a sane way to tell connect() to > time out without needing SIGALRM? Nope. Have you considered using a non-blocking connect() call, combining that with a select() call with a timeout? Bye, Erik. From nikodemus at random-state.net Mon Mar 28 20:52:15 2011 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Mon, 28 Mar 2011 23:52:15 +0300 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> Message-ID: On 28 March 2011 23:42, Erik Huelsmann wrote: >> Gods, I hate asynch timeouts. Is there a sane way to tell connect() to >> time out without needing SIGALRM? > > Nope. Have you considered using a non-blocking connect() call, > combining that with a select() call with a timeout? If that's the way it needs to be done, then that's the way it needs to be done... But realistically, I'm not going to have time to work on that in near future unless things suddenly change. Cheers, -- Nikodemus From binghe.lisp at gmail.com Mon Mar 28 22:43:47 2011 From: binghe.lisp at gmail.com (Chun Tian (binghe)) Date: Tue, 29 Mar 2011 06:43:47 +0800 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> Message-ID: <123DC7B5-AEB2-46B7-9BAF-C4F0CB672281@gmail.com> I know SBCL's WITH-TIMEOUT cannot nest, I learn this from GBBopen's portable-threads.lisp [1], and it also give a nested version SBCL's WITH-TIMEOUT, much shorter than yours: #+sbcl (defmacro with-timeout ((seconds &body timeout-body) &body timed-body) (let ((tag-sym (gensym)) (timer-sym (gensym))) `(block ,tag-sym (let ((,timer-sym (sb-ext:make-timer #'(lambda () (return-from ,tag-sym (progn , at timeout-body)))))) (sb-ext:schedule-timer ,timer-sym ,seconds) (unwind-protect (progn , at timed-body) (sb-ext:unschedule-timer ,timer-sym)))))) I didn't use this version simply because I think the WITH-TIMEOUT form in usocket's SOCKET-CONNECT has no chance to be nested. --binghe ? 2011-3-29?04:35? Nikodemus Siivola ??? > 2011/3/28 Chun Tian (binghe) : > >> Today I think out another way to solve the SBCL connection timeout issue, I wrap a >> SB-EXT:WITH-TIMEOUT on SB-BSD-SOCKET:SOCKET-CONNNECT [1], and the result work seems working well: > > That's along the lines I was thinking off, except that > SB-EXT:WITH-TIMEOUT is a broken construct. (Soon to be deprecated, in > all likelihood.) > > Consider this: > > (with-timeout 1.0 (handler-case (with-timeout 4.0 (sleep 2)) > (sb-ext:timeout ()))) > > which is to say that you cannot distinguish an outer timeout from an > inner one, which is bad. > > You need something like this, instead: > > (defmacro with-timeout-handler ((seconds timeout-form) &body body) > "Runs BODY as an implicit PROGN with timeout of SECONDS. If > timeout occurs before BODY has finished, BODY is unwound and > TIMEOUT-FORM is executed with its values returned instead. > > Note that BODY is unwound asynchronously when a timeout occurs, > so unless all code executed during it -- including anything > down the call chain -- is asynch unwind safe, bad things will > happen. Use with care." > (alexandria:with-gensyms (exec unwind timer timeout block) > `(block ,block > (tagbody > (flet ((,unwind () > (go ,timeout)) > (,exec () > , at body)) > (declare (dynamic-extent #',exec #',unwind)) > (let ((,timer (sb-ext:make-timer #',unwind))) > (declare (dynamic-extent ,timer)) > (sb-sys:without-interrupts > (unwind-protect > (progn > (sb-ext:schedule-timer ,timer ,seconds) > (return-from ,block > (sb-sys:with-local-interrupts > (,exec)))) > (sb-ext:unschedule-timer ,timer))))) > ,timeout > (return-from ,block ,timeout-form))))) > > with which > > (with-timeout-handler (1.0 :outer) (with-timeout-handler (4.0 > :inner) (sleep 10.0) :ok)) > > does the right thing. > > Gods, I hate asynch timeouts. Is there a sane way to tell connect() to > time out without needing SIGALRM? > > Cheers, > > -- Nikodemus From nikodemus at random-state.net Tue Mar 29 04:19:09 2011 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Tue, 29 Mar 2011 07:19:09 +0300 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <123DC7B5-AEB2-46B7-9BAF-C4F0CB672281@gmail.com> References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> <3A25E337-2050-45A5-B7E3-7E99F2D31475@gmail.com> <123DC7B5-AEB2-46B7-9BAF-C4F0CB672281@gmail.com> Message-ID: 2011/3/29 Chun Tian (binghe) : > I know SBCL's WITH-TIMEOUT cannot nest, I learn this from GBBopen's portable-threads.lisp [1], and > it also give a nested version SBCL's WITH-TIMEOUT, much shorter than yours: Amusingly, neither SBCL's own, nor GBBopen's WITH-TIMEOUT is asynch unwind safe. The one I posted is -- that's what the WITHOUT-INTERRUPTS and WITH-LOCAL-INTERRUPTS were for. :) But yeah, it's miles saner than the SB-EXT:WITH-TIMEOUT. The GBBopen's WITH-TIMEOUT also suffers from the minor defect that the dynamic context (special variables, handlers, restarts, etc) the timeout-body runs in is unpredictable -- for most cases this does not matter, but it can lead to hard to reproduce bugs. > I didn't use this version simply because I think the WITH-TIMEOUT form in usocket's SOCKET-CONNECT > has no chance to be nested. It has. A function in a library cannot know if it will be nested in some dynamic context or not. The corrollary is that library code should never use SB-EXT;WITH-TIMEOUT. (defun foo () ...stuff that uses socket-connect with a local timeout > 1.0 somewhere in its guts...) (with-timeout 1.0 (handler-case (foo) (error () :bad-foo))) If the fetch takes longer than intended, the outer timeout can get converted into an USOCKET-TIMEOUT-ERROR (whatever the exact name), which in turn gets swallowed by the HANDLER-CASE. Cheers, -- Nikodemus From nikodemus at random-state.net Tue Mar 29 04:38:09 2011 From: nikodemus at random-state.net (Nikodemus Siivola) Date: Tue, 29 Mar 2011 07:38:09 +0300 Subject: [drakma-devel] [usocket-devel] [patch] Resubmit drakma timeout for sbcl. In-Reply-To: <76FE19AC-6165-4045-AEA9-C38F79F06FAF@gmail.com> References:

<0763B497-9103-4DB8-A672-4B21C697DCDA@gmail.com> <3147912F-BDEA-4904-A7A0-0FF6ADD47A5D@gmail.com> <76FE19AC-6165-4045-AEA9-C38F79F06FAF@gmail.com> Message-ID: 2011/3/28 Chun Tian (binghe) : > Well, I think maybe the two projects (USOCKET and SBCL) didn't collaborate > quite well in past years. USOCKET authors (at least me) want to support as > most versions of SBCL as possible. ?This means, if we ask SBCL to add all > necessary support to make things on usocket side become trivial, then we will > loose support for older SBCL versions. ?Sometimes we have to do some dirty > work (all around SBCL backend code) to detect if a "feature" exist in a SBCL > version... While you can try to support as many versions as possible, by using eg. SB-UNIX:FAST-SELECT you're virtually guaranteeing that at some unspecified date an SBCL update will break usocket. (Sooner or later SB-UNIX:SELECT will probably go away and be SB-UNIX:FAST-SELECT will be renamed SB-UNIX:SELECT. For that matter, because SB-UNIX is one of the more common causes of "things break due to an SBCL update because someone used undocumented internals", we might rename the whole package something less tempting. Of course it's unrealistic for usocket to stop using stuff it has grown up using tomorrow... but I think trying to support old SBCL versions indefinitely in future usocket releases is a failing strategy. The manpower to take write and maintain the different versions just isn't there. I would suggested that usocket should happily use as new SBCL versions as it wants, and tells users of older SBCL versions to either update SBCL not update usocket... (Or donate their time or money to maintain the temporal portability if they really must be able to use new usocket releases with old SBCL versions.) As for what SBCL can do to support usocket better: Generally speaking, if you tell us that you need X, and X is either impossible or irritating to write using supported interfaces, we can either provide you with a supported X or support necessary lower-level interfaces. X can be anything, but generally it is easier to expose low-level functionality that gives you more rope than trying to get a high-level interface right the first time. If X is "a better networking substrate" ... it's going to take a while. :) Easier things can happen very quickly, more specific ones are typically easier. Cheers, -- Nikodemus