From marc.battyani at fractalconcept.com Fri Dec 8 21:38:01 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Fri, 8 Dec 2006 22:38:01 +0100 Subject: [mod-lisp-devel] Re: mod_lisp: broken pipe (UNIX::SIGPIPE-HANDLER) errors References: <4579AD5A.9050700@banrai.com> Message-ID: <017f01c71b11$25b9dc40$1402a8c0@marcx2> Hi Denis, [cc'ed to the mod-lisp-devel mailing list] > I'm using CMUCL-19c (running under detachtty), with mod_lisp and > Apache1.3 for a web project. > > For the most part, things work as expected, but occasionally the lisp > process crashes like this: > >> Error in function UNIX::SIGPIPE-HANDLER: SIGPIPE at #x6E87A2. >> > > When I run a backtrace in the debugger, I can see that the server > request had been filled, i.e. the lisp process is ready to write an html > string to *apache-stream* in reply, but it fails with the error shown > above. > > To make the connection to Apache, I used the code in > "modlisp-cmucl.lisp" with some minor modifications (attached). > > Have you seen this problem before, or can you suggest what to look at > exactly within the apache-listener code? I don't use CMUCL but there was some cases of Apache closing the socket which caused a similar error when Lisp was writting to the socket. IIRC this was fixed a long time ago. Are you sure to use the latest mod_lisp version? (from the repository) Marc From marc.battyani at fractalconcept.com Fri Dec 8 22:58:31 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Fri, 8 Dec 2006 23:58:31 +0100 Subject: [mod-lisp-devel] Re: mod_lisp: broken pipe (UNIX::SIGPIPE-HANDLER) errors References: <4579AD5A.9050700@banrai.com> <017f01c71b11$25b9dc40$1402a8c0@marcx2> <4579E295.9070504@banrai.com> Message-ID: <01ab01c71b1c$64e1c3a0$1402a8c0@marcx2> "Denis Papathanasiou" wrote: > Marc Battyani wrote: [cc'ed to the mod-lisp-devel mailing list] > Many thanks for letting me know about the mod-lisp-devel mailing list (I > didn't realize there was one!). > >> I don't use CMUCL but there was some cases of Apache closing the socket >> which caused a similar error when Lisp was writting to the socket. IIRC > > I did find a reference to "Odd behavior (socket leak)" in that mailing > list (http://common-lisp.net/pipermail/tbnl-devel/2004-July/000028.html) > but his complaint was different: it was opening multiple sockets for each > request w/o closing any of the old ones. That's not happening with our > installation. > >> this was fixed a long time ago. Are you sure to use the latest mod_lisp >> version? (from the repository) > > Yes; according to the logs, we built it on Feb. 11, 2006 using > MOD_LISP_VERSION "2.43" (Revision 120: /mod_lisp) from > http://www.fractalconcept.com:8000/public/open-source/mod_lisp/ > > I'll keep searching that mailing list (unfortunately, I'm doing it > indirectly with Google b/c it doesn't appear to be searchable -- is there > a mirror on gmane or someplace similar?) but let me know if there's a > specific month/year you can point me to. It's this fix: Version 2.3 Force Apache to read all the Lisp reply before eventually closing the socket or handling another request. This avoids trying to write to a closed socket or having Lisp and Apache out of sync. (contributed by Edi Weitz) Marc From marc.battyani at fractalconcept.com Fri Dec 8 23:19:30 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Sat, 9 Dec 2006 00:19:30 +0100 Subject: [mod-lisp-devel] Re: mod_lisp: broken pipe (UNIX::SIGPIPE-HANDLER) errors References: <4579AD5A.9050700@banrai.com> <017f01c71b11$25b9dc40$1402a8c0@marcx2> <4579E295.9070504@banrai.com> <01ab01c71b1c$64e1c3a0$1402a8c0@marcx2> <4579F18C.2020508@banrai.com> Message-ID: <01f601c71b1f$52e2a630$1402a8c0@marcx2> "Denis Papathanasiou" wrote: > Marc Battyani wrote: >> It's this fix: >> >> Version 2.3 >> Force Apache to read all the Lisp reply before eventually closing the >> socket or handling another request. >> This avoids trying to write to a closed socket or having Lisp and Apache >> out of sync. >> (contributed by Edi Weitz) > > Hmmm... so we have Edi's fix incorporated in the version we're already > using. > > I did look a bit further in the mailing list and saw your conversation > with him about lisp possibly taking too long for Apache, then timing out > (on the Apache side). > > I looked in the Apache error log and found entries like this: > >> [error] (70007)The timeout specified has expired: error reading from Lisp > > but in the backtrace, the lisp reply (in the form the html string) was > fully formed. > > If it is related to an Apache timeout b/c our lisp process (outside of > mod_lisp) took too long, wouldn't we land in the debugger *before* the > reply could become fully formed? Well it looks like you are one the good way to find out. :) It's probably some mismatch between the Lisp and Apache. Maybe the content-size is not right. Have you tried to set it to -1 so that Apache closes the socket after each reply, just to check? Marc From hughw at hughw.net Wed Dec 20 21:24:58 2006 From: hughw at hughw.net (Hugh Winkler) Date: Wed, 20 Dec 2006 15:24:58 -0600 Subject: [mod-lisp-devel] Chunked encoding Message-ID: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com> Hi all, I see this topic was raised a couple of months ago : http://common-lisp.net/pipermail/mod-lisp-devel/2006-October/thread.html#99. I've made changes to mod_lisp2.c to support chunked encoding. Sorry, I only have a 2.0.55 to run against atm, so I have not backported to 1.3. Notes: Apache 2 has eliminated REQUEST_CHUNKED_DECHUNK, meaning there's no way to receive chunked data ? Apache will always dechunkify it. I considered signaling the end of the stream by sending MSG_OOB. Well, apr does not support that. So the only option left was to re-chunk the data. I opted to re-chunk according to RFC 2616 so that on the Lisp side we can use Dr. Weitz' Chunga[1] to de-chunk. The new behavior is compatible with existing code on the Lisp side e.g. Kevin Rosenberg's modlisp. The mod_lisp2 code only chunks if the Content-length header is omitted; formerly, Lisp side code could not work properly if Content-length was omitted anyway. There is one maybe extraneous change in this patch: I eliminated some gcc compiler warnings issuing from local_lisp_cfg(). Here is my gcc version string: (GCC) 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5). This change is not essential but it's nice to build with no warnings. Thanks Mark for mod_lisp... sorry I'm no help on the 1.3 backport. Hugh [1] http://weitz.de/chunga/ --- ../libapache2-mod-lisp2-1.3.1-orig/mod_lisp2.c 2006-12-20 13:23:46.000000000 -0600 +++ mod_lisp2.c 2006-12-20 14:01:47.000000000 -0600 @@ -58,6 +58,11 @@ /* Change log: + If request contains chunked data, send chunks (length word plus data) + to backend lisp. Also eliminated a gcc warning. + -- Hugh Winkler + 2006-12-05 + Fixed the APR 1.2.2 detection. -- sent by several people 2006-1203 @@ -231,7 +236,7 @@ static header_map_t map_header_to_lisp_header; static int write_client_data (request_rec * r, const char * data, unsigned int n_bytes); - + /* Configuration data */ static lisp_cfg_t * @@ -284,19 +289,19 @@ static lisp_cfg_t * local_lisp_cfg (lisp_cfg_t *cfg) { - lisp_cfg_t *local_cfg = NULL; - - apr_threadkey_private_get((void**)&local_cfg, cfg_key); + void *local_cfg = NULL; + + apr_threadkey_private_get(&local_cfg, cfg_key); if (local_cfg == NULL) { local_cfg = copy_lisp_cfg (socket_pool, cfg); - apr_threadkey_private_set((void*)local_cfg, cfg_key); + apr_threadkey_private_set(local_cfg, cfg_key); return local_cfg; } check_cfg_for_reuse(local_cfg, cfg); - return local_cfg; + return (lisp_cfg_t*) local_cfg; } #else lisp_cfg_t *local_cfg; @@ -433,6 +438,26 @@ } static apr_status_t +write_lisp_data_chunk (apr_socket_t * socket, + const char * data, unsigned int n_bytes) +{ + char crlf[2] = {0xd, 0xa}; + char length[16]; + snprintf(length, 16, "%x", n_bytes); + + apr_status_t status = write_lisp_data (socket, length, strlen(length)); + if ( status == APR_SUCCESS) + { + status = write_lisp_data (socket, crlf, 2); + if ( status == APR_SUCCESS && n_bytes) + status = write_lisp_data (socket, data, n_bytes); + if ( status == APR_SUCCESS) + status = write_lisp_data (socket, crlf, 2); + } + return status; +} + +static apr_status_t write_lisp_line (apr_socket_t * socket, const char * data) { RELAY_ERROR (write_lisp_data (socket, data, (strlen (data)))); @@ -555,6 +580,7 @@ int content_length = (-1); int keep_socket_p = 0; apr_socket_t * socket; + const char * request_content_length = 0; cfg = local_lisp_cfg(cfg); @@ -600,6 +626,8 @@ ((copy_headers ((r->headers_in), map_header_to_lisp_header, socket)), "writing to Lisp"); + request_content_length = apr_table_get(r->headers_in, "Content-Length"); + /* Send the end-of-headers marker. */ ML_LOG_DEBUG (r, "write end-of-headers"); CVT_ERROR ((write_lisp_line (socket, "end")), "writing to Lisp"); @@ -619,21 +647,31 @@ close_lisp_socket (cfg); return (HTTP_INTERNAL_SERVER_ERROR); } - if (n_read == 0) - break; + /* for chunked case, when nread == 0, we will write + * a terminating 0.*/ + { - apr_status_t status = (write_lisp_data (socket, buffer, n_read)); + apr_status_t status = APR_SUCCESS; + + /* if there's no Content-Type header, the data must be chunked */ + if (request_content_length == NULL) + status = write_lisp_data_chunk (socket, buffer, n_read); + else if (n_read != 0) + status = write_lisp_data (socket, buffer, n_read); + if (APR_SUCCESS != status) { - while ((ap_get_client_block (r, buffer, (sizeof (buffer)))) - > 0) + while ((ap_get_client_block (r, buffer, sizeof(buffer))) + > 0) ; ML_LOG_ERROR (status, r, "writing to Lisp"); close_lisp_socket (cfg); return (HTTP_INTERNAL_SERVER_ERROR); } } + if( n_read == 0) + break; } } From edi at agharta.de Wed Dec 20 23:09:28 2006 From: edi at agharta.de (Edi Weitz) Date: Thu, 21 Dec 2006 00:09:28 +0100 Subject: [mod-lisp-devel] Chunked encoding In-Reply-To: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com> (Hugh Winkler's message of "Wed, 20 Dec 2006 15:24:58 -0600") References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com> Message-ID: On Wed, 20 Dec 2006 15:24:58 -0600, "Hugh Winkler" wrote: > I've made changes to mod_lisp2.c to support chunked encoding. Nice, thanks. I'll integrate this with Hunchentoot as soon as I find the time. From marc.battyani at fractalconcept.com Thu Dec 21 08:21:42 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Thu, 21 Dec 2006 09:21:42 +0100 Subject: [mod-lisp-devel] Chunked encoding References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com> Message-ID: <087d01c724d9$0eeb3ff0$1402a8c0@marcx2> "Hugh Winkler" wrote: >I see this topic was raised a couple of months ago : >http://common-lisp.net/pipermail/mod-lisp-devel/2006-October/thread.html#99. > >I've made changes to mod_lisp2.c to support chunked encoding. Sorry, >I only have a 2.0.55 to run against atm, so I have not backported to >1.3. > >Notes: > >Apache 2 has eliminated REQUEST_CHUNKED_DECHUNK, meaning there's no >way to receive chunked data ? Apache will always dechunkify it. > >I considered signaling the end of the stream by sending MSG_OOB. Well, >apr does not support that. So the only option left was to re-chunk the >data. I opted to re-chunk according to RFC 2616 so that on the Lisp >side we can use Dr. Weitz' Chunga[1] to de-chunk. > >The new behavior is compatible with existing code on the Lisp side >e.g. Kevin Rosenberg's modlisp. The mod_lisp2 code only chunks if the >Content-length header is omitted; formerly, Lisp side code could not >work properly if Content-length was omitted anyway. > >There is one maybe extraneous change in this patch: I eliminated some >gcc compiler warnings issuing from local_lisp_cfg(). Here is my gcc >version string: (GCC) 4.1.2 20060928 (prerelease) (Ubuntu >4.1.1-13ubuntu5). This change is not essential but it's nice to build >with no warnings. Thanks a lot for this new feature. :) I tried to apply your patch but subversion complains about its format. Can you send me the complete file or a svn compatible patch? >Thanks Mark for mod_lisp... sorry I'm no help on the 1.3 backport. Thanks for contributing. ;-) No problem for the 1.3 backport, I think that Apache 2 is available everywhere now. Marc From hughw at hughw.net Thu Dec 21 14:28:34 2006 From: hughw at hughw.net (Hugh Winkler) Date: Thu, 21 Dec 2006 08:28:34 -0600 Subject: [mod-lisp-devel] Chunked encoding In-Reply-To: <087d01c724d9$0eeb3ff0$1402a8c0@marcx2> References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com> <087d01c724d9$0eeb3ff0$1402a8c0@marcx2> Message-ID: <927441b30612210628p2a59fd21wd10d8fff2c32add5@mail.gmail.com> Hi Marc, On 12/21/06, Marc Battyani wrote: > Thanks a lot for this new feature. :) > I tried to apply your patch but subversion complains about its format. Can > you send me the complete file or a svn compatible patch? > I've attached the complete file. (I made the previous patch using diff -Naur, not svn). Regards, Hugh -------------- next part -------------- A non-text attachment was scrubbed... Name: mod_lisp2.c Type: text/x-csrc Size: 27718 bytes Desc: not available URL: From marc.battyani at fractalconcept.com Thu Dec 21 18:04:55 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Thu, 21 Dec 2006 19:04:55 +0100 Subject: [mod-lisp-devel] Chunked encoding References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com><087d01c724d9$0eeb3ff0$1402a8c0@marcx2> <927441b30612210628p2a59fd21wd10d8fff2c32add5@mail.gmail.com> Message-ID: <0a5401c7252a$88026160$1402a8c0@marcx2> "Hugh Winkler" wrote: > > On 12/21/06, Marc Battyani wrote: > >> Thanks a lot for this new feature. :) >> I tried to apply your patch but subversion complains about its format. >> Can >> you send me the complete file or a svn compatible patch? >> > > I've attached the complete file. (I made the previous patch using diff > -Naur, not svn). Thanks. I've put it on the repository. Marc From hughw at hughw.net Fri Dec 22 16:22:54 2006 From: hughw at hughw.net (Hugh Winkler) Date: Fri, 22 Dec 2006 10:22:54 -0600 Subject: [mod-lisp-devel] Re: Chunked encoding In-Reply-To: <927441b30612220819m292695b0l2a6580c038e299f2@mail.gmail.com> References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com> <927441b30612220819m292695b0l2a6580c038e299f2@mail.gmail.com> Message-ID: <927441b30612220822t17d823a3h8f8571047abac667@mail.gmail.com> the attachment is attached this time... :) On 12/22/06, Hugh Winkler wrote: > On 12/20/06, Hugh Winkler wrote: > > > I've made changes to mod_lisp2.c to support chunked encoding. > > As a guide, I've attached are some changes I have made to modlisp to > support consuming chunked data from mod_lisp2: > > ;; These are changes to Kevin Rosenberg's modlisp > ;; to consume chunked data. There's a > ;; change to package kmrcl to have the *modlisp-socket* > ;; created as a flexi-stream. > ;; ***n.b. this change has only been made for sbcl*** > ;; Then,if mod_lisp2.c has not sent a Content-length header, > ;; get-content-as-string() creates a chunga chunked > ;; input stream over the original stream; otherwise > ;; it just consumes the stream as before. > > ;; depends on chunga http://weitz.de/chunga > ;; and on flexi-streams http://weits.de/flexi-streams > > Thanks, Edi, for making this so easy with flex and chunga! > > Hugh > -------------- next part -------------- A non-text attachment was scrubbed... Name: modlisp2.lisp Type: application/octet-stream Size: 3065 bytes Desc: not available URL: