From marc.battyani at fractalconcept.com  Fri Dec  8 21:38:01 2006
From: marc.battyani at fractalconcept.com (Marc Battyani)
Date: Fri, 8 Dec 2006 22:38:01 +0100
Subject: [mod-lisp-devel] Re: mod_lisp: broken pipe (UNIX::SIGPIPE-HANDLER)
	errors
References: <4579AD5A.9050700@banrai.com>
Message-ID: <017f01c71b11$25b9dc40$1402a8c0@marcx2>

Hi Denis,

[cc'ed to the mod-lisp-devel mailing list]

> I'm using CMUCL-19c (running under detachtty), with mod_lisp and
> Apache1.3 for a web project.
>
> For the most part, things work as expected, but occasionally the lisp
> process crashes like this:
>
>> Error in function UNIX::SIGPIPE-HANDLER:  SIGPIPE at #x6E87A2.
>>
>
> When I run a backtrace in the debugger, I can see that the server
> request had been filled, i.e. the lisp process is ready to write an html
> string to *apache-stream* in reply, but it fails with the error shown 
> above.
>
> To make the connection to Apache, I used the code in
> "modlisp-cmucl.lisp" with some minor modifications (attached).
>
> Have you seen this problem before, or can you suggest what to look at
> exactly within the apache-listener code?

I don't use CMUCL but there was some cases of Apache closing the socket 
which caused a similar error when Lisp was writting to the socket. IIRC this 
was fixed a long time ago. Are you sure to use the latest mod_lisp version? 
(from the repository)

Marc 



From marc.battyani at fractalconcept.com  Fri Dec  8 22:58:31 2006
From: marc.battyani at fractalconcept.com (Marc Battyani)
Date: Fri, 8 Dec 2006 23:58:31 +0100
Subject: [mod-lisp-devel] Re: mod_lisp: broken pipe (UNIX::SIGPIPE-HANDLER)
	errors
References: <4579AD5A.9050700@banrai.com>
	<017f01c71b11$25b9dc40$1402a8c0@marcx2>
	<4579E295.9070504@banrai.com>
Message-ID: <01ab01c71b1c$64e1c3a0$1402a8c0@marcx2>

"Denis Papathanasiou" <denis at banrai.com> wrote:

> Marc Battyani wrote:

[cc'ed to the mod-lisp-devel mailing list]

> Many thanks for letting me know about the mod-lisp-devel mailing list (I 
> didn't realize there was one!).
>
>> I don't use CMUCL but there was some cases of Apache closing the socket 
>> which caused a similar error when Lisp was writting to the socket. IIRC
>
> I did find a reference to "Odd behavior (socket leak)" in that mailing 
> list (http://common-lisp.net/pipermail/tbnl-devel/2004-July/000028.html) 
> but his complaint was different: it was opening multiple sockets for each 
> request w/o closing any of the old ones.  That's not happening with our 
> installation.
>
>> this was fixed a long time ago. Are you sure to use the latest mod_lisp 
>> version? (from the repository)
>
> Yes; according to the logs, we built it on Feb. 11, 2006 using 
> MOD_LISP_VERSION "2.43" (Revision 120: /mod_lisp) from 
> http://www.fractalconcept.com:8000/public/open-source/mod_lisp/
>
> I'll keep searching that mailing list (unfortunately, I'm doing it 
> indirectly with Google b/c it doesn't appear to be searchable -- is there 
> a mirror on gmane or someplace similar?) but let me know if there's a 
> specific month/year you can point me to.

It's this fix:

Version 2.3
Force Apache to read all the Lisp reply before eventually closing the socket 
or handling another request.
This avoids trying to write to a closed socket or having Lisp and Apache out 
of sync.
(contributed by Edi Weitz)

Marc



From marc.battyani at fractalconcept.com  Fri Dec  8 23:19:30 2006
From: marc.battyani at fractalconcept.com (Marc Battyani)
Date: Sat, 9 Dec 2006 00:19:30 +0100
Subject: [mod-lisp-devel] Re: mod_lisp: broken pipe (UNIX::SIGPIPE-HANDLER)
	errors
References: <4579AD5A.9050700@banrai.com>
	<017f01c71b11$25b9dc40$1402a8c0@marcx2>
	<4579E295.9070504@banrai.com>
	<01ab01c71b1c$64e1c3a0$1402a8c0@marcx2>
	<4579F18C.2020508@banrai.com>
Message-ID: <01f601c71b1f$52e2a630$1402a8c0@marcx2>

"Denis Papathanasiou" <denis at banrai.com> wrote:
> Marc Battyani wrote:
>> It's this fix:
>>
>> Version 2.3
>> Force Apache to read all the Lisp reply before eventually closing the 
>> socket or handling another request.
>> This avoids trying to write to a closed socket or having Lisp and Apache 
>> out of sync.
>> (contributed by Edi Weitz)
>
> Hmmm... so we have Edi's fix incorporated in the version we're already 
> using.
>
> I did look a bit further in the mailing list and saw your conversation 
> with him about lisp possibly taking too long for Apache, then timing out 
> (on the Apache side).
>
> I looked in the Apache error log and found entries like this:
>
>> [error] (70007)The timeout specified has expired: error reading from Lisp
>
> but in the backtrace, the lisp reply (in the form the html string) was 
> fully formed.
>
> If it is related to an Apache timeout b/c our lisp process (outside of 
> mod_lisp) took too long, wouldn't we land in the debugger *before* the 
> reply could become fully formed?

Well it looks like you are one the good way to find out. :)
It's probably some mismatch between the Lisp and Apache.
Maybe the content-size is not right. Have you tried to set it to -1 so that 
Apache closes the socket after each reply, just to check?

Marc 



From hughw at hughw.net  Wed Dec 20 21:24:58 2006
From: hughw at hughw.net (Hugh Winkler)
Date: Wed, 20 Dec 2006 15:24:58 -0600
Subject: [mod-lisp-devel] Chunked encoding
Message-ID: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com>

Hi all,

I see this topic was raised  a couple of months ago :
http://common-lisp.net/pipermail/mod-lisp-devel/2006-October/thread.html#99.

I've made  changes to mod_lisp2.c to support chunked encoding. Sorry,
I only have a 2.0.55 to run against atm, so I have not backported to
1.3.

Notes:

Apache 2 has eliminated REQUEST_CHUNKED_DECHUNK, meaning there's no
way to receive chunked data ? Apache will always dechunkify it.

I considered signaling the end of the stream by sending MSG_OOB. Well,
apr does not support that. So the only option left was to re-chunk the
data. I opted to re-chunk according to RFC 2616 so that on the Lisp
side we can use Dr. Weitz'  Chunga[1] to de-chunk.

The new behavior is compatible with existing code on the Lisp side
e.g. Kevin Rosenberg's modlisp. The mod_lisp2 code only chunks if the
Content-length header is omitted; formerly, Lisp side code could not
work properly if Content-length was omitted anyway.

There is one maybe extraneous change in this patch: I eliminated some
gcc compiler warnings issuing from local_lisp_cfg(). Here is my gcc
version string: (GCC) 4.1.2 20060928 (prerelease) (Ubuntu
4.1.1-13ubuntu5). This change is not essential but it's nice to build
with no warnings.

Thanks Mark for mod_lisp...  sorry I'm no help on the 1.3 backport.

Hugh

[1] http://weitz.de/chunga/


--- ../libapache2-mod-lisp2-1.3.1-orig/mod_lisp2.c	2006-12-20
13:23:46.000000000 -0600
+++ mod_lisp2.c	2006-12-20 14:01:47.000000000 -0600
@@ -58,6 +58,11 @@
 /*
   Change log:

+  If request contains chunked data, send chunks (length word plus data)
+  to backend lisp. Also eliminated a gcc warning.
+  -- Hugh Winkler
+     2006-12-05
+
   Fixed the APR 1.2.2 detection.
   -- sent by several people
      2006-1203
@@ -231,7 +236,7 @@
 static header_map_t map_header_to_lisp_header;
 static int write_client_data
   (request_rec * r, const char * data, unsigned int n_bytes);
-

+
 /* Configuration data */

 static lisp_cfg_t *
@@ -284,19 +289,19 @@
 static lisp_cfg_t *
 local_lisp_cfg (lisp_cfg_t *cfg)
 {
-  lisp_cfg_t *local_cfg = NULL;
-
-  apr_threadkey_private_get((void**)&local_cfg, cfg_key);
+  void *local_cfg = NULL;
+
+  apr_threadkey_private_get(&local_cfg, cfg_key);
   if (local_cfg == NULL)
     {
       local_cfg = copy_lisp_cfg (socket_pool, cfg);
-      apr_threadkey_private_set((void*)local_cfg, cfg_key);
+      apr_threadkey_private_set(local_cfg, cfg_key);
       return local_cfg;
     }

   check_cfg_for_reuse(local_cfg, cfg);

-  return local_cfg;
+  return (lisp_cfg_t*) local_cfg;
 }
 #else
 lisp_cfg_t *local_cfg;
@@ -433,6 +438,26 @@
 }

 static apr_status_t
+write_lisp_data_chunk (apr_socket_t * socket,
+                 const char * data, unsigned int n_bytes)
+{
+  char crlf[2] =  {0xd, 0xa};
+  char length[16];
+  snprintf(length, 16, "%x", n_bytes);
+
+  apr_status_t status = write_lisp_data (socket, length, strlen(length));
+  if ( status == APR_SUCCESS)
+    {
+      status = write_lisp_data (socket, crlf, 2);
+      if ( status == APR_SUCCESS && n_bytes)
+	status = write_lisp_data (socket, data, n_bytes);
+      if ( status == APR_SUCCESS)
+	status = write_lisp_data (socket, crlf, 2);
+    }
+  return status;
+}
+
+static apr_status_t
 write_lisp_line (apr_socket_t * socket, const char * data)
 {
   RELAY_ERROR (write_lisp_data (socket, data, (strlen (data))));
@@ -555,6 +580,7 @@
   int content_length = (-1);
   int keep_socket_p = 0;
   apr_socket_t * socket;
+  const char * request_content_length = 0;

   cfg = local_lisp_cfg(cfg);

@@ -600,6 +626,8 @@
       ((copy_headers ((r->headers_in), map_header_to_lisp_header, socket)),
               "writing to Lisp");

+  request_content_length = apr_table_get(r->headers_in, "Content-Length");
+
   /* Send the end-of-headers marker.  */
   ML_LOG_DEBUG (r, "write end-of-headers");
   CVT_ERROR ((write_lisp_line (socket, "end")), "writing to Lisp");
@@ -619,21 +647,31 @@
 	      close_lisp_socket (cfg);
 	      return (HTTP_INTERNAL_SERVER_ERROR);
 	    }
-	  if (n_read == 0)
-	    break;

+	  /* for chunked case, when nread == 0, we will write
+	   * a terminating 0.*/
+	
 	  {
-	    apr_status_t status = (write_lisp_data (socket, buffer, n_read));
+	    apr_status_t status = APR_SUCCESS;
+	
+	    /* if there's no Content-Type header, the data must be chunked */
+	    if (request_content_length == NULL)
+	      status = write_lisp_data_chunk (socket, buffer, n_read);
+	    else if (n_read != 0)
+	      status = write_lisp_data (socket, buffer, n_read);
+	
 	    if (APR_SUCCESS != status)
 	      {
-		while ((ap_get_client_block (r, buffer, (sizeof (buffer))))
-		       		       > 0)
+		while ((ap_get_client_block (r, buffer, sizeof(buffer)))
+		       > 0)
 		  ;
 		ML_LOG_ERROR (status, r, "writing to Lisp");
 		close_lisp_socket (cfg);
 		return (HTTP_INTERNAL_SERVER_ERROR);
 	      }
 	  }
+	  if( n_read == 0)
+	    break;
 	}
     }


From edi at agharta.de  Wed Dec 20 23:09:28 2006
From: edi at agharta.de (Edi Weitz)
Date: Thu, 21 Dec 2006 00:09:28 +0100
Subject: [mod-lisp-devel] Chunked encoding
In-Reply-To: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com>
	(Hugh Winkler's message of "Wed, 20 Dec 2006 15:24:58 -0600")
References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com>
Message-ID: <uy7p2kvvb.fsf@agharta.de>

On Wed, 20 Dec 2006 15:24:58 -0600, "Hugh Winkler" <hughw at hughw.net> wrote:

> I've made  changes to mod_lisp2.c to support chunked encoding.

Nice, thanks.

I'll integrate this with Hunchentoot as soon as I find the time.


From marc.battyani at fractalconcept.com  Thu Dec 21 08:21:42 2006
From: marc.battyani at fractalconcept.com (Marc Battyani)
Date: Thu, 21 Dec 2006 09:21:42 +0100
Subject: [mod-lisp-devel] Chunked encoding
References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com>
Message-ID: <087d01c724d9$0eeb3ff0$1402a8c0@marcx2>

"Hugh Winkler" <hughw at hughw.net> wrote:

>I see this topic was raised  a couple of months ago :
>http://common-lisp.net/pipermail/mod-lisp-devel/2006-October/thread.html#99.
>
>I've made  changes to mod_lisp2.c to support chunked encoding. Sorry,
>I only have a 2.0.55 to run against atm, so I have not backported to
>1.3.
>
>Notes:
>
>Apache 2 has eliminated REQUEST_CHUNKED_DECHUNK, meaning there's no
>way to receive chunked data ? Apache will always dechunkify it.
>
>I considered signaling the end of the stream by sending MSG_OOB. Well,
>apr does not support that. So the only option left was to re-chunk the
>data. I opted to re-chunk according to RFC 2616 so that on the Lisp
>side we can use Dr. Weitz'  Chunga[1] to de-chunk.
>
>The new behavior is compatible with existing code on the Lisp side
>e.g. Kevin Rosenberg's modlisp. The mod_lisp2 code only chunks if the
>Content-length header is omitted; formerly, Lisp side code could not
>work properly if Content-length was omitted anyway.
>
>There is one maybe extraneous change in this patch: I eliminated some
>gcc compiler warnings issuing from local_lisp_cfg(). Here is my gcc
>version string: (GCC) 4.1.2 20060928 (prerelease) (Ubuntu
>4.1.1-13ubuntu5). This change is not essential but it's nice to build
>with no warnings.

Thanks a lot for this new feature. :)
I tried to apply your patch but subversion complains about its format. Can 
you send me the complete file or a svn compatible patch?

>Thanks Mark for mod_lisp...  sorry I'm no help on the 1.3 backport.

Thanks for contributing. ;-) No problem for the 1.3 backport, I think that 
Apache 2 is available everywhere now.

Marc



From hughw at hughw.net  Thu Dec 21 14:28:34 2006
From: hughw at hughw.net (Hugh Winkler)
Date: Thu, 21 Dec 2006 08:28:34 -0600
Subject: [mod-lisp-devel] Chunked encoding
In-Reply-To: <087d01c724d9$0eeb3ff0$1402a8c0@marcx2>
References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com>
	<087d01c724d9$0eeb3ff0$1402a8c0@marcx2>
Message-ID: <927441b30612210628p2a59fd21wd10d8fff2c32add5@mail.gmail.com>

Hi Marc,


On 12/21/06, Marc Battyani <marc.battyani at fractalconcept.com> wrote:

> Thanks a lot for this new feature. :)
> I tried to apply your patch but subversion complains about its format. Can
> you send me the complete file or a svn compatible patch?
>

I've attached the complete file. (I made the previous patch using diff
-Naur, not svn).

Regards,
Hugh
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mod_lisp2.c
Type: text/x-csrc
Size: 27718 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/mod-lisp-devel/attachments/20061221/82042562/attachment.c>

From marc.battyani at fractalconcept.com  Thu Dec 21 18:04:55 2006
From: marc.battyani at fractalconcept.com (Marc Battyani)
Date: Thu, 21 Dec 2006 19:04:55 +0100
Subject: [mod-lisp-devel] Chunked encoding
References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com><087d01c724d9$0eeb3ff0$1402a8c0@marcx2>
	<927441b30612210628p2a59fd21wd10d8fff2c32add5@mail.gmail.com>
Message-ID: <0a5401c7252a$88026160$1402a8c0@marcx2>

"Hugh Winkler" <hughw at hughw.net> wrote:
>
> On 12/21/06, Marc Battyani <marc.battyani at fractalconcept.com> wrote:
>
>> Thanks a lot for this new feature. :)
>> I tried to apply your patch but subversion complains about its format. 
>> Can
>> you send me the complete file or a svn compatible patch?
>>
>
> I've attached the complete file. (I made the previous patch using diff
> -Naur, not svn).

Thanks. I've put it on the repository.

Marc 



From hughw at hughw.net  Fri Dec 22 16:22:54 2006
From: hughw at hughw.net (Hugh Winkler)
Date: Fri, 22 Dec 2006 10:22:54 -0600
Subject: [mod-lisp-devel] Re: Chunked encoding
In-Reply-To: <927441b30612220819m292695b0l2a6580c038e299f2@mail.gmail.com>
References: <927441b30612201324l1c7aa545v2d820a8bc323dbf8@mail.gmail.com>
	<927441b30612220819m292695b0l2a6580c038e299f2@mail.gmail.com>
Message-ID: <927441b30612220822t17d823a3h8f8571047abac667@mail.gmail.com>

the attachment is attached this time... :)

On 12/22/06, Hugh Winkler <hwinkler at gmail.com> wrote:
> On 12/20/06, Hugh Winkler <hughw at hughw.net> wrote:
>
> > I've made  changes to mod_lisp2.c to support chunked encoding.
>
> As a guide, I've attached are some changes I have made to modlisp to
> support consuming chunked data from mod_lisp2:
>
> ;; These are changes to Kevin Rosenberg's modlisp
> ;; to consume chunked data.  There's a
> ;; change to package kmrcl to have the *modlisp-socket*
> ;; created as a flexi-stream.
> ;; ***n.b. this change has only been made for sbcl***
> ;; Then,if mod_lisp2.c has not sent a Content-length header,
> ;; get-content-as-string() creates a chunga chunked
> ;; input stream over the original stream; otherwise
> ;; it just consumes the stream as before.
>
> ;; depends on chunga http://weitz.de/chunga
> ;; and on flexi-streams http://weits.de/flexi-streams
>
> Thanks, Edi, for making this so easy with flex and chunga!
>
> Hugh
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: modlisp2.lisp
Type: application/octet-stream
Size: 3065 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/mod-lisp-devel/attachments/20061222/30d04e88/attachment.obj>