From edi at agharta.de Sat Jan 7 23:08:21 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 08 Jan 2006 00:08:21 +0100 Subject: [mod-lisp-devel] Error reading from Lisp (?) Message-ID: Hi! I'm currently having some trouble with a long-running TBNL/mod_lisp application on a public website which gets about 500,000 requests per month. It's an application delivered with LispWorks professional 4.4.6 and it runs on Linux behind Apache 2.0.54. >From time to time I see errors in the server log file and sometimes (very rarely) users also have reported errors. It turns out they're all of the "error reading from Lisp" kind. I hacked mod_lisp2.c a bit to make this more specific and I now get one of "error reading from Lisp (fill status)" or "error reading header name." This means that the error happens either when the reply entity is copied from Lisp to the client or when mod_lisp tries to read a header name from Lisp. I've attached a couple of these error message to the end of this email. What's suspicious for me is that they almost always come in chunks - half a dozen of them or more in a couple of seconds, then several hours without errors. I'm calling MARK-AND-SWEEP from time to time but at intervals that don't match the time of the error messages. (My first thought was that a full GC blocks LW long enough for mod_lisp to time out but that doesn't seem to be the case.) Other than that I only use locks in admin parts of the website which are clearly not used in the middle of the night when some of these errors happened. I'm lost. Does anyone have an idea where I should look to find the cause for these problems? Thanks, Edi. [Fri Jan 06 00:50:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:09 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:10 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:13 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:14 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:15 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:17 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:34 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:36 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:37 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:38 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:40 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:41 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:43 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:44 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:45 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:46 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:00 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:01 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:08 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:09 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:10 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:12 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 18:07:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Fri Jan 06 21:53:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 04:31:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 05:03:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 06:04:10 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 07:05:02 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 07:56:04 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 07:56:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 07:56:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 10:03:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 10:28:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:22:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:23:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:37:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:48:43 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:52:56 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:03:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:24:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:32:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:04:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:10:00 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:11:04 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:12:08 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:13:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:14:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:15:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:58:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:25:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:29:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:30:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:31:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:46:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:49:50 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:53:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 21:05:03 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:04 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:12 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:14 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:15 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:19 2006] [error] (70007)The timeout specified has expired: error reading header name From marc.battyani at fractalconcept.com Sat Jan 7 23:33:55 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Sun, 8 Jan 2006 00:33:55 +0100 Subject: [mod-lisp-devel] Error reading from Lisp (?) References: Message-ID: <058401c613e2$d37e1530$0a02a8c0@marcxp> "Edi Weitz" wrote: > > I'm currently having some trouble with a long-running TBNL/mod_lisp > application on a public website which gets about 500,000 requests per > month. It's an application delivered with LispWorks professional > 4.4.6 and it runs on Linux behind Apache 2.0.54. > > >From time to time I see errors in the server log file and sometimes > (very rarely) users also have reported errors. It turns out they're > all of the "error reading from Lisp" kind. I hacked mod_lisp2.c a bit > to make this more specific and I now get one of "error reading from > Lisp (fill status)" or "error reading header name." This means that > the error happens either when the reply entity is copied from Lisp to > the client or when mod_lisp tries to read a header name from Lisp. > > I've attached a couple of these error message to the end of this > email. What's suspicious for me is that they almost always come in > chunks - half a dozen of them or more in a couple of seconds, then > several hours without errors. > > I'm calling MARK-AND-SWEEP from time to time but at intervals that > don't match the time of the error messages. (My first thought was > that a full GC blocks LW long enough for mod_lisp to time out but that > doesn't seem to be the case.) Other than that I only use locks in > admin parts of the website which are clearly not used in the middle of > the night when some of these errors happened. > > I'm lost. Does anyone have an idea where I should look to find the > cause for these problems? Have you tried to log the time taken to process the mod_lisp command in Lisp (internally in the Lisp process) to see if there is a correlation with the Apache errors ? Marc From edi at agharta.de Sun Jan 8 00:52:19 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 08 Jan 2006 01:52:19 +0100 Subject: [mod-lisp-devel] Error reading from Lisp (?) In-Reply-To: <058401c613e2$d37e1530$0a02a8c0@marcxp> (Marc Battyani's message of "Sun, 8 Jan 2006 00:33:55 +0100") References: <058401c613e2$d37e1530$0a02a8c0@marcxp> Message-ID: On Sun, 8 Jan 2006 00:33:55 +0100, "Marc Battyani" wrote: > Have you tried to log the time taken to process the mod_lisp command > in Lisp (internally in the Lisp process) to see if there is a > correlation with the Apache errors ? You mean the time it takes to send the data from Lisp to Apache? Nah, haven't done that yet but it's an idea, thanks. I'll try it.