[tbnl-devel] Error reading from Lisp (?)

[Edi Weitz]

Hi!

I'm currently having some trouble with a long-running TBNL/mod_lisp
application on a public website which gets about 500,000 requests per
month.  It's an application delivered with LispWorks professional
4.4.6 and it runs on Linux behind Apache 2.0.54.

>From time to time I see errors in the server log file and sometimes
(very rarely) users also have reported errors.  It turns out they're
all of the "error reading from Lisp" kind.  I hacked mod_lisp2.c a bit
to make this more specific and I now get one of "error reading from
Lisp (fill status)" or "error reading header name."  This means that
the error happens either when the reply entity is copied from Lisp to
the client or when mod_lisp tries to read a header name from Lisp.

I've attached a couple of these error message to the end of this
email.  What's suspicious for me is that they almost always come in
chunks - half a dozen of them or more in a couple of seconds, then
several hours without errors.

I'm calling MARK-AND-SWEEP from time to time but at intervals that
don't match the time of the error messages.  (My first thought was
that a full GC blocks LW long enough for mod_lisp to time out but that
doesn't seem to be the case.)  Other than that I only use locks in
admin parts of the website which are clearly not used in the middle of
the night when some of these errors happened.

I'm lost.  Does anyone have an idea where I should look to find the
cause for these problems?

Thanks,
Edi.


[Fri Jan 06 00:50:05 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:06 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:07 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:09 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:10 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:11 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:13 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:14 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:15 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 00:50:17 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:34 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:36 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:37 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:38 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:40 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:41 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:43 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:44 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:45 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 03:41:46 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:00 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:01 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:05 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:06 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:07 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:08 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:09 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:10 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:11 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 16:52:12 2006] [error] (70007)The timeout specified has expired: error reading header name
[Fri Jan 06 18:07:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Fri Jan 06 21:53:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 04:31:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 05:03:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 06:04:10 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 07:05:02 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 07:56:04 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 07:56:05 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 07:56:06 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 10:03:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 10:28:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 11:22:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 11:23:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 11:37:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 11:48:43 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 11:52:56 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 13:03:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 13:24:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 13:32:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:04:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:10:00 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:11:04 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:12:08 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:13:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:14:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:15:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 14:58:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:25:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:29:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:30:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:31:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:46:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:49:50 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 15:53:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status)
[Sat Jan 07 21:05:03 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:04 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:06 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:07 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:12 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:14 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:15 2006] [error] (70007)The timeout specified has expired: error reading header name
[Sat Jan 07 21:05:19 2006] [error] (70007)The timeout specified has expired: error reading header name