From edi at agharta.de Sun Jan 1 18:13:29 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 01 Jan 2006 19:13:29 +0100 Subject: [tbnl-devel] New release 0.9.3 Message-ID: Hi and Happy New Year! ChangeLog: Version 0.9.3 2006-01-01 Fixed bug in READ-HTTP-REQUEST Download: Cheers, Edi. From edi at agharta.de Sun Jan 1 18:58:01 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 01 Jan 2006 19:58:01 +0100 Subject: [tbnl-devel] New Hunchentoot release 0.1.1 Message-ID: ChangeLog: Version 0.1.1 2005-12-31 Fixed package stuff and HYPERDOC support Download: Cheers, Edi. From edi at agharta.de Tue Jan 3 18:53:24 2006 From: edi at agharta.de (Edi Weitz) Date: Tue, 03 Jan 2006 19:53:24 +0100 Subject: [tbnl-devel] New: TBNL 0.9.4, Hunchentoot 0.1.2 Message-ID: ChangeLog TBNL: Version 0.9.4 2006-01-03 Handle "Expect: 100-continue" for non-Apache front-ends Re-introduced IGNORE-ERRORS in GET-REQUEST-DATA ChangeLog Hunchentoot: Version 0.1.2 2006-01-03 Mention TBNL version number in server name header Download: Have fun, Edi. From edi at agharta.de Sat Jan 7 23:08:21 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 08 Jan 2006 00:08:21 +0100 Subject: [tbnl-devel] Error reading from Lisp (?) Message-ID: Hi! I'm currently having some trouble with a long-running TBNL/mod_lisp application on a public website which gets about 500,000 requests per month. It's an application delivered with LispWorks professional 4.4.6 and it runs on Linux behind Apache 2.0.54. >From time to time I see errors in the server log file and sometimes (very rarely) users also have reported errors. It turns out they're all of the "error reading from Lisp" kind. I hacked mod_lisp2.c a bit to make this more specific and I now get one of "error reading from Lisp (fill status)" or "error reading header name." This means that the error happens either when the reply entity is copied from Lisp to the client or when mod_lisp tries to read a header name from Lisp. I've attached a couple of these error message to the end of this email. What's suspicious for me is that they almost always come in chunks - half a dozen of them or more in a couple of seconds, then several hours without errors. I'm calling MARK-AND-SWEEP from time to time but at intervals that don't match the time of the error messages. (My first thought was that a full GC blocks LW long enough for mod_lisp to time out but that doesn't seem to be the case.) Other than that I only use locks in admin parts of the website which are clearly not used in the middle of the night when some of these errors happened. I'm lost. Does anyone have an idea where I should look to find the cause for these problems? Thanks, Edi. [Fri Jan 06 00:50:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:09 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:10 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:13 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:14 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:15 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 00:50:17 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:34 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:36 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:37 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:38 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:40 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:41 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:43 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:44 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:45 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 03:41:46 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:00 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:01 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:08 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:09 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:10 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 16:52:12 2006] [error] (70007)The timeout specified has expired: error reading header name [Fri Jan 06 18:07:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Fri Jan 06 21:53:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 04:31:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 05:03:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 06:04:10 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 07:05:02 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 07:56:04 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 07:56:05 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 07:56:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 10:03:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 10:28:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:22:53 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:23:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:37:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:48:43 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 11:52:56 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:03:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:24:57 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 13:32:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:04:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:10:00 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:11:04 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:12:08 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:13:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:14:19 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:15:26 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 14:58:12 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:25:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:29:14 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:30:45 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:31:49 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:46:11 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:49:50 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 15:53:46 2006] [error] (70007)The timeout specified has expired: error reading from Lisp (fill status) [Sat Jan 07 21:05:03 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:04 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:06 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:07 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:11 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:12 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:14 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:15 2006] [error] (70007)The timeout specified has expired: error reading header name [Sat Jan 07 21:05:19 2006] [error] (70007)The timeout specified has expired: error reading header name From marc.battyani at fractalconcept.com Sat Jan 7 23:33:55 2006 From: marc.battyani at fractalconcept.com (Marc Battyani) Date: Sun, 8 Jan 2006 00:33:55 +0100 Subject: [tbnl-devel] Re: [mod-lisp-devel] Error reading from Lisp (?) References: Message-ID: <058401c613e2$d37e1530$0a02a8c0@marcxp> "Edi Weitz" wrote: > > I'm currently having some trouble with a long-running TBNL/mod_lisp > application on a public website which gets about 500,000 requests per > month. It's an application delivered with LispWorks professional > 4.4.6 and it runs on Linux behind Apache 2.0.54. > > >From time to time I see errors in the server log file and sometimes > (very rarely) users also have reported errors. It turns out they're > all of the "error reading from Lisp" kind. I hacked mod_lisp2.c a bit > to make this more specific and I now get one of "error reading from > Lisp (fill status)" or "error reading header name." This means that > the error happens either when the reply entity is copied from Lisp to > the client or when mod_lisp tries to read a header name from Lisp. > > I've attached a couple of these error message to the end of this > email. What's suspicious for me is that they almost always come in > chunks - half a dozen of them or more in a couple of seconds, then > several hours without errors. > > I'm calling MARK-AND-SWEEP from time to time but at intervals that > don't match the time of the error messages. (My first thought was > that a full GC blocks LW long enough for mod_lisp to time out but that > doesn't seem to be the case.) Other than that I only use locks in > admin parts of the website which are clearly not used in the middle of > the night when some of these errors happened. > > I'm lost. Does anyone have an idea where I should look to find the > cause for these problems? Have you tried to log the time taken to process the mod_lisp command in Lisp (internally in the Lisp process) to see if there is a correlation with the Apache errors ? Marc From edi at agharta.de Sun Jan 8 00:52:19 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 08 Jan 2006 01:52:19 +0100 Subject: [tbnl-devel] Re: [mod-lisp-devel] Error reading from Lisp (?) In-Reply-To: <058401c613e2$d37e1530$0a02a8c0@marcxp> (Marc Battyani's message of "Sun, 8 Jan 2006 00:33:55 +0100") References: <058401c613e2$d37e1530$0a02a8c0@marcxp> Message-ID: On Sun, 8 Jan 2006 00:33:55 +0100, "Marc Battyani" wrote: > Have you tried to log the time taken to process the mod_lisp command > in Lisp (internally in the Lisp process) to see if there is a > correlation with the Apache errors ? You mean the time it takes to send the data from Lisp to Apache? Nah, haven't done that yet but it's an idea, thanks. I'll try it. From edi at agharta.de Sat Jan 21 23:37:12 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 22 Jan 2006 00:37:12 +0100 Subject: [tbnl-devel] New release 0.9.5 Message-ID: ChangeLog: Version 0.9.5 2006-01-22 Made creation of REQUEST object safer (thanks to Robert J. Macomber) Replaced some erroneous DECLAIMs with DECLAREs (thanks to SBCL's style warnings) Slight documentation enhancements Download: Cheers, Edi. From edi at agharta.de Sun Jan 22 02:00:43 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 22 Jan 2006 03:00:43 +0100 Subject: [tbnl-devel] New release 0.9.6 (Was: New release 0.9.5) In-Reply-To: (Edi Weitz's message of "Sun, 22 Jan 2006 00:37:12 +0100") References: Message-ID: Please use 0.9.6 instead - I forgot something in 0.9.5. Sorry for the noise. From tbnl at rojoma.com Sun Jan 22 03:41:11 2006 From: tbnl at rojoma.com (Robert J. Macomber) Date: Sat, 21 Jan 2006 20:41:11 -0700 Subject: [tbnl-devel] Re: TBNL file descriptor leak In-Reply-To: References: <20060114184332.GB29042@oja.no> Message-ID: <20060122034111.GA23565@oja.no> On Sun, Jan 22, 2006 at 12:41:15AM +0100, Edi Weitz wrote: > > [not subscribed to tbnl-devel] > > You should be... :) All right; now I am. > I don't really like that idea. Instead, I now decided to send a 400 > (Bad Request) reply to the client. Check out TBNL 0.9.5 and see if it > works for you. It fixes the problem I was seeing, but I've uncovered a new one. However, this one's not really TBNL's fault. If you send a non-UTF8 urlencoded sequence _before_ the `?' in the request, TBNL dies the same as before, because headers are read from the bivalent stream with read-line (that is, in character mode) but Apache(*) de-url-encodes names before passing them on for the `script-filename' header. It can be fixed by either making kmrcl open the bivalent stream explicitly with :external-format :latin-1 or by running sbcl in a latin-1 locale. In any case, I haven't hit any more socket leaks now that that one's been tracked down, so it looks like all's right in the world, at least as far as TBNL's concerned. (*) now I'm using apache2 and mod_lisp2 version 1.3 -- Robert Macomber tbnl at rojoma.com From edi at agharta.de Sun Jan 22 12:05:13 2006 From: edi at agharta.de (Edi Weitz) Date: Sun, 22 Jan 2006 13:05:13 +0100 Subject: [tbnl-devel] Re: TBNL file descriptor leak In-Reply-To: <20060122034111.GA23565@oja.no> (Robert J. Macomber's message of "Sat, 21 Jan 2006 20:41:11 -0700") References: <20060114184332.GB29042@oja.no> <20060122034111.GA23565@oja.no> Message-ID: On Sat, 21 Jan 2006 20:41:11 -0700, "Robert J. Macomber" wrote: > It fixes the problem I was seeing, but I've uncovered a new one. > > However, this one's not really TBNL's fault. If you send a non-UTF8 > urlencoded sequence _before_ the `?' in the request, TBNL dies the > same as before, because headers are read from the bivalent stream > with read-line (that is, in character mode) but Apache(*) > de-url-encodes names before passing them on for the > `script-filename' header. Do you have an example for this? Seems I'm a bit dumb at the moment. Thanks, Edi. From edi at agharta.de Mon Jan 23 10:11:25 2006 From: edi at agharta.de (Edi Weitz) Date: Mon, 23 Jan 2006 11:11:25 +0100 Subject: [tbnl-devel] Re: TBNL file descriptor leak In-Reply-To: <20060122172004.GA29409@oja.no> (Robert J. Macomber's message of "Sun, 22 Jan 2006 10:20:04 -0700") References: <20060114184332.GB29042@oja.no> <20060122034111.GA23565@oja.no> <20060122172004.GA29409@oja.no> Message-ID: On Sun, 22 Jan 2006 10:20:04 -0700, "Robert J. Macomber" wrote: > In my apache2 config, I've got this [simplified]: > > LispServer 127.0.0.1 3000 lisp > > ServerName lisp.rojoma.com > DocumentRoot /home/robertm/mod_lisp > > SetHandler lisp-handler > > > > If you visit the url http://lisp.rojoma.com/%ef.l apache sends bytes > for the script-filename parameter which can be interpreted as the > latin-1 string "/home/robertm/mod_lisp/?.l". If they're interpreted > as UTF-8, like sbcl's READ-LINE tries to do when running in an UTF-8 > locale (since kmrcl doesn't specify an explicit :external-format in > ACCEPT-TCP-CONNECTION) an error is thrown in GET-REQUEST-DATA. > > In fact, a user can make apache send arbitrary bytes to lisp while > it's reading the request headers in other ways too (e.g., sending a > "Mumble: {random octets}" header) but script-filename is the only > one where Apache will do so more or less on its own. > > I changed kmrcl here rather than making tbnl send a "bad request" > reply because you never know what random junk proxies and such will > throw into users' HTTP request headers, and I didn't want to deny > service just because some proxy is adding the (accented) name of > their ISP to user-agent strings or something. I haven't actually > seen that happen yet, but the net is vast. OK, thanks for the info. I've been thinking about extracting the relevant bits from KMRCL and integrating them into TBNL directly anyway. We only use a small fraction of what's in there and it might make sense to make some of this user-configurable, not only because of the external format but also because of the interface(s) TBNL will listen on, for example. I'll do that one day... :) Cheers, Edi. From keith.irwin at gmail.com Mon Jan 30 20:17:31 2006 From: keith.irwin at gmail.com (Keith Irwin) Date: Mon, 30 Jan 2006 12:17:31 -0800 Subject: [tbnl-devel] tbnl as unix daemon Message-ID: <8aff81590601301217q580f79e8r322de982384a921f@mail.gmail.com> Folks-- I want to be able to start/stop a tbnl based app via a script. I've done this before with my own server apps, so it all seems relatively straight forward. In other words, I have a Makefile which builds a core, copies shared-libs, packages it all up in a tarball, etc, etc, such that I can deploy the app on any machine, regardless of whether or not SBCL is installed. The problem is that (start-tbnl) starts up a thread, then returns, and with that return, sbcl quits, and I got no daemon. Is there a way to start tbnl without it returning? I suppose I can do something like: (defun daemon-start () (tbnl:start-tbnl) (wait-forever)) and then let my init.d script send a kill signal as I've already planned. Is that the recommended method (assuming I can find a wait-forever function, or just (read) or something)? In case it needs be said, I really value running tbnl inside a repl with slime and all that, but this is for the case where that's not desired or necessary. Thanks for any lore about what you all do in a similar case! ;) Keith From tolstoy at zentrope.com Mon Jan 30 21:18:20 2006 From: tolstoy at zentrope.com (Tolstoy) Date: Mon, 30 Jan 2006 13:18:20 -0800 Subject: [tbnl-devel] tbnl as unix daemon In-Reply-To: <8aff81590601301217q580f79e8r322de982384a921f@mail.gmail.com> References: <8aff81590601301217q580f79e8r322de982384a921f@mail.gmail.com> Message-ID: <1138655900.3834.28.camel@keith1.cv.hp.com> Well, I solved this one. Not saying it's the best solution, but it goes something like: (defvar *waiter* (make-waitqueue)) (defvar *locker* (make-mutex)) (defun unix-signals (sig code scp) (declare (ignorable code scp)) (handler-case (condition-notify *waiter*) (condition (c) (format *standard-output* "~a" c))) (sb-ext::quit)) (defun daemon-mode-start () (load-sample-data) (sb-sys:enable-interrupt sb-unix:sighup #'unix-signals) (sb-sys:enable-interrupt sb-unix:sigterm #'unix-signals) (sb-sys:enable-interrupt sb-unix:sigkill #'unix-signals) (tbnl:start-tbnl) (with-mutex (*locker*) (log-message :debug "wait mode") (condition-wait *waiter* *locker*) (log-message :debug "terminated.") (tbnl:stop-tbnl) (sb-ext::quit))) *Shrug*. Good enough for now! ;) Keith On Mon, 2006-01-30 at 12:17 -0800, Keith Irwin wrote: > Folks-- > > I want to be able to start/stop a tbnl based app via a script. I've > done this before with my own server apps, so it all seems relatively > straight forward. In other words, I have a Makefile which builds a > core, copies shared-libs, packages it all up in a tarball, etc, etc, > such that I can deploy the app on any machine, regardless of whether > or not SBCL is installed. > > The problem is that (start-tbnl) starts up a thread, then returns, and > with that return, sbcl quits, and I got no daemon. > > Is there a way to start tbnl without it returning? > > I suppose I can do something like: > > (defun daemon-start () > (tbnl:start-tbnl) > (wait-forever)) > > and then let my init.d script send a kill signal as I've already > planned. Is that the recommended method (assuming I can find a > wait-forever function, or just (read) or something)? > > In case it needs be said, I really value running tbnl inside a repl > with slime and all that, but this is for the case where that's not > desired or necessary. > > Thanks for any lore about what you all do in a similar case! ;) > > Keith > _______________________________________________ > tbnl-devel site list > tbnl-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/tbnl-devel From ocorrain at yahoo.com Tue Jan 31 23:31:30 2006 From: ocorrain at yahoo.com (=?iso-8859-1?Q?Tiarn=E1n_=D3_Corr=E1in?=) Date: Tue, 31 Jan 2006 23:31:30 +0000 Subject: [tbnl-devel] tbnl as unix daemon In-Reply-To: <8aff81590601301217q580f79e8r322de982384a921f@mail.gmail.com> (Keith Irwin's message of "Mon, 30 Jan 2006 12:17:31 -0800") References: <8aff81590601301217q580f79e8r322de982384a921f@mail.gmail.com> Message-ID: Keith Irwin writes: > Is there a way to start tbnl without it returning? Have you looked at detachtty/attachtty? -- Tiarn?n