[hunchentoot-devel] Hunchentoot and threads

[Hans Hübner]

Hi Cyrus,

when you see worker threads accumulate, do you also see that there is
a large number of connections to Hunchentoot that are not being
closed?  Or are there just threads, but no connections?  Are you
running Hunchentoot behind a proxy/frontend, or standalone?  How many
dead workers did you see?

NB:  I do certainly think that it is possible to write threaded web
servers that work, I just believe that it is hard, in many respects.
Threads, in my personal opinion, are not the best hammer to solve the
I/O multiplexing problem that needs to be solved in a web server.
Certainly, the unbounded worker thread creation strategy that
Hunchentoot uses is not suitable for servers that see load peaks,
which is why I recommend not using threaded Hunchentoot for such
sites.

That said:  I do have stability issues with my non-threaded
Hunchentoot installation on FreeBSD, too.  I use a multi threaded
SBCL, but run Hunchentoot in a single thread behind squid.  In some
situations, no new connections are being accepted for no apparent
reasons, but I failed to properly analyze the problem last time it
happened as the customer was already nervous.  I have seen this happen
two times in the last four months, on a moderately busy site. Thus,
the problem may actually not be related to Hunchentoot's threads usage
(I'm running non-threaded, you run threaded), but could as well be
located in SBCL's thread implementation of FreeBSD, in usocket or
somewhere else.

Any further data points would help getting down to the bottom of this.

As for future work on Hunchentoot:  We do have the new connection
manager class in place which is meant to support the implementation of
thread pools.  Thread pools would help putting limits on the number of
threads created, helping with getting through load peaks.  I do not
personally need such a connection manager, but rather want to spend
some time on making Hunchentoot be able to use single threaded I/O
multiplexing using select/kpoll/whatever.

-Hans

On Sun, Nov 16, 2008 at 18:03, Cyrus Harmon <ch-tbnl at bobobeach.com> wrote:
>
> I seem to recall that, a long time ago, TBNL/hunchentoot required
> threads. At some point Edi rigged up a singled threaded version but
> claimed that that was just for development/testing work and shouldn't
> be used in the real world. Now I understand that Hans has an aversion
> to threads as the multiplexing abstraction for webservers and that
> single threaded is "the one true way." I bring this up as background
> to me real problem, which is that, at least on FreeBSD, I seem to have
> a number of zombie threads that stick around after requests are made.
> Before I dig into the problem, I'd be curious to hear what folks'
> recommendations for running a low-traffic, but nevertheless hopefully
> reliable, site are. I had reasonably good luck with SBCL+threads and
> hunchentoot before the big rewrite, but since then, it has been
> reliable refusing to accept new connections after a week or two of use.
>
> Everything starts out fine and I can field some traffic and all is good:
>
> CL-USER> (sb-thread:list-all-threads)
> (#<SB-THREAD:THREAD "reader-thread" RUNNING {59B6B241}>
>  #<SB-THREAD:THREAD "repl-thread" RUNNING {59B6B0F1}>
>  #<SB-THREAD:THREAD "control-thread" RUNNING {59B60359}>
>  #<SB-THREAD:THREAD "auto-flush-thread" RUNNING {59B601D9}>
>  #<SB-THREAD:THREAD "initial thread" RUNNING {598D79B1}>)
>
> However, after some period, I see an accumulation of worker threads
> that won't die and eventually I'm unable to start a new thread to
> field the request. Who knows where the problem is... Could be in
> hunchentoot proper, in usocket, in SBCL's FreeBSD threads, etc... At
> the moment I don't have any debugging info on the problem, as all
> seems to be working ok since I restarted the web server a few minutes
> ago. The next time the problem crops up, I'll reply to this with some
> debugging information. This seems to be the one major "regression"
> from the old hunchentoot that has me considering going back to the
> stable release (although I've already ported my code forward for the
> new API, so I'd hate to have to do that). Better yet would be to
> either track down and fix the problem or to be convinced that this is
> just broken and is never going to work and figure out how to make my
> stuff work in a single threaded lisp.
>
> One more question regarding threads, is it possible to use a threaded
> lisp to act like the single threaded server does? That is to use
> threads, but to only have one thread doing the hard work, using serve-
> event for the multiplexing? My initialization code relies on setting
> some state after starting to listen for requests, so I can't just flip
> the switch to single-threaded mode and have everything work out of the
> box.
>
> Thanks for all of the hunchentoot rewrite efforts! On the whole it
> seems like a good thing, but fixing this issue would make me a much
> happier hunchentoot user.
>
> Cyrus
>
>
> _______________________________________________
> tbnl-devel site list
> tbnl-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/tbnl-devel
>