[hunchentoot-devel] Hunchentoot Errors (e.g. Couldn't write to #<SB-SYS:FD-STREAM for "socket ..." ...>)

[Cyrus Harmon]

Hello tbnl-folk,

So, I've been seeing transient problems with hunchentoot for as long as I can remember, and I haven't had the chance to try to track down the problems until now. It seems there are a couple of classes of problems. In order of severity, the are:

1. things that crash hunchentoot/SBCL (or, in some cases, the host OS)

2. things that throw "unexpected" errors

3. things that throw "expected" errors (e.g. 404)

As for class 1, these are tough to track down as the server is usually crashed by the time I get to it, but some causes I consider plausible are:

* running out of some limited resource such as RAM, threads, or network sockets
* possible SBCL bug in, say, signal handling or dealing with lots of threads

I don't have any data to present on this class, at the moment, so let's move on to the next class.

I often see errors such as this in the log:

[2011-04-05 00:04:52 [ERROR]] Couldn't write to #<SB-SYS:FD-STREAM
                    for "socket 74.115.254.12:80, peer: 217.72.218.248:2741"
                    {B6FB859}>:
  Broken pipe
0: (SB-DEBUG::MAP-BACKTRACE
    #<CLOSURE (LAMBDA #) {B71EC4D}>
    :START
    0
    :COUNT
    536870911)
1: (BACKTRACE 536870911 #<SB-IMPL::STRING-OUTPUT-STREAM {B71EBE9}>)
2: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM
    #<SB-IMPL::STRING-OUTPUT-STREAM {B71EBE9}>)
3: (HUNCHENTOOT::GET-BACKTRACE)
4: ((FLET #:LAMBDA271)
    #<SB-INT:SIMPLE-STREAM-ERROR "~@<~?: ~2I~_~A~:>" {B71EA51}>)
5: (SIGNAL #<SB-INT:SIMPLE-STREAM-ERROR "~@<~?: ~2I~_~A~:>" {B71EA51}>)
6: (ERROR
    SB-INT:SIMPLE-STREAM-ERROR
    :STREAM
    #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741"
      {B6FB859}>
    :FORMAT-CONTROL
    "~@<~?: ~2I~_~A~:>"
    :FORMAT-ARGUMENTS
    ("Couldn't write to ~S"
     (#<SB-SYS:FD-STREAM
        for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>)
     "Broken pipe"))
7: (SB-IMPL::SIMPLE-STREAM-PERROR
    "Couldn't write to ~S"
    #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741"
      {B6FB859}>
    32)
8: (SB-IMPL::WRITE-OUTPUT-FROM-QUEUE
    #<SB-SYS:FD-STREAM for "socket 74.115.254.12:80, peer: 217.72.218.248:2741"
      {B6FB859}>)
9: (SB-IMPL::SUB-SUB-SERVE-EVENT NIL NIL)
10: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL)
11: (SB-SYS:SERVE-ALL-EVENTS NIL)
12: (SB-IMPL::FINISH-FD-STREAM-OUTPUT
     #<SB-SYS:FD-STREAM
       for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>)
13: (FINISH-OUTPUT
     #<SB-SYS:FD-STREAM
       for "socket 74.115.254.12:80, peer: 217.72.218.248:2741" {B6FB859}>)
14: (HUNCHENTOOT:HANDLE-STATIC-FILE
     #P"/home/sly/projects/www-cyrusharmon-org/static/images/fish-clusters.png"
     NIL)
15: ((SB-PCL::FAST-METHOD HUNCHENTOOT:HANDLE-REQUEST
      (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST))
     #<unavailable argument>
     #<unavailable argument>
     #<HUNCHENTOOT:ACCEPTOR (host *, port 80)>
     #<HUNCHENTOOT:REQUEST {B701D09}>)
16: ((SB-PCL::FAST-METHOD HUNCHENTOOT:PROCESS-REQUEST (T))
     #<unavailable argument>
     #<unavailable argument>
     #<HUNCHENTOOT:REQUEST {B701D09}>)
17: ((SB-PCL::FAST-METHOD HUNCHENTOOT:PROCESS-CONNECTION
      (HUNCHENTOOT:ACCEPTOR T))
     #<unavailable argument>
     #<unavailable argument>
     #<HUNCHENTOOT:ACCEPTOR (host *, port 80)>
     #<USOCKET:STREAM-USOCKET {B6FB8F1}>)
18: ((SB-PCL::FAST-METHOD HUNCHENTOOT:PROCESS-CONNECTION :AROUND
      (HUNCHENTOOT:ACCEPTOR T))
     #<unavailable argument>
     #S(SB-PCL::FAST-METHOD-CALL
        :FUNCTION #<FUNCTION #>
        :PV NIL
        :NEXT-METHOD-CALL NIL
        :ARG-INFO (2))
     #<HUNCHENTOOT:ACCEPTOR (host *, port 80)>
     #<USOCKET:STREAM-USOCKET {B6FB8F1}>)
19: ((LAMBDA ()))
20: ((FLET #:WITHOUT-INTERRUPTS-BODY-[BLOCK353]358))
21: ((FLET SB-THREAD::WITH-MUTEX-THUNK))
22: ((FLET #:WITHOUT-INTERRUPTS-BODY-[CALL-WITH-MUTEX]300))
23: (SB-THREAD::CALL-WITH-MUTEX
     #<CLOSURE (FLET SB-THREAD::WITH-MUTEX-THUNK) {B0E14235}>
     #S(SB-THREAD:MUTEX
        :NAME "thread result lock"
        :%OWNER #<SB-THREAD:THREAD
                  "Hunchentoot worker (client: 217.72.218.248:2741)" RUNNING
                  {B6FCE39}>
        :STATE 1)
     #<SB-THREAD:THREAD
       "Hunchentoot worker (client: 217.72.218.248:2741)" RUNNING {B6FCE39}>
     T)
24: (SB-THREAD::INITIAL-THREAD-FUNCTION)
25: ("foreign function: call_into_lisp")
26: ("foreign function: funcall0")
27: ("foreign function: new_thread_trampoline")
28: ("foreign function: #xB7FC5506")

What does this whole "broken pipe" mean? Is this more of an "expected error" or is something going wrong here?

On a slightly different note, I'd like to be able to run a relatively reliable hunchentoot server that can serve a modest traffic load without having to resort to caching proxy servers and the like. I recall some discussion of thread pools, epoll, etc... Has the state of the art for high-reliability hunchentoot instances changed in the last couple years? Any suggestions on improving it?

thanks!

Cyrus