[hunchentoot-devel] CLOSE_WAIT and errcode 24 (too many files) when using https

[Lucas Hope]

Hi Hans,

Thanks for testing and replying. After a bit more detective work, I can't
isolate this to Hunchentoot either. A colleague accidentally filled the
disk around the same timeframe this occurred. The monitoring service may
have been holding connections over time, but we have disabled it now.

It seems likely that this was a one-off fluke occurrence, and it won't
bother us again. That being the case, I can't justify digging into it any
more. I reported the issue in case it was a simple bug and easy fix to make
Hunchentoot that little bit more robust (it is very robust already! :-) ).

Hunchentoot is an excellent library which I have used over some years.
Thank you for your ongoing maintenance. I hope I haven't wasted your time.

Regards,

-Luke

On Wed, Mar 27, 2013 at 5:15 PM, Hans Hübner <hans.huebner at gmail.com> wrote:

> Luke,
>
> unfortunately, I cannot reproduce the problem in on my machine - I do not
> see any open file descriptors left over after I connect to Hunchentoot
> server with SSL using a non-SSL client, and when I connect to the same
> server using a browser, I get an "Empty response" error.
>
> In order to help you with this, I need you to isolate the problem further.
>  Some ideas for debugging:
>
> - Can you create a lingering file descriptor when you connect to your
> server using telnet, then disconnect?
> - Are the CLOSE_WAIT sockets actually connected to your monitoring service?
> - Do you see the CLOSE_WAIT in lsof or in netstat?  I see it in netstat
> for sockets that have been closed by the application, but for which the FIN
> TCP packet has not been acknowledged by the client.  Note that CLOSE_WAIT
> sockets are closed in the application and do no longer consume file
> descriptors.  The final TCP handshake is performed by the kernel, and there
> is nothing that Hunchentoot can do to make these sockets disappear.
>
> I'm not saying that Hunchentoot has no bugs, but from your description, it
> appears to me that your problem is related to your kernel configuration
> (you're hitting the system-wide limit for sockets, not the process limit
> for file descriptors).
>
> -Hans
>
>
> On Wed, Mar 27, 2013 at 1:12 AM, Lucas Hope <lucas.r.hope at gmail.com>wrote:
>
>> Hi there,
>>
>> This is a bug report. I'm using Hunchentoot 1.2.2 from quicklisp. I
>> encountered a server resource limit due to (I think) http connections on an
>> https port.
>>
>> Cliffs notes:
>>
>> https server is not hanging up properly when a connection is attempted
>> using normal http. This eventually causes the process to hit a kernel
>> socket limit.
>>
>> Details:
>>
>> I recently upgraded my hunchentoot webserver from http to full ssl/https.
>> The webservice runs on a dedicated port, and I left the port the same.
>>
>> We had a monitoring service running to check the server was alive.
>> However the monitoring service wasn't updated to use https.
>>
>> So every half an hour or so, we got:
>>
>> [2013-03-26 06:49:26 [ERROR]] Error while processing connection: A
>> failure in the SSL library occurred on handle #.(SB-SYS:INT-SAP #X04B69660)
>> (return code: 1). SSL error queue:
>>
>> error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
>>
>>
>> Also, if I connect myself using http instead of https, the browser
>> connection times out. The server certainly doesn't hang up instantly like
>> I'd expect.
>>
>> Last night, my webserver listener threw a bsd socket error - code 24 (too
>> many files). When I ran an lsof of the process, there were a large number
>> of long term CLOSE_WAIT connections, which my internet research tells me is
>> due to the server not closing connections properly. I think the connections
>> piled up, and I thus hit the socket limit for that process.
>>
>> Anyone else have that issue? Is it an easy fix?
>>
>> Cheers,
>>
>> -Luke
>>
>> _______________________________________________
>> tbnl-devel site list
>> tbnl-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/tbnl-devel
>>
>
>
> _______________________________________________
> tbnl-devel site list
> tbnl-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/tbnl-devel
>



-- 
---------------------------------------------------
Dr Lucas Hope - lucas.r.hope at skype
Machine Learning and Software Engineering Consultant
Melbourne, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/tbnl-devel/attachments/20130401/a894ce8e/attachment.html>