From bknr at bknr.net Thu Mar 3 07:21:56 2011 From: bknr at bknr.net (BKNR Commits) Date: Thu, 03 Mar 2011 08:21:56 +0100 Subject: [bknr-cvs] edi changed trunk/thirdparty/flexi-streams/ Message-ID: Revision: 4658 Author: edi URL: http://bknr.net/trac/changeset/4658 CMU fix U trunk/thirdparty/flexi-streams/CHANGELOG U trunk/thirdparty/flexi-streams/mapping.lisp Modified: trunk/thirdparty/flexi-streams/CHANGELOG =================================================================== --- trunk/thirdparty/flexi-streams/CHANGELOG 2011-02-16 12:24:35 UTC (rev 4657) +++ trunk/thirdparty/flexi-streams/CHANGELOG 2011-03-03 07:21:56 UTC (rev 4658) @@ -1,3 +1,5 @@ +Fix for CMUCL (Raymond Toy, Xu Jingtao) + Version 1.0.7 2008-08-26 Don't read a second time if the first READ-SEQUENCE already reached EOF (Drakma bug report by Stas Boukarev) Modified: trunk/thirdparty/flexi-streams/mapping.lisp =================================================================== --- trunk/thirdparty/flexi-streams/mapping.lisp 2011-02-16 12:24:35 UTC (rev 4657) +++ trunk/thirdparty/flexi-streams/mapping.lisp 2011-03-03 07:21:56 UTC (rev 4658) @@ -47,7 +47,8 @@ (deftype char-code-integer () "The subtype of integers which can be returned by the function CHAR-CODE." - '(integer 0 #.(1- char-code-limit))) + #-:cmu '(integer 0 #.(1- char-code-limit)) + #+:cmu '(integer 0 65533)) (deftype code-point () "The subtype of integers that's just big enough to hold all Unicode From bknr at bknr.net Wed Mar 9 18:15:45 2011 From: bknr at bknr.net (BKNR Commits) Date: Wed, 09 Mar 2011 19:15:45 +0100 Subject: [bknr-cvs] hans changed trunk/thirdparty/hunchentoot/doc/index.xml Message-ID: Revision: 4659 Author: hans URL: http://bknr.net/trac/changeset/4659 documentation update from Scott McKay and Dan Weinreb to document the resource usage limitation changes. U trunk/thirdparty/hunchentoot/doc/index.xml Modified: trunk/thirdparty/hunchentoot/doc/index.xml =================================================================== --- trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-03 07:21:56 UTC (rev 4658) +++ trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-09 18:15:44 UTC (rev 4659) @@ -214,7 +214,7 @@ Starting your own web server is pretty easy. Do something like this: -
(hunchentoot:start (make-instance 'hunchentoot:acceptor :port 4242))
+
(hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242))
That's it. Now you should be able to enter the address "http://127.0.0.1:4242/" in your browser and see something, albeit nothing very interesting @@ -591,8 +591,9 @@ responsible for settings things up to wait for clients to connect. For each connection which comes in, HANDLE-INCOMING-CONNECTION is applied to - the taskmaster which will call - PROCESS-CONNECTION. + the taskmaster which will either call + PROCESS-CONNECTION directly, + or will create a thread to call it. PROCESS-CONNECTION calls INITIALIZE-CONNECTION-STREAM before it does anything else, then it selects and calls a function which @@ -756,11 +757,48 @@ straightforward to create a taskmaster which allocates threads from a fixed pool instead of creating a new one for each connection. +

- If you want to implement your own taskmasters, you should - subclass TASKMASTER and specialize the - generic functions in this section. + You can control the resources consumed by a threaded taskmaster via + two initargs. :max-thread-count lets you set the maximum + number of request threads that can be processes simultaneously. If + this is nil, the is no thread limit imposed. + + :max-accept-count lets you set the maximum number of requests + that can be outstanding (i.e. being processed or queued for processing). + + If :max-thread-count is supplied and :max-accept-count + is NIL, then a +HTTP-SERVICE-UNAVAILABLE+ + error will be generated if there are more than the max-thread-count + threads processing requests. If both :max-thread-count + and :max-accept-count are supplied, then max-thread-count + must be less than max-accept-count; if more than max-thread-count + requests are being processed, then requests up to max-accept-count + will be queued until a thread becomes available. If more than + max-accept-count requests are outstanding, then a +HTTP-SERVICE-UNAVAILABLE+ + error will be generated. + + In a load-balanced environment with multiple Hunchentoot servers, it's + reasonable to provide :max-thread-count but leave + :max-accept-count null. This will immediately result + in +HTTP-SERVICE-UNAVAILABLE+ when one server is + out of resources, so the load balancer can try to find another server. + + In an environment with a single Hunchentoot server, it's reasonable + to provide both :max-thread-count and a somewhat larger value + for :max-accept-count. This will cause a server that's almost + out of resources to wait a bit; if the server is completely out of resources, + then the reply will be +HTTP-SERVICE-UNAVAILABLE+. + The default for these values is 100 and 120, respectively.

+ +

+ If you want to implement your own taskmasters, you should subclass + TASKMASTER or one of its subclasses, + SINGLE-THREADED-TASKMASTER or + ONE-THREAD-PER-CONNECTION-TASKMASTER, and + specialize the generic functions in this section. +

@@ -822,6 +860,43 @@ the incoming connection by calling the PROCESS-CONNECTION method of the acceptor instance. The socket argument is passed to PROCESS-CONNECTION as an argument. + + If the taskmaster is a multi-threaded taskmaster, HANDLE-INCOMING-THREAD + will call CREATE-TASKMASTER-THREAD, which will call + PROCESS-CONNECTION in a new thread. + HANDLE-INCOMING-THREAD might issue a + +HTTP-SERVICE-UNAVAILABLE+ error + if there are too many request threads or it might block waiting for a + request thread to finish. + + + + + taskmaster socket + + thread + + This function is called by HANDLE-INCOMING-THREAD + to create a new thread which calls PROCESS-CONNECTION. + If you specialize this function, you must be careful to have the thread + call DECREMENT-TASKMASTER-REQUEST-COUNT before + it exits. A typical method will look like this: + +
(defmethod create-taskmaster-thread ((taskmaster monitor-taskmaster) socket)
+            (bt:make-thread
+             (lambda ()
+               (with-monitor-error-handlers
+                 (unwind-protect
+                      (with-monitor-variable-bindings
+                        (process-connection (taskmaster-acceptor taskmaster) socket))
+                   (decrement-taskmaster-request-count taskmaster))))))
+ + + + + + +
From bknr at bknr.net Tue Mar 15 05:36:47 2011 From: bknr at bknr.net (BKNR Commits) Date: Tue, 15 Mar 2011 06:36:47 +0100 Subject: [bknr-cvs] hans changed trunk/thirdparty/hunchentoot/ Message-ID: Revision: 4660 Author: hans URL: http://bknr.net/trac/changeset/4660 Improve standard logging facility by allowing log destinations to be set to a pathname, an open stream or to NIL to suppress logging. U trunk/thirdparty/hunchentoot/acceptor.lisp U trunk/thirdparty/hunchentoot/doc/index.xml U trunk/thirdparty/hunchentoot/log.lisp U trunk/thirdparty/hunchentoot/packages.lisp Modified: trunk/thirdparty/hunchentoot/acceptor.lisp =================================================================== --- trunk/thirdparty/hunchentoot/acceptor.lisp 2011-03-09 18:15:44 UTC (rev 4659) +++ trunk/thirdparty/hunchentoot/acceptor.lisp 2011-03-15 05:36:46 UTC (rev 4660) @@ -131,17 +131,20 @@ :accessor acceptor-shutdown-lock :documentation "The lock protecting the shutdown-queue condition variable and the requests-in-progress counter.") - (access-log-pathname :initarg :access-log-pathname - :accessor acceptor-access-log-pathname - :documentation "Pathname of the access log -file which contains one log entry per request handled in a format - similar to Apache's access.log.") - (message-log-pathname :initarg :message-log-pathname - :accessor acceptor-message-log-pathname - :documentation "Pathname of the server error -log file which is used to log informational, -warning and error messages in a free-text -format intended for human inspection") + (access-log-destination :initarg :access-log-destination + :accessor acceptor-access-log-destination + :documentation "Destination of the access log +which contains one log entry per request handled in a format similar +to Apache's access.log. Can be set to a pathname or string +designating the log file, to a open output stream or to NIL to +suppress logging.") + (message-log-destination :initarg :message-log-destination + :accessor acceptor-message-log-destination + :documentation "Destination of the server +error log which is used to log informational, warning and error +messages in a free-text format intended for human inspection. Can be +set to a pathname or string designating the log file, to a open output +stream or to NIL to suppress logging.") (error-template-directory :initarg :error-template-directory :accessor acceptor-error-template-directory :documentation "Directory pathname that @@ -168,8 +171,8 @@ :persistent-connections-p t :read-timeout *default-connection-timeout* :write-timeout *default-connection-timeout* - :access-log-pathname nil - :message-log-pathname nil + :access-log-destination *error-output* + :message-log-destination *error-output* :document-root (load-time-value (default-document-directory)) :error-template-directory (load-time-value (default-document-directory "errors/"))) (:documentation "To create a Hunchentoot webserver, you make an @@ -399,11 +402,11 @@ (defmethod acceptor-log-access ((acceptor acceptor) &key return-code) "Default method for access logging. It logs the information to the -file determined by (ACCEPTOR-ACCESS-LOG-PATHNAME ACCEPTOR) \(unless -that value is NIL) in a format that can be parsed by most Apache log -analysis tools.)" +destination determined by (ACCEPTOR-ACCESS-LOG-DESTINATION ACCEPTOR) +\(unless that value is NIL) in a format that can be parsed by most +Apache log analysis tools.)" - (with-open-file-or-console (stream (acceptor-access-log-pathname acceptor) *access-log-lock*) + (with-log-stream (stream (acceptor-access-log-destination acceptor) *access-log-lock*) (format stream "~:[-~@[ (~A)~]~;~:*~A~@[ (~A)~]~] ~:[-~;~:*~A~] [~A] \"~A ~A~@[?~A~] ~ ~A\" ~D ~:[-~;~:*~D~] \"~:[-~;~:*~A~]\" \"~:[-~;~:*~A~]\"~%" (remote-addr*) @@ -428,10 +431,10 @@ (defmethod acceptor-log-message ((acceptor acceptor) log-level format-string &rest format-arguments) "Default function to log server messages. Sends a formatted message - to the file denoted by (ACCEPTOR-MESSAGE-LOG-PATHNAME ACCEPTOR). FORMAT and - ARGS are as in FORMAT. LOG-LEVEL is a keyword denoting the log - level or NIL in which case it is ignored." - (with-open-file-or-console (stream (acceptor-message-log-pathname acceptor) *message-log-lock*) + to the destination denoted by (ACCEPTOR-MESSAGE-LOG-DESTINATION + ACCEPTOR). FORMAT and ARGS are as in FORMAT. LOG-LEVEL is a + keyword denoting the log level or NIL in which case it is ignored." + (with-log-stream (stream (acceptor-message-log-destination acceptor) *message-log-lock*) (format stream "[~A~@[ [~A]~]] ~?~%" (iso-time) log-level format-string format-arguments))) Modified: trunk/thirdparty/hunchentoot/doc/index.xml =================================================================== --- trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-09 18:15:44 UTC (rev 4659) +++ trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-15 05:36:46 UTC (rev 4660) @@ -476,7 +476,7 @@ - + acceptor (or pathname null) @@ -497,7 +497,7 @@ - + acceptor (or pathname null) @@ -2767,19 +2767,22 @@ - Hunchentoot can log accesses and diagnostic messages to two separate - files in the file system. Logging to files is enabled and disabled by - setting the ACCESS-LOG-PATHNAME and - MESSAGE-LOG-PATHNAME slots in the - ACCEPTOR instance of the running server, either - by providing the :ACCESS-LOG-PATHNAME and :MESSAGE-LOG-PATHNAME - initialization arguments when creating the acceptor or by setting the - slots through its ACCEPTOR-MESSAGE-LOG-PATHNAME - and ACCEPTOR-ACCESS-LOG-PATHNAME accessors. + Hunchentoot can log accesses and diagnostic messages to two + separate destinations, which can be either files in the file + system or streams. Logging can also be disabled by setting the + ACCESS-LOG-DESTINATION and + MESSAGE-LOG-DESTINATION slots in the + ACCEPTOR instance of the running server, + either by providing the :ACCESS-LOG-DESTINATION and + :MESSAGE-LOG-DESTINATION initialization arguments when creating the + acceptor or by setting the slots through its + ACCEPTOR-MESSAGE-LOG-DESTINATION and + ACCEPTOR-ACCESS-LOG-DESTINATION accessors.

- When the path for the message or accept log is set to NIL, - hunchentoots writes corresponding log entries to the *ERROR-OUTPUT* of - the running Lisp. This is the default. + When the path for the message or accept log is set to a + variable holding an output stream, hunchentoots writes + corresponding log entries to that stream. By default, + Hunchentoot logs to *STANDARD-ERROR*.

Access logging is done in a format similar to what @@ -2798,9 +2801,10 @@ file format.

- Errors happening within a handler which are - not caught by the handler itself are handled by Hunchentoot by logging - them to the log file. + Errors happening within a handler + which are not caught by the handler itself are handled by + Hunchentoot by logging them to the established + ACCEPTOR-MESSAGE-LOG-DESTINATION.

Modified: trunk/thirdparty/hunchentoot/log.lisp =================================================================== --- trunk/thirdparty/hunchentoot/log.lisp 2011-03-09 18:15:44 UTC (rev 4659) +++ trunk/thirdparty/hunchentoot/log.lisp 2011-03-15 05:36:46 UTC (rev 4660) @@ -29,30 +29,40 @@ (in-package :hunchentoot) -(defmacro with-open-file-or-console ((stream-var pathname lock) &body body) +(defmacro with-log-stream ((stream-var destination lock) &body body) "Helper macro to write log entries. STREAM-VAR is a symbol that will be bound to the logging stream during the execution of BODY. -PATHNAME is the pathname designator of the log file or NIL if logging -should be done to *ERROR-OUTPUT*. LOCK refers to the lock that should -be held during the logging operation. If PATHNAME is not NIL, a flexi -stream with UTF-8 encoding will be created and bound to STREAM-VAR. -If an error occurs while writing to the log file, that error will be -logged to *ERROR-OUTPUT*." +DESTINATION is the logging destination, which can be either a pathname +designator of the log file, a symbol designating an open stream or NIL +if logging should be done to *ERROR-OUTPUT*. LOCK refers to the lock +that should be held during the logging operation. If DESTINATION is a +pathname, a flexi stream with UTF-8 encoding will be created and +bound to STREAM-VAR. If an error occurs while writing to the log +file, that error will be logged to *ERROR-OUTPUT*. + +Note that logging to a file involves opening and closing the log file +for every logging operation, which is overall costly. Web servers +with high throughput demands should make use of a specialized logging +function rather than relying on Hunchentoot's default logging +facility." (with-unique-names (binary-stream) - (with-rebinding (pathname) + (with-rebinding (destination) (let ((body body)) - `(if ,pathname - (with-lock-held (,lock) - (with-open-file (,binary-stream ,pathname - :direction :output - :element-type 'octet - :if-does-not-exist :create - :if-exists :append - #+:openmcl #+:openmcl - :sharing :lock) - (let ((,stream-var (make-flexi-stream ,binary-stream :external-format +utf-8+))) - , at body))) - (let ((,stream-var *error-output*)) - (prog1 (progn , at body) - (finish-output *error-output*)))))))) + `(when ,destination + (with-lock-held (,lock) + (etypecase ,destination + ((or string pathname) + (with-open-file (,binary-stream ,destination + :direction :output + :element-type 'octet + :if-does-not-exist :create + :if-exists :append + #+:openmcl #+:openmcl + :sharing :lock) + (let ((,stream-var (make-flexi-stream ,binary-stream :external-format +utf-8+))) + , at body))) + (stream + (let ((,stream-var ,destination)) + (prog1 (progn , at body) + (finish-output *error-output*))))))))))) Modified: trunk/thirdparty/hunchentoot/packages.lisp =================================================================== --- trunk/thirdparty/hunchentoot/packages.lisp 2011-03-09 18:15:44 UTC (rev 4659) +++ trunk/thirdparty/hunchentoot/packages.lisp 2011-03-15 05:36:46 UTC (rev 4660) @@ -39,7 +39,6 @@ #+:lispworks (:import-from :lw #:WITH-UNIQUE-NAMES #:WHEN-LET) (:export #:*ACCEPTOR* - #:*ACCESS-LOG-PATHNAME* #:*APPROVED-RETURN-CODES* #:*CATCH-ERRORS-P* #+:lispworks @@ -61,7 +60,6 @@ #:*LOG-LISP-BACKTRACES-P* #:*LOG-LISP-ERRORS-P* #:*LOG-LISP-WARNINGS-P* - #:*MESSAGE-LOG-PATHNAME* #:*METHODS-FOR-POST-PARAMETERS* #:*REPLY* #:*REQUEST* @@ -121,14 +119,14 @@ #:ABORT-REQUEST-HANDLER #:ACCEPT-CONNECTIONS #:ACCEPTOR - #:ACCEPTOR-ACCESS-LOG-PATHNAME + #:ACCEPTOR-ACCESS-LOG-DESTINATION #:ACCEPTOR-ADDRESS #:ACCEPTOR-DISPATCH-REQUEST #:ACCEPTOR-ERROR-TEMPLATE-DIRECTORY #:ACCEPTOR-INPUT-CHUNKING-P #:ACCEPTOR-LOG-ACCESS #:ACCEPTOR-LOG-MESSAGE - #:ACCEPTOR-MESSAGE-LOG-PATHNAME + #:ACCEPTOR-MESSAGE-LOG-DESTINATION #:ACCEPTOR-NAME #:ACCEPTOR-OUTPUT-CHUNKING-P #:ACCEPTOR-PERSISTENT-CONNECTIONS-P From bknr at bknr.net Tue Mar 15 05:44:16 2011 From: bknr at bknr.net (BKNR Commits) Date: Tue, 15 Mar 2011 06:44:16 +0100 Subject: [bknr-cvs] hans changed trunk/thirdparty/hunchentoot/ Message-ID: Revision: 4661 Author: hans URL: http://bknr.net/trac/changeset/4661 Revised patch by JTK to increase the default listen backlog for acceptors and make the value settable. U trunk/thirdparty/hunchentoot/acceptor.lisp U trunk/thirdparty/hunchentoot/doc/index.xml Modified: trunk/thirdparty/hunchentoot/acceptor.lisp =================================================================== --- trunk/thirdparty/hunchentoot/acceptor.lisp 2011-03-15 05:36:46 UTC (rev 4660) +++ trunk/thirdparty/hunchentoot/acceptor.lisp 2011-03-15 05:44:16 UTC (rev 4661) @@ -114,6 +114,12 @@ :accessor acceptor-listen-socket :documentation "The socket listening for incoming connections.") + #-:lispworks + (listen-backlog :initarg :listen-backlog + :reader acceptor-listen-backlog + :documentation "Number of pending connections + allowed in the listen socket before the kernel rejects + further incoming connections.") (acceptor-shutdown-p :initform nil :accessor acceptor-shutdown-p :documentation "A flag that makes the acceptor @@ -164,6 +170,7 @@ :name (gensym) :request-class 'request :reply-class 'reply + :listen-backlog 50 :taskmaster (make-instance (cond (*supports-threads-p* 'one-thread-per-connection-taskmaster) (t 'single-threaded-taskmaster))) :output-chunking-p t @@ -458,6 +465,7 @@ usocket:*wildcard-host*) (acceptor-port acceptor) :reuseaddress t + :backlog (acceptor-listen-backlog acceptor) :element-type '(unsigned-byte 8))) (values)) Modified: trunk/thirdparty/hunchentoot/doc/index.xml =================================================================== --- trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-15 05:36:46 UTC (rev 4660) +++ trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-15 05:44:16 UTC (rev 4661) @@ -459,6 +459,19 @@
+ + listen-backlog + + number-of-pending-connections + + + + + Number of pending connections allowed in the listen socket + before the kernel rejects further incoming connections. + Non-LispWorks only. + + acceptor From bknr at bknr.net Mon Mar 21 05:17:31 2011 From: bknr at bknr.net (BKNR Commits) Date: Mon, 21 Mar 2011 06:17:31 +0100 Subject: [bknr-cvs] hans changed trunk/thirdparty/hunchentoot/acceptor.lisp Message-ID: Revision: 4662 Author: hans URL: http://bknr.net/trac/changeset/4662 Patch to allow restarting an acceptor after it has been stopped, by Desmond O. Chang. U trunk/thirdparty/hunchentoot/acceptor.lisp Modified: trunk/thirdparty/hunchentoot/acceptor.lisp =================================================================== --- trunk/thirdparty/hunchentoot/acceptor.lisp 2011-03-15 05:44:16 UTC (rev 4661) +++ trunk/thirdparty/hunchentoot/acceptor.lisp 2011-03-21 05:17:30 UTC (rev 4662) @@ -120,7 +120,7 @@ :documentation "Number of pending connections allowed in the listen socket before the kernel rejects further incoming connections.") - (acceptor-shutdown-p :initform nil + (acceptor-shutdown-p :initform t :accessor acceptor-shutdown-p :documentation "A flag that makes the acceptor shutdown itself when set to something other than NIL.") @@ -266,6 +266,7 @@ ;; general implementation (defmethod start ((acceptor acceptor)) + (setf (acceptor-shutdown-p acceptor) nil) (start-listening acceptor) (let ((taskmaster (acceptor-taskmaster acceptor))) (setf (taskmaster-acceptor taskmaster) acceptor) From bknr at bknr.net Tue Mar 29 11:08:10 2011 From: bknr at bknr.net (BKNR Commits) Date: Tue, 29 Mar 2011 13:08:10 +0200 Subject: [bknr-cvs] edi changed trunk/thirdparty/hunchentoot/doc/index.xml Message-ID: Revision: 4663 Author: edi URL: http://bknr.net/trac/changeset/4663 Link to RESTAS U trunk/thirdparty/hunchentoot/doc/index.xml Modified: trunk/thirdparty/hunchentoot/doc/index.xml =================================================================== --- trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-21 05:17:30 UTC (rev 4662) +++ trunk/thirdparty/hunchentoot/doc/index.xml 2011-03-29 11:08:10 UTC (rev 4663) @@ -300,6 +300,8 @@ Here is some software which extends Hunchentoot or is based on it:

    +
  • Andrey Moskvitin's RESTAS framework. +
  • Tomo Matsumoto's web application framework web4r.