From dan at telent.net Fri Jul 9 13:34:15 2004 From: dan at telent.net (Daniel Barlow) Date: Fri, 09 Jul 2004 14:34:15 +0100 Subject: [threads-standard-discuss] MP standardization proposal Message-ID: <87acy9fgo8.fsf@noetbook.telent.net> Welcome to the threads-standard-discuss mailing list. It was created as a result of discussion at the Libre Software Meeting in Bordeaux in July 2004, as a venue for creating a portable de facto standard for use by multithreaded Common Lisp code. I have prepared an initial draft specification entitled BORDEAUX-MP, based on CLIM-SYS (and the needs of McCLIM, which are in fact not quite the same), which is available at http://cvs.telent.net/cgi-bin/viewcvs.cgi/bordeaux-mp/Specification?view=markup Eventually there will also be a sample implementation to go with it, using code borrowed from McCLIM. I invite comments/feedback/flames from all interested parties. Points which have been raised so far include 1) behaviour of specials in a new thread: "did you just make it undefined so that the SBCL implementation would be simpler?" I propose to add a list of specials which automatically get thread-local bindings in each new process: something like ACL's *cl-default-special-bindings* 2) "condition variable" is an unfortunate name as there is already a similarly named but entirely different CONDITION object in CL I don't have any good ideas about this one. 3) A process-state reader might be nice I personally don't see the point, much, unless the semantics of its allowable values are also defined. (Personal position: I'm far more concerned about the ability to write real threaded applications than I am about writing a replacement for top(1) in CL) 4) Some form of control over process priority This might be useful if you have for example a foreground display process that you wish to remain responsive, and a background computation task that only needs to run when nothing else is happening. On the other hand, it might be a fast track to priority inversion and some form of starvation. Discuss. A handy summary of some of the various terms involved is at , for anyone wanting definitions -dan -- "please make sure that the person is your friend before you confirm" -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 188 bytes Desc: not available URL: From dan at loaclhost.telent.net Mon Jul 12 01:22:03 2004 From: dan at loaclhost.telent.net (Daniel Barlow,,,) Date: Mon, 12 Jul 2004 01:22:03 +0000 Subject: [threads-standard-discuss] bordeaux-mp Specification,1.3,1.4 Message-ID: Update of /usr/local/src/cvs/bordeaux-mp In directory loaclhost.telent.net:/tmp/cvs-serv1515 Modified Files: Specification Log Message: rewrite explanation of condition variables Index: Specification =================================================================== RCS file: /usr/local/src/cvs/bordeaux-mp/Specification,v retrieving revision 1.3 retrieving revision 1.4 diff -u -d -r1.3 -r1.4 --- Specification 10 Jul 2004 23:18:45 -0000 1.3 +++ Specification 12 Jul 2004 01:22:01 -0000 1.4 @@ -170,22 +170,18 @@ * Resource contention: condition variables -Condition variables are for use when you have a shared resource with -some kind of state, and the decision as to which process may run next -is dependent on that state. For example: a buffer with readers and a -writer, where the readers can only run when there is data in the -buffer, and the writer can only refill the buffer when it is empty. - -There are three parts to any system involving a condition variable - - - the condition itself. In this example it would be "buffer is (not) -empty" and "buffer is not empty". This is not apparent in the actual -code. - - - a lock to protect the sections of code that evaluate the condition. +A condition variable is a queue of processes which are waiting for +some event. When the event occurs, the queue will be notified, +causing one or more of the stopped processes to be resumed. - - a condition-variable object; this is essentially a queue of -processes waiting for the condition to change. +Condition variables are used to signal events when the state of some +shared resource changes: for example, consider a buffer with readers +and a writer, where the readers can only run when there is data in the +buffer, and the writer can only refill the buffer when it is empty. +By providing an atomic "release lock and sleep" operation, a condition +variable guards against the "lost notification" problem, which is +where the writer adds data to the buffer in between the time that a +reader tests it (finding it empty) and goes to sleep. make-condition-variable () [function] @@ -194,11 +190,11 @@ condition-wait (condition-variable lock) [function] -Atomically release LOCK and enqueue ourselves waiting for +Atomically release LOCK and enqueue the calling process waiting for CONDITION-VARIABLE. The process will resume when another thread has -notified us using CONDITION-NOTIFY, or we are interrupted, or in other +notified it using CONDITION-NOTIFY, or we are interrupted, or in other implementation-dependent circumstances: the caller must always test -the condition on waking before continuing normal processing. +on waking that it can proceed, before continuing normal processing. However and for whatever reason the process is resumed, the system always reacquires LOCK before returning to the caller. It is an error @@ -216,6 +212,8 @@ In an implementation that does not support multiple processes, this function has no effect. +* Miscellaneous + process-yield [Function] Allows other processes to run. It may be necessary or desirable to From strandh at labri.fr Mon Jul 12 12:36:51 2004 From: strandh at labri.fr (Robert Strandh) Date: Mon, 12 Jul 2004 14:36:51 +0200 Subject: [threads-standard-discuss] MP standardization proposal In-Reply-To: <87acy9fgo8.fsf@noetbook.telent.net> References: <87acy9fgo8.fsf@noetbook.telent.net> Message-ID: <16626.34275.857742.927222@serveur5.labri.fr> Daniel Barlow writes: > * Process Creation > make-process function &key name [Function] > > Creates a process named NAME. The new process will evaluate the > function FUNCTION. On systems that do not support multi-processing, > make-process will signal an error. Three problems. First, the function is not evaluated but called (I suppose). Second, we should specify how the function is called. Third, we should say what the return value is. For instance: * Process Creation make-process function &key name [Function] Creates a process named NAME. The new process will call the function FUNCTION with no arguments. The default value for name is NIL. The value returned is the newly created process. On systems that do not support multi-processing, make-process will signal an error. > *default-special-bindings* > > This variable holds an alist associating special variable symbols with > forms to evaluate for binding values. Special variables named in this > list will be locally bound in the new process. > > This may be rebound around calls to MAKE-PROCESS to add/alter default Preferable: This variable may... > bindings. The effect of mutating this list is undefined, but earlier > forms take precedence over later forms for the same symbol, so > defaults may be overridden by consing to the head of the list. > > [ Forms are evaluated in the new thread or in the calling thread? ] I suggest in the old process so that the new process can have its behavior modified by the creating process. > > [ Standard contents of this list: print/reader control, etc. Can > borrow the franz equivalent? ] > > Initial values of other special variables are implementation-defined. > For example, they may be the same as in the process that called > make-process, or set from the global values of the variables. Well, it would be useful to standardize what happens to variables that are not rebound by the process that called make-process. For instance: Initial values of other special variables that have not been bound by the process calling make-process preserve their values. Initial values of other special variables that have been bound by the process calling make-process are implementation-defined. For example ... > The new thread may rebind special variables: the new bindings are > local to that thread. If it assigns to special variables that have > not been bound (either at process creation time or subsequently), the > effect is implementation-defined. I would think relying on the global binding to be shared would be an important programming technique so that you can communicate between processes. No? > process-name process [Function] > > Returns the name of the process, as supplied to MAKE-PROCESS "or NIL if no name was supplied to MAKE-PROCESS." ? > * Atomic operations > > atomic-incf reference [Function] > atomic-decf reference [Function] Were these meant to be "[Macro]"? What is a reference? The same thing as a place? Less general? > Increments (or decrements) the fixnum value referred to by reference > as a single, atomic operation. If the operation would take the value > out of fixnum range, behaviour is undefined. We should specify what value is returned, for instance: The value of the call is the fixnum after incrementation. > * Resource contention: locks and recursive locks > > make-lock &optional name [Function] > > Creates a lock (a mutex) whose name is NAME. If the system does not > support multiple processes this will still return some object, > but it may not be used for very much. > > [ in CLIM-SYS this is a freshly consed list (NIL). I don't know if > there's some good reason it should be said structure or that it be > freshly consed - EQ comparison of locks? ] I do not think there is a good reason to specify what is returned. > acquire-lock lock &optional (wait-p t) [Function] > > Acquire the lock LOCK for the calling process. If the lock is not > available, and if WAIT-P is non-NIL, wait until it is. Specify return values and what happens when WAIT-P is NIL: Acquire the lock LOCK for the calling process. Return true if the lock was successfully acquired, and NIL otherwise. If the lock is not available, and if WAIT-P is non-NIL, wait until it is. Thus if WAIT-P is non-NIL, ACQUIRE-LOCK will always return true (if it returns at all). If WAIT-P is NIL and the lock is not available, ACQUIRE-LOCK immediately returns NIL. > This specification does not define what happens if a process attempts > to acquire a lock that it already holds. For applications that > require locks to be safe when acquired recursively, see instead > MAKE-RECURSIVE-LOCK and friends. > > release-lock lock [Function] > > Release LOCK. It is an error to call this unless the lock has > previously been acquired (and not released) by the same process. If > other processes are waiting for the lock, the ACQUIRE-LOCK call in > one of them will now be able to continue. Specify return value? > with-lock-held (place) &body body [Macro] > > Evaluates BODY with the lock named by PLACE, which is a reference to a > lock created by MAKE-LOCK. Evaluates BODY with the lock named by PLACE, the value of which is a lock created by MAKE-LOCK. Before the forms in BODY are evaluated, the lock is acquired using ACQUIRE-LOCK. After the forms in BODY have been evaluated, the lock is released using RELEASE-LOCK. > The lock will still be released even if a > non-local control transfer (e.g. by THROW or SIGNAL) is signalled, but I suspect one does not say that a throw is "signalled" (which is my book is spelled "signaled" so as to avoid an emphasis on the `a'). Perhaps : The lock will still be released even if a non-local control transfer (e.g. by THROW or SIGNAL) terminates the execution of BODY, but > note that if the debugger is entered, the lock may only be released > /after/ execution has been restarted. "may" is an unfortunate word in English. It could mean that the programmer does not have the right to release it, or that it might be released. > If the process is unable to acquire the lock, BODY will not be > evaluated. This phrase seems superfluous. Perhaps we should allow for a value of WAIT-P to be passed to ACQUIRE-LOCK? If so, specify what happens when WAIT-P is nil and the lock could not be acquired. > make-recursive-lock &optional name [Function] > > Creates a recursive lock whose name is NAME. A recursive lock differs > from an ordinary lock in that a process that already holds the > recursive lock can acquire it again without blocking. The process > must then release the lock twice before it becomes available for > another process. Specify return value. > acquire-recursive-lock lock [Function] Perhaps use different return values depending on whether the lock is already held or not. Why do we need a different function here? > As for ACQUIRE-LOCK, but for recursive locks. > > release-recursive-lock lock [Function] Idem. > Release the recursive LOCK. The lock will only become free after as > many Release operations as there have been Acquire operations. See > RELEASE-LOCK for other information. > > with-recursive-lock-held (place &key timeout) &body body [Macro] Idem. > Evaluates BODY with the recursive lock named by PLACE, which is a > reference to a recursive lock created by MAKE-RECURSIVE-LOCK. See > WITH-LOCK-HELD etc etc -- Robert Strandh --------------------------------------------------------------------- Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp. --------------------------------------------------------------------- From dan at loaclhost.telent.net Mon Jul 12 18:45:18 2004 From: dan at loaclhost.telent.net (Daniel Barlow,,,) Date: Mon, 12 Jul 2004 18:45:18 +0000 Subject: [threads-standard-discuss] bordeaux-mp Specification,1.4,1.5 Message-ID: Update of /usr/local/src/cvs/bordeaux-mp In directory loaclhost.telent.net:/tmp/cvs-serv15887 Modified Files: Specification Log Message: Many edits in response to feedback from Robert Strandh - Rewrite section on special variables to specify what happens to variables which only have global bindings - do we really want/need atomic-{inc,dec}f? - general attention to detail re: return values and arguments for many functions - replaced 'thread' with 'process' in more than a few places - some notes on condition variables, but I think that bit needs rewriting again. Also note that the object returned by all-processes may not be safe to mutate. Index: Specification =================================================================== RCS file: /usr/local/src/cvs/bordeaux-mp/Specification,v retrieving revision 1.4 retrieving revision 1.5 diff -u -d -r1.4 -r1.5 --- Specification 12 Jul 2004 01:22:01 -0000 1.4 +++ Specification 12 Jul 2004 18:45:15 -0000 1.5 @@ -40,7 +40,7 @@ that does not support multiple processes. Process creation and some process inspection operations will not work, but the locking functions are still present (though they may do nothing) so that thread-safe -code can be compiled on both threaded and single-thread +code can be compiled on both multiprocess and single-process implementations without need of conditionals. To avoid conflict with existing MP/threading interfaces in @@ -52,19 +52,43 @@ make-process function &key name [Function] -Creates a process named NAME. The new process will evaluate the -function FUNCTION. On systems that do not support multi-processing, -make-process will signal an error. +Creates and returns a process named NAME, which will call the function +FUNCTION with no arguments: when FUNCTION returns, the process +terminates. NAME defaults to NIL if unsupplied. + +On systems that do not support multi-processing, MAKE-PROCESS will +signal an error. + +The interaction between processes and dynamic variables is in some +cases complex, and depends on whether the variable has only a global +binding (as established by e.g. DEFVAR/DEFPARAMETER/top-level SETQ) +or has been bound locally (e.g. with LET or LET*) in the calling +process. + + - Global bindings are shared between threads: the initial value of a + global variable in the new thread will be the same as in the + parent, and an assignment to such a variable in any thread will be + visible to all threads in which the global binding is visible. + + - Local bindings are local to the thread they are introduced in, + except that + + - Local bindings in the the caller of MAKE-PROCESS may or may not be + shared with the new process that it creates: this is + implementation-defined. Portable code should not depend on + particular behaviour in this case, nor should it assign to such + variables without first rebinding them in the new process. *default-special-bindings* This variable holds an alist associating special variable symbols with forms to evaluate for binding values. Special variables named in this -list will be locally bound in the new process. +list will be locally bound in the new process before it begins +executing user code. -This may be rebound around calls to MAKE-PROCESS to add/alter default -bindings. The effect of mutating this list is undefined, but earlier -forms take precedence over later forms for the same symbol, so +This variable may be rebound around calls to MAKE-PROCESS to add/alter +default bindings. The effect of mutating this list is undefined, but +earlier forms take precedence over later forms for the same symbol, so defaults may be overridden by consing to the head of the list. [ Forms are evaluated in the new thread or in the calling thread? ] @@ -72,15 +96,6 @@ [ Standard contents of this list: print/reader control, etc. Can borrow the franz equivalent? ] -Initial values of other special variables are implementation-defined. -For example, they may be the same as in the process that called -make-process, or set from the global values of the variables. - -The new thread may rebind special variables: the new bindings are -local to that thread. If it assigns to special variables that have -not been bound (either at process creation time or subsequently), the -effect is implementation-defined. - current-process [Function] Returns the process object for the calling process. This is the same @@ -104,6 +119,9 @@ as a single, atomic operation. If the operation would take the value out of fixnum range, behaviour is undefined. +[ do we really want these? is a reference a place? what's the return +value? shouldn't they be macros anyway?] + * Resource contention: locks and recursive locks make-lock &optional name [Function] @@ -118,8 +136,11 @@ acquire-lock lock &optional (wait-p t) [Function] -Acquire the lock LOCK for the calling process. If the lock is not -available, and if WAIT-P is non-NIL, wait until it is. +Acquire the lock LOCK for the calling process. WAIT-P governs what +happens if the lock is not available: if WAIT-P is true, the calling +process will wait until the lock is available and then acquire it; if +WAIT-P is NIL, ACQUIRE-LOCK will return immediately. ACQUIRE-LOCK +returns true if the lock was acquired and NIL otherwise. This specification does not define what happens if a process attempts to acquire a lock that it already holds. For applications that @@ -133,24 +154,27 @@ other processes are waiting for the lock, the ACQUIRE-LOCK call in one of them will now be able to continue. +This function has no interesting return value. + with-lock-held (place) &body body [Macro] -Evaluates BODY with the lock named by PLACE, which is a reference to a -lock created by MAKE-LOCK. The lock will still be released even if a -non-local control transfer (e.g. by THROW or SIGNAL) is signalled, but -note that if the debugger is entered, the lock may only be released -/after/ execution has been restarted. +Evaluates BODY with the lock named by PLACE, the value of which is a +lock created by MAKE-LOCK. Before the forms in BODY are evaluated, +the lock is acquired as if by using ACQUIRE-LOCK. After the forms in +BODY have been evaluated, or if a non-local control transfer is caused +(e.g. by THROW or SIGNAL), the lock is released as if by RELEASE-LOCK. -If the process is unable to acquire the lock, BODY will not be -evaluated. +Note that if the debugger is entered, it is unspecified whether the +lock is released at debugger entry or at debugger exit when execution +is restarted. make-recursive-lock &optional name [Function] -Creates a recursive lock whose name is NAME. A recursive lock differs -from an ordinary lock in that a process that already holds the -recursive lock can acquire it again without blocking. The process -must then release the lock twice before it becomes available for -another process. +Create and return a recursive lock whose name is NAME. A recursive +lock differs from an ordinary lock in that a process that already +holds the recursive lock can acquire it again without blocking. The +process must then release the lock twice before it becomes available +for another process. acquire-recursive-lock lock [Function] @@ -174,6 +198,8 @@ some event. When the event occurs, the queue will be notified, causing one or more of the stopped processes to be resumed. +[ It's not a queue really ] + Condition variables are used to signal events when the state of some shared resource changes: for example, consider a buffer with readers and a writer, where the readers can only run when there is data in the @@ -191,14 +217,15 @@ condition-wait (condition-variable lock) [function] Atomically release LOCK and enqueue the calling process waiting for -CONDITION-VARIABLE. The process will resume when another thread has -notified it using CONDITION-NOTIFY, or we are interrupted, or in other -implementation-dependent circumstances: the caller must always test -on waking that it can proceed, before continuing normal processing. +CONDITION-VARIABLE. The process will resume when another process has +notified it using CONDITION-NOTIFY; it may also resume if interrupted +by some external event or in other implementation-dependent +circumstances: the caller must always test on waking that it can +proceed, before continuing normal processing. However and for whatever reason the process is resumed, the system always reacquires LOCK before returning to the caller. It is an error -to call this unless from the thread that holds LOCK. +to call this unless from the process that holds LOCK. In an implementation that does not support multiple processes, this function signals an error. @@ -209,6 +236,10 @@ The caller needs to acquire the lock associated with the condition variable before calling this. +[ wake-one vs wake-all: when does >1 process get woken ] + + + In an implementation that does not support multiple processes, this function has no effect. @@ -229,7 +260,8 @@ all-processes [Function] -Returns a sequence of all of the processes. +Returns a sequence of all of the processes. This may or may not be +freshly-allocated, so the caller should not modify it. process-interrupt process function [Function] From dan at loaclhost.telent.net Mon Jul 12 23:28:45 2004 From: dan at loaclhost.telent.net (Daniel Barlow,,,) Date: Mon, 12 Jul 2004 23:28:45 +0000 Subject: [threads-standard-discuss] bordeaux-mp Specification,1.5,1.6 Message-ID: Update of /usr/local/src/cvs/bordeaux-mp In directory loaclhost.telent.net:/tmp/cvs-serv21191 Modified Files: Specification Log Message: rewrite condition variables section Index: Specification =================================================================== RCS file: /usr/local/src/cvs/bordeaux-mp/Specification,v retrieving revision 1.5 retrieving revision 1.6 diff -u -d -r1.5 -r1.6 --- Specification 12 Jul 2004 18:45:15 -0000 1.5 +++ Specification 12 Jul 2004 23:28:43 -0000 1.6 @@ -194,20 +194,36 @@ * Resource contention: condition variables -A condition variable is a queue of processes which are waiting for -some event. When the event occurs, the queue will be notified, -causing one or more of the stopped processes to be resumed. +A condition variable provides a mechanism for processes to put +themselves to sleep while waiting for the state of something to +change, then to be subsequently woken by another process which +has changed the state. -[ It's not a queue really ] +A condition variable must be used in conjunction with a lock to +protect access to the state of the object of interest. The procedure +is as follows: -Condition variables are used to signal events when the state of some -shared resource changes: for example, consider a buffer with readers -and a writer, where the readers can only run when there is data in the -buffer, and the writer can only refill the buffer when it is empty. -By providing an atomic "release lock and sleep" operation, a condition -variable guards against the "lost notification" problem, which is -where the writer adds data to the buffer in between the time that a -reader tests it (finding it empty) and goes to sleep. +Suppose two processs A and B, and some kind of notional event channel +C. A is consuming events in C, and B is producing them. CV is a +condition-variable + +1) A acquires the lock that safeguards access to C +2) A processes and removes all events that are present +3) When the channel is empty, A calls CONDITION-WAIT, which atomically + releases the lock and puts A to sleep on CV +4) Loop back to step 1, for as long as processing should continue + +When B generates an event E, it +1) acquires the lock guarding C +2) adds E to the channel +3) releases the lock +4) calls CONDITION-NOTIFY on CV to wake any sleeping process + +The implementation must guarantee that CONDITION-WAIT in process A +atomically release the lock and sleeps. If this does not happen, and +process B can add an event an call CONDITION-NOTIFY between the lock +release and the sleep, the notify call will not see A. This is the +"lost wakeup" problem. make-condition-variable () [function] @@ -220,8 +236,8 @@ CONDITION-VARIABLE. The process will resume when another process has notified it using CONDITION-NOTIFY; it may also resume if interrupted by some external event or in other implementation-dependent -circumstances: the caller must always test on waking that it can -proceed, before continuing normal processing. +circumstances: the caller must always test on waking that there is +processing to be done, instead of assuming that it can go ahead. However and for whatever reason the process is resumed, the system always reacquires LOCK before returning to the caller. It is an error @@ -232,16 +248,16 @@ condition-notify (condition-variable) [function] -Notify one or more of the processes waiting for CONDITION-VARIABLE. -The caller needs to acquire the lock associated with the condition -variable before calling this. - -[ wake-one vs wake-all: when does >1 process get woken ] - - +Notify at least one of the processes waiting for CONDITION-VARIABLE. +It is implementation-dependent whether one or more than one (and +possibly all) processes are woken, but if the implementation is +capable of waking only a single process (not all are) this is probably +preferable for efficiency reasons. The order of wakeup is unspecified +and does not necessarily relate to the order that the processes went +to sleep in. -In an implementation that does not support multiple processes, this -function has no effect. +CONDITION-NOTIFY has no useful return value. In an implementation +that does not support multiple processes, it has no effect. * Miscellaneous @@ -252,7 +268,6 @@ processes automatically. On systems that do not support multi-processing, this does nothing. - * Introspection/debugging The following functions may be provided for debugging purposes, but From dan at loaclhost.telent.net Mon Jul 12 23:41:53 2004 From: dan at loaclhost.telent.net (Daniel Barlow,,,) Date: Mon, 12 Jul 2004 23:41:53 +0000 Subject: [threads-standard-discuss] bordeaux-mp Specification,1.6,1.7 Message-ID: Update of /usr/local/src/cvs/bordeaux-mp In directory loaclhost.telent.net:/tmp/cvs-serv28367 Modified Files: Specification Log Message: condition variables: minor edits Index: Specification =================================================================== RCS file: /usr/local/src/cvs/bordeaux-mp/Specification,v retrieving revision 1.6 retrieving revision 1.7 diff -u -d -r1.6 -r1.7 --- Specification 12 Jul 2004 23:28:43 -0000 1.6 +++ Specification 12 Jul 2004 23:41:50 -0000 1.7 @@ -208,8 +208,8 @@ condition-variable 1) A acquires the lock that safeguards access to C -2) A processes and removes all events that are present -3) When the channel is empty, A calls CONDITION-WAIT, which atomically +2) A processes and removes all events that are available in C +3) When C is empty, A calls CONDITION-WAIT, which atomically releases the lock and puts A to sleep on CV 4) Loop back to step 1, for as long as processing should continue @@ -219,11 +219,12 @@ 3) releases the lock 4) calls CONDITION-NOTIFY on CV to wake any sleeping process -The implementation must guarantee that CONDITION-WAIT in process A -atomically release the lock and sleeps. If this does not happen, and -process B can add an event an call CONDITION-NOTIFY between the lock -release and the sleep, the notify call will not see A. This is the -"lost wakeup" problem. +To avoid the "lost wakeup" problem, the implementation must guarantee +that CONDITION-WAIT in process A atomically release the lock and +sleeps. If this is not guaranteed there is the possibility that +process B can add an event and call CONDITION-NOTIFY between the lock +release and the sleep - in this case the notify call would not see A, +which would be left sleeping despite there being events available. make-condition-variable () [function] From strandh at labri.fr Tue Jul 13 05:08:55 2004 From: strandh at labri.fr (Robert Strandh) Date: Tue, 13 Jul 2004 05:08:55 -0000 Subject: [threads-standard-discuss] MP standardization proposal In-Reply-To: <87acy9fgo8.fsf@noetbook.telent.net> References: <87acy9fgo8.fsf@noetbook.telent.net> Message-ID: <16626.37680.182844.73241@serveur5.labri.fr> I am having a hard time understanding condition variables. > * Resource contention: condition variables > > A condition variable is a queue of processes which are waiting for > some event. When the event occurs, the queue will be notified, > causing one or more of the stopped processes to be resumed. What determines whether one or more than one process gets resumed? Since it is a queue (FIFO), is the first process in always resumed? > Condition variables are used to signal events when the state of some > shared resource changes: for example, consider a buffer with readers > and a writer, where the readers can only run when there is data in the > buffer, and the writer can only refill the buffer when it is empty. OK, I follow this far. > By providing an atomic "release lock and sleep" operation, a condition > variable guards against the "lost notification" problem, which is > where the writer adds data to the buffer in between the time that a > reader tests it (finding it empty) and goes to sleep. I do not see how having a "release lock and sleep" operation fixes the "lost notification" problem. Who would use the operation? The reader or the writer or both? > make-condition-variable () [function] > > Returns a new condition-variable object for use with CONDITION-WAIT > and CONDITION-NOTIFY. > > condition-wait (condition-variable lock) [function] > > Atomically release LOCK and enqueue the calling process waiting for > CONDITION-VARIABLE. The process will resume when another thread has You probably mean `process' as opposed to `thread'. > notified it using CONDITION-NOTIFY, or we are interrupted, How would we get interrupted? > or in other > implementation-dependent circumstances: the caller must always test > on waking that it can proceed, before continuing normal processing. How does it test that it can proceed? Doesn't that test reintroduce the problem that condition-variables were designed to avoid? > However and for whatever reason the process is resumed, the system > always reacquires LOCK before returning to the caller. It is an error > to call this unless from the thread that holds LOCK. > > In an implementation that does not support multiple processes, this > function signals an error. > > condition-notify (condition-variable) [function] > > Notify one or more of the processes waiting for CONDITION-VARIABLE. > The caller needs to acquire the lock associated with the condition > variable before calling this. There has been no mention of "lock associated with condition variable" before. How is this association determined? > In an implementation that does not support multiple processes, this > function has no effect. -- Robert Strandh --------------------------------------------------------------------- Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp. --------------------------------------------------------------------- From gilbert at base-electronic.de Tue Jul 13 17:43:12 2004 From: gilbert at base-electronic.de (Gilbert Baumann) Date: Tue, 13 Jul 2004 19:43:12 +0200 Subject: [threads-standard-discuss] MP standardization proposal In-Reply-To: <16626.37680.182844.73241@serveur5.labri.fr> References: <87acy9fgo8.fsf@noetbook.telent.net> <16626.37680.182844.73241@serveur5.labri.fr> Message-ID: <16628.7984.90566.719267@nx01.base> Robert Strandh writes: > I am having a hard time understanding condition variables. > > > By providing an atomic "release lock and sleep" operation, a condition > > variable guards against the "lost notification" problem, which is > > where the writer adds data to the buffer in between the time that a > > reader tests it (finding it empty) and goes to sleep. > > I do not see how having a "release lock and sleep" operation fixes the > "lost notification" problem. Who would use the operation? The reader > or the writer or both? > The common use case is like this: (defun reader () (with-lock-held (*lock*) (loop (condition-wait *condvar* *lock*) (when )))) (defun writer () (loop (with-lock-held (*lock*) (condition-notify *condvar*)))) The notification cannot be lost since entering CONDITION-WAIT is while the lock is held at which point the writer cannot have it. Only after things are set up within the deep magic of CONDITION-WAIT to actually wait for the condition to be notified the lock is released. Or put the other way round: The reader only releases the lock while being blocked within CONDITION-WAIT. The writer can only notify while holding the lock. => CONDITION-NOTIFY can only be called when the reader is blocked in CONDITION-WAIT => no notifications get lost. qed :) -- Gilbert Baumann Base Electronic GmbH, Norderstedt gilbert at base-electronic.de http://www.base-electronic.de/ From strandh at labri.fr Thu Jul 15 09:18:50 2004 From: strandh at labri.fr (Robert Strandh) Date: Thu, 15 Jul 2004 11:18:50 +0200 Subject: [threads-standard-discuss] MP standardization proposal In-Reply-To: <16628.7984.90566.719267@nx01.base> References: <87acy9fgo8.fsf@noetbook.telent.net> <16626.37680.182844.73241@serveur5.labri.fr> <16628.7984.90566.719267@nx01.base> Message-ID: <16630.19450.719932.867160@serveur5.labri.fr> Hello, Gilbert Baumann writes: > > The common use case is like this: > > (defun reader () > (with-lock-held (*lock*) > (loop > (condition-wait *condvar* *lock*) > (when > )))) Hmmm... This is not what the new version of the API seems to be saying, but instead (defun reader () (acquire-lock *lock*) (loop while do ) (condition-wait *condvar* *lock*)) In fact, I think in your version, release-lock will be called when it is already released (provided that with-lock-held calls release lock. Also, you version seems to release the lock (through calling condition-wait) before consumption starts. Perhaps we need a macro that captures the preceding idiom, like: (defun reader () (with-condition-variable-and-lock (*condvar* *lock*) (loop while do ))) > (defun writer () > (loop > (with-lock-held (*lock*) > > (condition-notify *condvar*)))) This also does not correspond. I interpret the new version of the API like this: (defun writer () (loop (with-lock-held (*lock*) ) (condition-notify *condvar*))) > The notification cannot be lost since entering CONDITION-WAIT is > while the lock is held at which point the writer cannot have it. Only > after things are set up within the deep magic of CONDITION-WAIT to > actually wait for the condition to be notified the lock is released. OK, I think I got it. > Or put the other way round: The reader only releases the lock while > being blocked within CONDITION-WAIT. The writer can only notify while > holding the lock. > > => CONDITION-NOTIFY can only be called when the reader is blocked in > CONDITION-WAIT > => no notifications get lost. > > qed :) Thanks for the explanation. -- Robert Strandh --------------------------------------------------------------------- Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp. --------------------------------------------------------------------- From strandh at labri.fr Thu Jul 15 12:10:20 2004 From: strandh at labri.fr (Robert Strandh) Date: Thu, 15 Jul 2004 14:10:20 +0200 Subject: [threads-standard-discuss] MP standardization proposal In-Reply-To: <87acy1bmam.fsf@noetbook.telent.net> References: <87acy9fgo8.fsf@noetbook.telent.net> <16626.37680.182844.73241@serveur5.labri.fr> <16628.7984.90566.719267@nx01.base> <16630.19450.719932.867160@serveur5.labri.fr> <87acy1bmam.fsf@noetbook.telent.net> Message-ID: <16630.29740.913754.68478@serveur5.labri.fr> Daniel Barlow writes: > Robert Strandh writes: > > > Hmmm... This is not what the new version of the API seems to be > > saying, but instead > > > > (defun reader () > > (acquire-lock *lock*) > > (loop while > > do ) > > (condition-wait *condvar* *lock*)) > > > > In fact, I think in your version, release-lock will be called when it > > is already released (provided that with-lock-held calls release lock. > > condition-wait reacquires the lock before it returns to its caller, so > in fact Gilbert's version is balanced and this one isn't. OK, I see that this fact is indicated in the API proposal. But then this part is not true: 1) A acquires the lock that safeguards access to C 2) A processes and removes all events that are available in C 3) When C is empty, A calls CONDITION-WAIT, which atomically releases the lock and puts A to sleep on CV 4) Loop back to step 1, for as long as processing should continue 4 should say loop back to step 2, right? > > > > (defun writer () > > > (loop > > > (with-lock-held (*lock*) > > > > > > (condition-notify *condvar*)))) > > > > This also does not correspond. I interpret the new version of the API > > like this: > > > > > > (defun writer () > > (loop (with-lock-held (*lock*) > > ) > > (condition-notify *condvar*))) > > I think this is an API bug, and Gilbert's approach - *lock* must be > held when calling condition-notify - is correct. It's certainly the > way that Java does it (Object.wait/Object.notify) No wonder I am having a hard time figuring this out when it is so hard to explain what it is supposed to do. -- Robert Strandh --------------------------------------------------------------------- Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp. --------------------------------------------------------------------- From dan at loaclhost.telent.net Thu Jul 15 15:26:54 2004 From: dan at loaclhost.telent.net (Daniel Barlow,,,) Date: Thu, 15 Jul 2004 15:26:54 +0000 Subject: [threads-standard-discuss] bordeaux-mp Specification,1.7,1.8 Message-ID: Update of /usr/local/src/cvs/bordeaux-mp In directory loaclhost.telent.net:/tmp/cvs-serv1400 Modified Files: Specification Log Message: more bug fixes, courtesy Robert Strandh Index: Specification =================================================================== RCS file: /usr/local/src/cvs/bordeaux-mp/Specification,v retrieving revision 1.7 retrieving revision 1.8 diff -u -d -r1.7 -r1.8 --- Specification 12 Jul 2004 23:41:50 -0000 1.7 +++ Specification 15 Jul 2004 15:26:51 -0000 1.8 @@ -211,20 +211,22 @@ 2) A processes and removes all events that are available in C 3) When C is empty, A calls CONDITION-WAIT, which atomically releases the lock and puts A to sleep on CV -4) Loop back to step 1, for as long as processing should continue +4) Wait to be notified; CONDITION-WAIT will acquire the lock again + before returning +5) Loop back to step 2, for as long as processing should continue When B generates an event E, it 1) acquires the lock guarding C 2) adds E to the channel -3) releases the lock -4) calls CONDITION-NOTIFY on CV to wake any sleeping process +3) calls CONDITION-NOTIFY on CV to wake any sleeping process +4) releases the lock To avoid the "lost wakeup" problem, the implementation must guarantee -that CONDITION-WAIT in process A atomically release the lock and +that CONDITION-WAIT in process A atomically releases the lock and sleeps. If this is not guaranteed there is the possibility that process B can add an event and call CONDITION-NOTIFY between the lock release and the sleep - in this case the notify call would not see A, -which would be left sleeping despite there being events available. +which would be left sleeping despite there being an event available. make-condition-variable () [function] From bmastenb at cs.indiana.edu Wed Jul 21 17:25:23 2004 From: bmastenb at cs.indiana.edu (Brian Mastenbrook) Date: Wed, 21 Jul 2004 12:25:23 -0500 (EST) Subject: [threads-standard-discuss] Worker thread pools and some kind of thread-finished-p predicate Message-ID: Howdy, Araneida uses a worker thread pool to generate its HTTP responses when running in threaded mode. In order to replenish the pool, it needs to determine whether a thread is "alive" in some metaphorical sense or is "finished". I know Dan has expressed reservation over some kind of general process state function, but I believe that alive versus finished is appropriate to all known threading systems, and furthermore necessary to be able to do serious work with thread pools. To that end I would propose that a predicate thread-finished-p should be added, which returns true when a thread is no longer eligible for execution, either by normal or abnormal termination. Thoughts? -- Brian Mastenbrook "God made the natural numbers; http://www.cs.indiana.edu/~bmastenb/ all else is the work of man." bmastenb at cs.indiana.edu -- Leopold Kroneker From peter at armedbear.org Wed Jul 21 18:22:10 2004 From: peter at armedbear.org (Peter Graves) Date: Wed, 21 Jul 2004 11:22:10 -0700 Subject: [threads-standard-discuss] Worker thread pools and some kind of thread-finished-p predicate In-Reply-To: References: Message-ID: <20040721112210.838@prufrock> On Wed, 21 Jul 2004 at 12:25:23 -0500, Brian Mastenbrook wrote: > To that end I would propose that a predicate thread-finished-p should > be added, which returns true when a thread is no longer eligible for > execution, either by normal or abnormal termination. I like the idea of this predicate, but I'd rather see it called THREAD- ALIVE-P, instead of THREAD-FINISHED-P. "Finished" might carry some vague connotation that the thread has run through to an orderly completion ("I've finished my work), but "alive" is just the opposite of dead. -Peter