[elephant-devel] Concurrency problem?

[Ian Eslick]

The current HEAD has a draft of a reentrant serializer, but only for
allegro and I think SBCL.

If you run db_deadlock on the open BDB database then all the
transactions through BDB are thread safe.

The next release will try to clean this up so it works for all the backends.

I've been out of the country the past two weeks so hope to catch up on
list mail in the next few days.

Cheers,
Ian

Robert L. Read wrote:
> I have had similar problems.  My own solution has been to put mutual
> exclusion locks into
> DCM.  Since my own app only accesses the database through DCM, this
> works for me;
> I think it would be inconvenient but not impossible if you are using
> Elephant directly.
> DCM is a "director-based" system that is in the contrib directory,
> that aims more toward
> a prevalence based system.
>
> There are/were at least three problems that have to be dealt with:
>
> 1)  The serializer is definitely not thread-safe.  In the next
> release, we need to improve this,
> but for the time being I have added mutexes (these are not checked in
> yet.)
>
> 2)  CLSQL is not threadsafe unless you use a separate connection in
> each thread.
> Elephant's basic use of CLSQL reuses the connections, and so you can have
> big problems even at the CLSQL level in using Elephant in that way.
>
> So, based on what you have described, I can suggest a few solutions,
> neither of
> which are great:
>
> 1)  If your access to Elephant is in some relatively narrow place,
> that is, you can find
> a few pieces of code where all of the elephant requests occur, you can
> put mutexes around
> those. (Code to define and use these mutexes is included at the end of
> this message.  It
> defines a macro, "defemthodex", which is just like defmethod but takes
> an extra argument
> used to define the mutex.  However, this is HIGHLY SBCL specific, and
> would not
> be trivial to adapt to anything other than DCM --- this is (in additio
> to being busy) why
> I have not put the thread-safe version of DCM in the contrib directory
> yet.  It is, however,
> an example of how to use mutexes in SBCL, which you can easily copy.)
>
> 2)  Depending on the demands of your application, you might be able to
> put in a smaller
> number of higher-level mutexes (that is, that operate around larger
> bodies of code), and
> still serve pages sufficiently rapidly.
>
> 3)  If you use BDB, it will seem better, but the current version of
> Elephant is still not
> thread-safe, because of the serializer, although it will take you a
> lot longer to hit a
> problem than under CLSQL.
>
> I think really Ian and I, and whoever else is interested, need to plan
> on solving this
> problem in the next major release.  To my knoledge, this would mean:
>
> 1)  Writing a small connection-pool manager that is threadsafe in
> Elephant,
> 2)  Making a reentrant serializer.    (I know Ian is planning this
> already - AMEN!)
> 3)  I'd be happy to publish my current version of DCM to anyone who
> wants it,
> but it is SBCL specific, and I personally don't know how to do mutexes
> under
> any other LISP.
>
> * * *
> I know that may not be terribly helpful ---- but let's continue this
> discussion until
> we work out a solution.  I am extraordinarily busy launching my
> business, but am
> committed to at least managing patches provided by others or getting
> out of the way
> if someone else (such as Ian) wants to be the official maintainer.
>
>
>
> (defvar *dcm-mutexes* (make-hash-table :test 'equal))
>
> (defvar *a-mutex* (sb-thread::make-mutex :name "my lock"))
>
> (defun insure-mutex (name)
>    (let ((mtx (gethash name *dcm-mutexes*))
> )
>      (or mtx (setf (gethash name *dcm-mutexes*) (sb-thread:make-mutex
> :name name)))
>      )
>   )
>
> ;; This assumes that the the variable "dir" is being defined and that
> we can can
> ;; create
> (defmacro defmethodex (mname dir args &body body)
>   `(defmethod ,mname ,(cons dir args)
> ;;    (format t "Thread ~A running ~%" sb-thread::*current-thread*)   
> (sb-thread:with-mutex ((insure-mutex (format nil "mutex-~A" ,(car dir))))
> ;;        (format t "Thread ~A got the lock~%"
> sb-thread::*current-thread*)
> (let ((ret
> , at body))
> ;;        (format t "Thread ~A dropping lock~%"
> sb-thread::*current-thread*)
>   ret
> )
>     )
>   )
>   )
>
>
>
> On Tue, 2006-10-31 at 14:43 +0100, Pierre THIERRY wrote:
>> I may have a problem with concurrency and elephant. I'm developing a web
>> application using SBCL and Araneida (with it's threaded listener).
>> Whereas araneida by irself doesn't seem to have problems dealing with
>> many requests at a time (I tested it with apache's benckmark tool, with
>> thousands of request, with from 1 to 16 concurrent requests to see how
>> it reacts), my application just collapses when 3 requests come in a very
>> short time.
>>
>> I had an error I did not copied, which was about a persistent slot of an
>> object being unbound (but I know it should be bound when accessed at
>> this time), and when trying to access one more time my app, I end up
>> with this in the debugger:
>>
>> While accessing database #<POSTGRESQL-DATABASE localhost/efco-immo/pierre OPEN {A80FB99}>
>>   with expression "SELECT value FROM keyvalue WHERE ((clctn_id = 0) and (key = 'FSAAAAAyAAAANAAAACAAAABUAAAASQAAAFQAAABSAAAARQAAAA=='))":
>>   Error NIL / 
>>   has occurred.
>>    [Condition of type CLSQL-SYS:SQL-DATABASE-DATA-ERROR]
>>
>> The database itself doesn't seem corrupted, as restarting the
>> application from scratch, with the same store, works well. For the
>> moment, I had to shield my app behind an HTTP cache so that requests are
>> basicaly served to the user by the cache instead of my app...
>>
>> Concurrently,
>> Nowhere man
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net <mailto:elephant-devel at common-lisp.net>
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>     
> ------------------------------------------------------------------------
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel