[elephant-devel] Concurrency problem?

[Robert L. Read]

I have had similar problems.  My own solution has been to put mutual
exclusion locks into
DCM.  Since my own app only accesses the database through DCM, this
works for me;
I think it would be inconvenient but not impossible if you are using
Elephant directly.
DCM is a "director-based" system that is in the contrib directory, that
aims more toward
a prevalence based system.

There are/were at least three problems that have to be dealt with:

1)  The serializer is definitely not thread-safe.  In the next release,
we need to improve this,
but for the time being I have added mutexes (these are not checked in
yet.)

2)  CLSQL is not threadsafe unless you use a separate connection in each
thread.
Elephant's basic use of CLSQL reuses the connections, and so you can
have 
big problems even at the CLSQL level in using Elephant in that way.

So, based on what you have described, I can suggest a few solutions,
neither of 
which are great:

1)  If your access to Elephant is in some relatively narrow place, that
is, you can find
a few pieces of code where all of the elephant requests occur, you can
put mutexes around
those. (Code to define and use these mutexes is included at the end of
this message.  It 
defines a macro, "defemthodex", which is just like defmethod but takes
an extra argument
used to define the mutex.  However, this is HIGHLY SBCL specific, and
would not 
be trivial to adapt to anything other than DCM --- this is (in additio
to being busy) why 
I have not put the thread-safe version of DCM in the contrib directory
yet.  It is, however,
an example of how to use mutexes in SBCL, which you can easily copy.)

2)  Depending on the demands of your application, you might be able to
put in a smaller
number of higher-level mutexes (that is, that operate around larger
bodies of code), and 
still serve pages sufficiently rapidly.

3)  If you use BDB, it will seem better, but the current version of
Elephant is still not
thread-safe, because of the serializer, although it will take you a lot
longer to hit a 
problem than under CLSQL.

I think really Ian and I, and whoever else is interested, need to plan
on solving this
problem in the next major release.  To my knoledge, this would mean:

1)  Writing a small connection-pool manager that is threadsafe in
Elephant,
2)  Making a reentrant serializer.    (I know Ian is planning this
already - AMEN!)
3)  I'd be happy to publish my current version of DCM to anyone who
wants it,
but it is SBCL specific, and I personally don't know how to do mutexes
under 
any other LISP.

* * *
I know that may not be terribly helpful ---- but let's continue this
discussion until
we work out a solution.  I am extraordinarily busy launching my
business, but am 
committed to at least managing patches provided by others or getting out
of the way
if someone else (such as Ian) wants to be the official maintainer.



(defvar *dcm-mutexes* (make-hash-table :test 'equal))

(defvar *a-mutex* (sb-thread::make-mutex :name "my lock"))

(defun insure-mutex (name)
   (let ((mtx (gethash name *dcm-mutexes*))
 	)
     (or mtx (setf (gethash name *dcm-mutexes*) (sb-thread:make-
mutex :name name)))
     )
  )

;; This assumes that the the variable "dir" is being defined and that we
can can 
;; create 
(defmacro defmethodex (mname dir args &body body)
  `(defmethod ,mname ,(cons dir args)
;;    (format t "Thread ~A running ~%" sb-thread::*current-thread*)    
	(sb-thread:with-mutex ((insure-mutex (format nil "mutex-~A" ,(car
dir))))
;;        (format t "Thread ~A got the lock~%" sb-thread::*current-
thread*)
	(let ((ret
	, at body))
;;        (format t "Thread ~A dropping lock~%" sb-thread::*current-
thread*)
	  ret
	)
    )
  )
  )



On Tue, 2006-10-31 at 14:43 +0100, Pierre THIERRY wrote:

> I may have a problem with concurrency and elephant. I'm developing a web
> application using SBCL and Araneida (with it's threaded listener).
> Whereas araneida by irself doesn't seem to have problems dealing with
> many requests at a time (I tested it with apache's benckmark tool, with
> thousands of request, with from 1 to 16 concurrent requests to see how
> it reacts), my application just collapses when 3 requests come in a very
> short time.
> 
> I had an error I did not copied, which was about a persistent slot of an
> object being unbound (but I know it should be bound when accessed at
> this time), and when trying to access one more time my app, I end up
> with this in the debugger:
> 
> While accessing database #<POSTGRESQL-DATABASE localhost/efco-immo/pierre OPEN {A80FB99}>
>   with expression "SELECT value FROM keyvalue WHERE ((clctn_id = 0) and (key = 'FSAAAAAyAAAANAAAACAAAABUAAAASQAAAFQAAABSAAAARQAAAA=='))":
>   Error NIL / 
>   has occurred.
>    [Condition of type CLSQL-SYS:SQL-DATABASE-DATA-ERROR]
> 
> The database itself doesn't seem corrupted, as restarting the
> application from scratch, with the same store, works well. For the
> moment, I had to shield my app behind an HTTP cache so that requests are
> basicaly served to the user by the cache instead of my app...
> 
> Concurrently,
> Nowhere man
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20061031/5af07b4a/attachment.html>