[elephant-devel] Understanding real-world use of Elephant

[Robert L. Read]

I would have to disagree about the documentation for Elephant not being
abundant---Ian has written a 118 page manual.

Nonetheless, you are correct that the use of Elephant in a multi-
threaded webserver environment is not heavily documented.  Ian and I
have discussed the need for a killer "example app" and eagerly await
someone contributing one.

I'm a little rusty on some of this, but if you will agree to take my
opinion about skeptically, I will describe my thoughts on the subject.
My own application, Konsenti, is a multi-threaded web application.

In the first place, one has to go back to basics a little bit.  Whenever
you have concurrency, you have to have concurrency control.  Personally,
I think to think of this at the object level, but I know it is now
common to think of it at the "database level".  You are generally
correct that if you are using SQL (or BDB, for that matter) as a
database and you keep ALL of your state in the database, then you can
generally rely on the concurrency control of those systems to serialize
everything and not allow two threads to interfere with each other.
However, almost ANY application will have to think about concurrency; if
your are SQL-orieneted, you will do this by defining "transactions".  If
you define your transactions wrong, you will have concurrency errors,
even though the database serializes transactions perfectly.

For example, generally, since the Web forces us into "page based" or
"form based" interaction (which javascript, CSS and Ajax promptly
complicate), one can generally think of a web applications as "one-page
turn == one transaction".  But even that might not be true---you could
take 10 page turns to submit an order, and the order must be atomic---
that is, you either order the thing with 10 pages of specifications, or
you don't order it.  A half-order is a corrupt order.

Elephant has the "with-tranaction" macro.  This is really the best way
to use Elephant in most circumstances --- but even then, if you are
keeping ANYTHING in memory, whether to cache it for speed or because you
don't want to put it in the database (session-based information would be
a typical example), you may still have to "serialize" access to that
state.  That unavoidable means that you have to understand some sort of
mutual exclusion lock; unfortunately, these are not 100% standard across
different lisps.  However, most lisps do provide a basic "with-mutex"
facility.  I use this in the DCM code (in the contrib directory) to
serialize access to a "director", which is very similar to a version of
Ian's persistent classes, but based on a "keep it all in memory and push
writes to the DB" model (that is, Prevalence.)

So before we can really answer you question deeply, I think we would
have to understand more about what you are doing.

If you will forgive me over-simplifying things, if:
1) You can legitimately think of every page turn as a transaction, and
2) You keep all of the meaningful state in the Elephant DB, and 
3) You wrap your basic "process-a-page" function in "with-transaction",

then you won't have a concurrency control problem.

That is a completely appropriate style for certain relatively simple
web-applications; for other kinds of web-applications it would be very
constraining and slow --- but one should never optimize for speed before
it is necessary.

I don't know if that's a useful answer --- if you can refine your
question, I will try again.




On Thu, 2007-05-24 at 18:08 -0400, lists at infoway.net wrote:

> Hi all,
> 
> I'm still on my quest to learn to effectively use Elephant. Although documentation is not so abundant, I've gotten a pretty good start with the available documentation.
> 
> However, I don't have such a strong background on using ODBs and mainly come from the SQL world. So, just for curiousity's sake, I read the tutorial for AllegroCache which tries to show the "proper" way to use AllegroCache in real-world systems (http://www.franz.com/products/allegrocache/docs/acachetutorial.pdf).
> 
> Well, conceptwise, I followed everything in that tutorial and could pretty much relate everything to Elephant. However, the truth is that I thought there was a tremendous "overhead" in using AllegroCache when dealing with multiple threads. The reality is that the applications we are thinking on using Elephant for are web-based applications served by multiple web servers, so the practicality of the tutorial was more inline to our objective use of Elephant.
> 
> So, I figured that both systems (Elephant and AllegroCache) have more or less the same usage for practical purposes. But, none of the Elephant documentation I've read on Elephant has even referred to the things mentioned in the tutorial when dealing with multiple threads. I have read on the list that Elephant is thread safe, but am wondering if anyone could help me understand how Elephant would be different from AllegroCache.
> 
> For example, do I still need to handle connection pools and thread locks and all of that in order to do simple multi-threaded requests? Not that I'm looking for an easy way out, but coming from the world of simply "using" a SQL-database and being presented with the needs of having to incorporate a lot of "logic" that the database server would've handled for you into your application seems a bit overwhelming at first.
> 
> I don't know if it has anything to do with the way AllegroCache is architected (e.g. things like isolation between connections, etc) or other factors I'm not aware of. Again, I only read the tutorial in trying to learn more how to "apply the concepts" of ODB so I could understand them better.
> 
> Any comments would be greatly appreciated.
> 
> Thanks,
> Daniel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070524/07dfc300/attachment.html>