[elephant-devel] Representational Question

[Robert L. Read]

On Thu, 2008-03-06 at 10:10 -0500, Ian Eslick wrote:
> I agree with Robert.  The best way to start is to use lisp as a
> query  
> language and essential do a search/match over the object graph.
> 
> The rub comes when you start looking at performance.  A linear scan
> of   

I neglected to mention that in my use of Elephant, when I was attempting
to run a commercial website, I was using the Data Collection Management
(DCM) stuff that you can find in the contrib/rread directory of the
project.

This system provides strategy-based directors.  That is, there is a
basic factory object for each collection of objects that implements
basic Create, Read, Update, Delete operations.

When you initialize a director, you specify a storage strategy:

*) In-memory hash, (no persistence, for transient objects)
*) Elephant (no caching)
*) Cache backed by Elephant (read in memory, with writes immediately
flushed to the store)
*) Generational system, in which each generation can have its own
storage strategy.

Everything Ian wrote in the last email about scanning and locality of
reference makes perfect sense, but is assuming that you don't have every
object cached.  That approach is therefore not very "Prevalence"-like in
its performance, but is very "Prevalence"-like in its convenience.
Using DCM, or any other caching where most of the object are cached,
tends to you go the performance described in the IBM article on
Prevalence that I referenced.

However, DCM was written BEFORE Ian got the class indexing and
persistence working.  DCM is not nearly as pretty and clean as the
persistent classes.  You end up having to make storage decisions
yourself.

A perfect system might be persistent classes with really excellent
control over the caching/write-updating policy.

For any application, I a would recommend using Ian's persistent classes
at the beginning project stages, and then when your performance tests
reveal you have a problem, consider at that point whether to add
indexes, move to explicitly keeping a class in memory, or some other
solution.