[elephant-devel] Indexing support in HEAD - requesting comments on design

[Ian Eslick]

I'll look out for that if I try to support the interactive transaction
mode later.  Thanks for the heads up.

BDB does seem to be richly featured but poorly abstracted.  I keep
running into lockup conditions and am having trouble figuring out how to
get it to timeout instead of wait.  Even when I set the environment
timeouts it still freezes in a select call inside db_lget (called from
db-put-raw in lisp).  Sigh...

Ian

Ben wrote:
> Hi Ian,
>
> Looks pretty sweet.  About long-lived transactions, I don't really
> know of any problem.  The biggest issue I know of appears to be static
> memory limits in BDB, which make big transactions run out of memory,
> or limits the number of transactions / locks, etc.  That might be a
> problem, but might also be configurable.
>
> In general I have felt a bit uncomfortable with BDB, not that I think
> it is a bad product but it seems to require a lot of arcane knowledge
> to wield properly, something against my original elephant philosophy. 
> On the other hand, the sleepycat developers that hang out on the
> google groups page for sleepycat are fairly responsive and helpful.
>
> Sorry I can't be more help, Ben
>
> On 2/8/06, Ian Eslick <eslick at csail.mit.edu> wrote:
>   
>> To the Elephant Community,
>>
>> The current development HEAD contains beta code for the upcoming 0.6.0
>> release, the major feature of which is providing the option to generate
>> secondary indices on slot values very similar to Statice and Allegro
>> Cache/Store.
>>
>> Independant documentation is forthcoming, however the API in
>> src/indexing.lisp is reasonably documented.  There's also a simple
>> example in src/index-tutorial.lisp and a (potentially out of date) top
>> level file to run the example in tests/RunIndexingTutorial.lisp.  If
>> anyone would like to take a look at this I'm open to suggestions on
>> improving the interface before I commit a naming convention and usage
>> model.
>>
>> Also, does anyone have a problem with me adding a macro defpclass to
>> metaclasses.lisp that handles adding the persistent-metaclass option to
>> a normal defclass?
>>
>> Design decisions:
>>
>> I was reading the Static and AllegroCache docs today which inspired a
>> couple of trivial features to consider adding:
>>
>> - A future ability to declare a slot index to be unique (:indexed t =
>> :any) (:indexed :unique)  My intention is to
>>   change the :indexed t option to use the allegro cache convention of
>> :index :any and :index :any-unique
>>   before we release 0.6.0.
>>
>> - The ability to define secondary index accessors with :inverse-accessor
>> name slot options.  This implicitly
>>   enables the :indexed :any option and creates a defmethod
>>
>> - Class :indexed option (this is the default case in Statice and
>> AllegroCache - all classes are implicitly
>>   indexed which is not the current elephant default).  I'll leave the
>> default to 'off' for backwards compatibility.
>>
>> I've already planned to add something like the Statice 'foreach' macro
>> which is a loop-like syntax that iterates over a set of objects
>> satisfying certain constraints, optionally sorted.  When there are
>> inverse indices available the iteration time will be highly compressed.
>>
>> Transactions:
>>
>> Allegro Cache supports an interactive model where there is an implicit
>> start-transaction if you do any database writes but you don't have to
>> commit until you are ready to commit a set of changes (over seconds or
>> minutes) by calling (commit) or (rollback).  Ben, do you know if BDB
>> will have problems handling long-lived transactions with very few write
>> operations in standalone mode?  This is nice for exploring/testing
>> without commit long-lived changes.
>>
>> If anyone has questions or comments, I'm trying to finalize these
>> tradeoffs and the API naming conventions in the next two weeks so I can
>> release 0.6.0 and move on with my life, so would appreciate input.
>>
>> Also testing help is needed.  The current release works on my
>> Allegro/MacOSX configuration but is failing on SBCL/Linux.  I'm setting
>> up SBCL/MacOSX on my system to test locally but it would be great to
>> accelerate this process.
>>
>> Regards,
>> Ian
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>>