[elephant-devel] QDBM Support

[Ian Eslick]

In general, I'm with Henrik on this.  I'd rather see us get Elephant  
to a reasonable degree of feature completeness before we start to add  
more non-lisp datastore functionality.  You can use postmodern for  
licensing purposes and BDB for performance.

The answer to all of this, I think, is having a native lisp version  
that has BDB's performance and no licensing restrictions.  Then  
supporting the other two becomes: Postmodern for a higher degree of  
reliability as well as for distributed systems and BDB for legacy  
reasons.

I have a pretty good idea in my head of what an all-lisp backend  
requires and having one would lay to rest all of these discussions of  
bringing up "yet another backend".  Edi Weitz and I discussed  
collaborating on this, but unfortunately he had some other projects  
that took priority.

Is there a small critical mass of people out there that care enough  
about this that they'd be willing to contribute to such a project?  I  
don't have the time to do it on my own, but if we broke it up into  
small projects over the next handful of months, I don't think it's a  
ton of work.  I can put in a solid chunk of integration work in mid to  
late April.

So what is involved?

The tricky problems I've discovered so far are:
- An efficient model of BTree-like storage for Elephant
   1) BDB-like paged data + explicit page cache + operations over fields
   2) Something more customized?
- Efficient pointer-based indexing (BTtree plus ptrs to data in main  
BTree)
- Performing sorting and searching on serialized data rather than  
having to
   deserialize to sort as in the clsql backend (required to do BTree  
insertions)
- Transaction/logging architecture; how to store transaction data,  
track conflicts, etc.
   (at lisp layer, in page cache ala BDB?)
   multi-thread and multi-process safe?
- locking to enable transactions on all 3 platforms; multi-process safe?
   (Is there a free library that has a C library that does this already,
    I think having a simple library that compensates for some of the  
missing
    features in lisps is fine)

Some additional considerations:
- Do we add support for persistent heap garbage collection?
- Do we want to add supports for large persistent sets?
- Do we want a server mode for N:1 distributed transactions?

This is by no means a trivial design, but I think if we sketched out  
the architecture there are a set of subsystems that could be made  
somewhat independent:
- BTrees and disk storage
- Database maintenance ops: (reconstruct DB from log files, dump db,  
optimize, etc)
- transaction support and logging
- low-level locking library
- online garbage collection

Cheers,
Ian

On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote:

> I had never heard of this project, but I it seems that Tokyo Cabinet
> describes itself as fast, has transactions and can handle multiple
> clients which is good. And it has a tcp/ip interface and protocol so
> you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo
> cabinet seems to map to the bdb model good, so it should probably be
> easier to do an interface than the sql interfaces. One observation
> though, do we need yet another backend at this time, there are other
> things to fix first on my personal wishlist.
>
> Henrik
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel