[elephant-devel] QDBM Support

[Ben]

one (perhaps insane) idea to make an all-lisp backend easier to
implement was to leverage the underlying file system ala ZODB
directory storage, since the file system is probably using B-trees
anyways.  there are fairly good architecture docs on

http://dirstorage.sourceforge.net/

tokyo cabinet looks good too.

b

On Feb 13, 2008 7:41 AM, Ian Eslick <eslick at media.mit.edu> wrote:
> In general, I'm with Henrik on this.  I'd rather see us get Elephant
> to a reasonable degree of feature completeness before we start to add
> more non-lisp datastore functionality.  You can use postmodern for
> licensing purposes and BDB for performance.
>
> The answer to all of this, I think, is having a native lisp version
> that has BDB's performance and no licensing restrictions.  Then
> supporting the other two becomes: Postmodern for a higher degree of
> reliability as well as for distributed systems and BDB for legacy
> reasons.
>
> I have a pretty good idea in my head of what an all-lisp backend
> requires and having one would lay to rest all of these discussions of
> bringing up "yet another backend".  Edi Weitz and I discussed
> collaborating on this, but unfortunately he had some other projects
> that took priority.
>
> Is there a small critical mass of people out there that care enough
> about this that they'd be willing to contribute to such a project?  I
> don't have the time to do it on my own, but if we broke it up into
> small projects over the next handful of months, I don't think it's a
> ton of work.  I can put in a solid chunk of integration work in mid to
> late April.
>
> So what is involved?
>
> The tricky problems I've discovered so far are:
> - An efficient model of BTree-like storage for Elephant
>    1) BDB-like paged data + explicit page cache + operations over fields
>    2) Something more customized?
> - Efficient pointer-based indexing (BTtree plus ptrs to data in main
> BTree)
> - Performing sorting and searching on serialized data rather than
> having to
>    deserialize to sort as in the clsql backend (required to do BTree
> insertions)
> - Transaction/logging architecture; how to store transaction data,
> track conflicts, etc.
>    (at lisp layer, in page cache ala BDB?)
>    multi-thread and multi-process safe?
> - locking to enable transactions on all 3 platforms; multi-process safe?
>    (Is there a free library that has a C library that does this already,
>     I think having a simple library that compensates for some of the
> missing
>     features in lisps is fine)
>
> Some additional considerations:
> - Do we add support for persistent heap garbage collection?
> - Do we want to add supports for large persistent sets?
> - Do we want a server mode for N:1 distributed transactions?
>
> This is by no means a trivial design, but I think if we sketched out
> the architecture there are a set of subsystems that could be made
> somewhat independent:
> - BTrees and disk storage
> - Database maintenance ops: (reconstruct DB from log files, dump db,
> optimize, etc)
> - transaction support and logging
> - low-level locking library
> - online garbage collection
>
> Cheers,
> Ian
>
>
> On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote:
>
> > I had never heard of this project, but I it seems that Tokyo Cabinet
> > describes itself as fast, has transactions and can handle multiple
> > clients which is good. And it has a tcp/ip interface and protocol so
> > you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo
> > cabinet seems to map to the bdb model good, so it should probably be
> > easier to do an interface than the sql interfaces. One observation
> > though, do we need yet another backend at this time, there are other
> > things to fix first on my personal wishlist.
> >
> > Henrik
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>