[elephant-devel] Backend musings

[Leslie P. Polzer]

> It strikes me that Tokyo Cabinet's only major benefit is have a better
> licensing model.  Getting it to the same state as BDB would be a
> significant project (and it doesn't support Windows, I believe).

I thought that TC might have a better deadlock handling (after
all there are other concurrency models that don't have that
problem), but I might have been mistaken.



> Mirroring Robert's sentiments, I think we'd be far better off spending
> limited developer cycles on a lisp-only solution running that solves
> both licensing, integration, performance, usability and other
> headaches experienced with our ffi or socket based solutions.

Guess I'm with you both here.



> - Support persistent slot writes

Isn't this trivial with the current serializer architecture?


> - Expose a persistent key-value store, indexed key-value store and
>    iterator/cursor interfaces over them

Comes for free with every B(+)-tree, I guess.


> - Support a transaction model that guarantees the ACID properties
> - Work correctly in a threading model
> - Ideally would also work in a multi-process, shared-memory scenario
>
> I think the big design decisions for a lisp data store include:
> - Backing-store model (prevalence-style logging, binary paging,
> Rucksack style maps, etc)
>
> - Multi-threaded transaction support (page/object locking vs. per-txn
> replication)

OL seems more simple and therefore better to me.
I don't know what per-txn replication is, and neither anything
about the backing store methods you mentioned.



> - Multi-process support?

...should definitely be included. My model of having separate
processes for frontend and admin backend is not too uncommon.



> My latest thinking is that we can spend disk space to save time/
> complexity and assume that we have lots of main memory available.
> This can allow us to avoid messing with fixed-sized pages and locking.

Yup.


> - We serialize commit ops and cancel active transactions that have
> read or written data

What is meant by "serializing" here?



> We can make serialization of btree nodes easier by just serializing
> the lisp structure and having bins of differently sized disk regions
> to write them to rather than maintaining fixed sized pages.  Later, we
> could choose to enhance the system by converting to a low-level read/
> write into the C byte stream that we fetch from files.

Err... "C byte stream"?


> PS - I don't see B+Trees in CL-CONTAINERS.  Is it in there with
> another name?

It should be the QUAD-TREE for all I know.


> Also, we need a B+Tree that actually is mapped to a
> disk file rather than into memory as seems to be the case with most of
> cl-containers

Yes. I'm not sure how easily extensible the CL-CONTAINERS
tree is, but it might be a good base to start with.

  Leslie