[elephant-devel] Elephant Reorganization in preparation for release 0.6.0

[Ian Eslick]

I just checked in a rather sweeping reorganization of elephant to HEAD
for 0.6.0.  We tagged the HEAD version prior to this with
ELEPHANT-0-6-0-beta-rc1 so anyone making modifications can continue to
work on that until I can help you integrate your changes into the new HEAD.

This change pulled a number of little features planned for 0.6.1 into
0.6.0.  The completed changes are green for Sleepycat under SBCL and ACL
8.0 under Mac OS X. 

I'll check in fixes to make the CLSQL backend work shortly.  I'll also
fix the remaining outstanding items that are partially done and marked
with '*' below. 

More info can be found in the main TODO file.  I also added a src/README
file as a guide to the new sources.

While there is some loss of backward compatibility in these changes, the
impact to existing source code should be minor and easy to catch. 
Interfaces changed include open-store, with-store-controller,
from-root-existsp (just renamed to root-existp), migrate functions and
any code which served as a backend.

- Directory reorganization
  - Broke the source tree into backends, serialization utils and a main
elephant directory

- Build enhancements
  - A user only loads :elephant via asdf.   When backends are created,
they load the
    appropriate additional files via asdf and the generic functions for
that backend
    automatically dispatch on the appropriate store-controller type.  If
the user
    wants to pre-load a backend they can do this via the asdf files in
the top directory
  - Changed the way that backends are registered and managed
  * ASDF will call make on most platforms (no need for manual make) to build
    elephant libraries
  * Compiled C libraries are put into the proper directories
  * non-elephant library location definitions have been lifted to the
top-level asdf files

- Backend factoring
  - Extracted sql and bdb backends into separate directories & packages
  - Clarified the backend interface via generic functions in the main
source files:
    controller, collections, transactions
  - Added a package :elephant-backend for backends to import to get
access to internal
    elephant symbols and all the generic functions they need to
override.  This clarifies what
    symbols are shared between units and keeps any other packages from
seeing them in the
    :elephant exports.
  - To implement a new backend all you need to do is implement the
appropriate generic functions
    for store-controller, btree, indexed-btree, btree-index and
btree-cursor and a few miscellaneous
    ones exported from elephant-backend.
  - Changed the inheritance heirarchy for btrees and factored out some
slots so they were local
     to the different backend.  There was some suspicious sharing going
on earlier. 
  - Backends are added by adding an additional backend table entry to the
    global var *elephant-backends* in variables.lisp which indicates the
function to
    create a store, the asdf dependencies keyed by the db type
  - Check-con is now get-con and takes an instance of type persistent
rather than requiring the
    accessor call at each use
  - Factored out slots from the base store-controller into the new
backend packages

- Serialization
  - I separated serialization from sleepycat into it's own package.  The
berkeley db depends
    on uffi and memory functionality in this sub-package, but binary
serialization and the
    buffer-stream interface is now separated and can be loaded without
loading berkeley-db
  - I also improved allegro's string handling, but this may cause
problems on non Mac OSX
    platforms so please test this!

- Transaction macro
  - Fixed the transaction macro to support different backend via generic
functions and closures
  - No change to the user API however
 * Provided a generic way of creating a thread that binds local vars
depending on what backends are loaded

- New open-store interface
  - open-store takes an extended type spec.  All specs are conses with
the first cons being the type
    of backend you intend to open.  The remainder is as usual.  The sql
backend ensures that the
    asdf files for the specific SQL backend are loaded.
  - with-store-controller is now deprecated for user use
  - For clarity, user API for making indices is make-XX instead of
build-XX which is the backend
    generic function and requires a store controller argument, whereas
make will default to *store-controller*
 
- Improved migration interface
  * I rewrote the migration interface to use a generic function protocol
to allow for user customization
    of the copy process from one repos to another.
  * Added support to copy one repository to another wholesale (a
poor-man's stop-and-copy
    garbage collector!)

Sorry for any hassle this causes the users, but I think this will payoff
in flexibility, clarity and stability in the very near future.

Ian