[elephant-cvs] CVS elephant

Mon Sep 4 05:42:43 UTC 2006

Update of /project/elephant/cvsroot/elephant
In directory clnet:/tmp/cvs-serv17940

Modified Files:
	TODO 
Log Message:
Further rethink of roadmap and TODO tasks

--- /project/elephant/cvsroot/elephant/TODO	2006/09/04 05:01:05	1.24
+++ /project/elephant/cvsroot/elephant/TODO	2006/09/04 05:42:43	1.25
@@ -8,28 +8,19 @@
 
 Bugs or Observations:
 
-Multi-threading operation:
-- Make elephant thread-bound variables dynamic and modifiable by backends
-- Ensure serialization is multi-threaded
-- Verify that operations such as indexing are thread safe
-
 Stability:
 - Review all the NOTE comments in the code
 - Remove build gensym warnings in sleepycat
-- Port elephant to closer-to-MOP to make it easier to support additional lisps (Both)
-- (From Ben's e-mail) We are storing persistent objects incorrectly. They should be
-  stored only as OIDs, and we should have a separate OID->class table.  This way 
-  change-class can be handled correctly.  This also non-trivially compresses storage 
-  in the database as we only need to store oids rather than serialized class names.
-  [Ian comment: only problem with this is an extra access to oid table each time a 
-   class is deserialized and overall storage is constant.  Would make it easy to 
-   invalidate objects though!]
+- Remove sleepycat name.  Change sleepycat to db-bdb to reflect oracle ownership and avoid
+  confusion for new users
 - Delete persistent slot values from the slot store with remove-kv to ensure that
   there's no data left lying around if you define then redefine a class and add
   back a persistent slot name that you thought was deleted and it gets the old
   value by default.
+- Cleaner failure modes if operations are performed without repository or without
+  transaction or auto-commit (Both)
 
-Stores: 
+Store variables:
 - Think through default *store-controller* vs. explicit parameter passing 
   referencing all over the APIs 
 - Think about dynamic vs. object based store & transaction resolution?
@@ -37,57 +28,74 @@
   - Current store specific *current-transaction* stack
 - Throw condition when store spec is invalid, etc
 
-Transactionalism:
-- Cleaner failure modes if operations are performed without repository or without
-  transaction or auto-commit (Both)
+Multi-threading operation:
+- Make elephant threads appropriately bind dynamic variables
+- Verify that operations such as indexing are thread safe
 
 BDB Features:
+~ Automatically run db_deadlock when opening a bdb backend?  Requires path to
+  functions and ability to launch shell command.  Closing the store stops the
+  sub-process.
+- Always support locks that timeout?  Tradeoffs?
+- Roll deprecation of *auto-commit* through code base so leaf functions stop referring to it
 - Trace all paths to db-put or db-delete and ensure that there is a check or a 
   default with-transaction around the primitive components - write a document 
   clarifying transaction design & assumptions in the backend]  Add asserts if 
   *auto-index* is false and we're not in a transaction to help users avoid lockups 
   in bdb?  Should be able to turn off for performance but it will help catch 
   missing with-transaction statemetns in user code. (Both)
-~ Automatically run db_deadlock when opening a bdb backend?  Requires path to
-  functions and ability to launch shell command.  Closing the store stops the
-  sub-process.
-- Always support locks that timeout?  Tradeoffs?
+- Figure out how to compact a specific btree and/or key-range using optimize-storage.
+  Probably need to update keyword part of the API
+
+Indexing efficiency and policies:
+- Add :inverse-reader to slot options to create a named method that indexes into objects 
+  based on slot values.  Is this a GF or defun?  Do we dispatch on class name or bake it in?
+- Reclaim table storage on index drop?  It's nice to be able to reconnect sometimes!
+  Perhaps an API command that allows explicit dropping of tables for a class and a policy
+  parameter that determines if this is the default?
+- Should we delete slot-values in the db when redefining classes, currently those values
+  stay around - probably indefinitely unless we GC
 
 Performance:
 - Metering and understanding locking issues.  Large transactions seem
-  to use a lot of locks.  In general understanding how to use Sleepycat
+  to use a lot of locks.  In general understanding how to use Berkeley DB
   efficiently seems like a good thing. (From Ben)
 - Add dependency information into secondary index callback functions so that
   we can more easily compute which indices need to be updated to avoid the
   global remove/add in order to maintain consistency (Ian)
 - Improve SQL serializer performance (Robert/Ian)
 
-Indexing features:
-- Add :inverse-reader to slot options to create a named method that indexes into objects 
-  based on slot values.  Is this a GF or defun?  Do we dispatch on class name or bake it in?
-
-Compliance and Efficiency:
-  - Update to support BDB 4.4
-    - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs
-  - Reclaim table storage on index drop 
-  - Should we delete slot-values in the db when redefining classes, currently those values
-    stay around - probably indefinitely unless we GC
+Test coverage:
+- Test for optimize storage method (just add probe-file methods to get file size)
+- Multi-threading stress tests?  Ensure that there are conflicts and lots of serialization
+  happening concurrently to make sure that multi-threading is in good shape
+
+Documentation:
+- Add notes about with-transaction usage (abort & commit behavior on exit)
+- Add notes about optimize-storage
+- Add notes about new BDB 4.4 *auto-commit* behavior.  Default for entire store-controller,
+  will auto create a transaction if none is active if open with :auto-commit t or will
+  never auto-commit (regardless of operator flags) if it is not.  Make sure open-store
+  defaults to auto-commit and there is a flag to turn it off.
 
 0.6.1 - Features COMPLETED to date
 ----------------------------------
+x Ensure serialization is multi-threaded and efficient
 x Determine how to detect deadlock conditions as an optional run-safe mode?
 x BDB overwrite of values makes DB grow 
   [So far I can only find that it grows on the 2nd write, but not after that...artifact of
    page allocation or caching of memory pools?]
 x FEATURE: Investigate BDB record size; it's 2x larger than expected? 
   [Ditto above]
+x Update to support BDB 4.4
+  x Add ability from within lisp to reclaim DB space after deleting btree key-value pairs
 
-0.6.2 - Query & indexing expansion (Fall '06)
+
+0.6.2 - Advanded work, low-hanging fruit (Fall '06)
 --------------------------------------------------
-  - Simple object query language (Ian - orthogonal, on main branch)
-  - Add needed support (if any) for persistent graph structures & queries (Ian on a branch)
+  - Port elephant to closer-to-MOP to make it easier to support additional lisps and to
+    seriously clean up metaclasses.lisp and classes.lisp protocols
   - A wrapper around migration that emulates a stop-and-copy GC
-  - Fast serializer port & upgrade strategy
 
 0.6.3 - Documentation & Tools (Winter '06)
 --------------------------------------------------
@@ -99,21 +107,19 @@
     - A guide to dealing with multiple open stores
     - A guide to performance
     - An overview of licensing issues...
-  - Repository browser (Ian - orthogonal, on main branch)
-    (a simple REPL tool to see what classes are in a repository and
-     what state they're in...useful for long-lived repositories)
 
-0.6.4 - Additional datastructures (?)
+0.7.0: Fast In-Memory Database (Not backwards compatible)
 --------------------------------------------------
-  - Native BDB persistent hashes (easy; can do on SQL backends?)
-  - Support for cheap persistent sets (medium? can do on SQL?)
-
-Some placeholders & dreams features below... :)
-
-0.7+: Major features (Winter '07)
---------------------------------------------------
-  - A native lisp backend controller (Ian)
-  - Integrate prevalence-like in-memory database system (Robert?)
+  - Integrate prevalence-like in-memory database system 
+  - Fast serializer port w/ upgrade strategy and prevalence like storage solution
+    - Further improve SQL 64-bit serialization performance (if possible)
+  - (From Ben's e-mail) We are storing persistent objects incorrectly. They should be
+    stored only as OIDs, and we should have a separate OID->class table.  This way 
+    change-class can be handled correctly.  This also non-trivially compresses storage 
+    in the database as we only need to store oids rather than serialized class names.
+    [Ian comment: only problem with this is an extra access to oid table each time a 
+    class is deserialized and overall storage is constant.  Would make it easy to 
+    invalidate objects though!]
   - Richer controller modes: 
     - Single-user mode (cache values in instance slots for fast reads, write-through)
     - Prevalence mode (read/write to normal slots except on object creation or synch)
@@ -122,6 +128,31 @@
     - Concurrent mode (for backends that allow multiple processes to connect, current default)
     - Controller 'switches'
       - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance
+  - Usage model examples
+
+0.7.1 - Elephant/BDB/SQL Production Release
+--------------------------------------------------
+  - More work on testing, examples and documentation
+  - Intent is for this to be a major, long-term supported release prior 
+    to work on the new backend
+
+0.7.2 - Additional Tools
+--------------------------------------------------
+  - Add needed support (if any) for persistent graph structures & queries (Ian on a branch)
+  - Simple object query language (Ian - orthogonal, on main branch)
+  - Repository browser (Ian - orthogonal, on main branch)
+    (a simple REPL tool to see what classes are in a repository and
+     what state they're in...useful for long-lived repositories)
+
+0.8.0 - Native Backend & Datastructure Library (
+--------------------------------------------------
+  - A native lisp backend controller (Ian)
+  - Native BDB persistent hashes (easy; can do on SQL backends?)
+  - Support for cheap persistent sets (medium? can do on SQL?)
+  - Usage model examples
+
+
+
 
 ========================================================
 ========================================================


