[elephant-cvs] CVS elephant

Wed Apr 26 17:53:43 UTC 2006

Update of /project/elephant/cvsroot/elephant
In directory clnet:/tmp/cvs-serv12311

Modified Files:
	NOTES TODO elephant.asd 
Log Message:

Significant additions to the 0.6.0 release on the trunk.  Updates to 
documentation, 0.5.0 compliance, final on 0.6.0 features.  There are
one or two BDB interactions on migration to work out but this should
be a nearly code complete 0.6.0 release.  Please start testing and
evaluating this - especially the ability to open and tag 0.5.0 databases.

Features:
- Database version tagging
- Support for 0.5.0 namespaces & databases
- New migration system
- class indexing without slot indexing
- various bug fixes
- reverted fast allegro/sbcl string support to
  allow 0.5.0 databases to work correctly.  I 
  couldn't find a good way to work around this
  without creating infinite headaches
- validated that running db_deadlock will stop all
  lisp freezes that I've encountered.  This has to
  be run each time a DB environment is opened/created
  so eventually should be made part of the open-controller
  functionality for the BDB backend

--- /project/elephant/cvsroot/elephant/NOTES	2006/01/25 21:52:54	1.6
+++ /project/elephant/cvsroot/elephant/NOTES	2006/04/26 17:53:43	1.7
@@ -3,10 +3,9 @@
 GENERAL
 -------
 
-this has been optimized for use with CMUCL / SBCL.  it has
-been tested and somewhat optimized for allegro.  OpenMCL is
-definitely also a target.  Lispworks is a target as well but
-less so: i don't have access to it.
+this has been optimized for use with CMUCL / SBCL / Allegro.  
+OpenMCL has been minimally supported.  Lispworks is a target as well 
+but less so as the developers don't have access to it.
 
 Theoretically one can port this to any lisp with a decent
 FFI and MOP.  However since those are two of the less
@@ -19,7 +18,7 @@
 persistent meta-object, persistent collections
 controller
 serializer
-SLEEPYCAT package
+memutils package
 UFFI / implementation specific stuff
 libsleepycat.so
 Sleepycat 4.2/3
--- /project/elephant/cvsroot/elephant/TODO	2006/03/07 14:12:22	1.21
+++ /project/elephant/cvsroot/elephant/TODO	2006/04/26 17:53:43	1.22
@@ -1,13 +1,27 @@
 
-Feb 23, 2006
+April 23, 2006
 
-Release plan in-discussion with Robert and Ian
+Ongoing release plan notes
+
+Features completed in 0.6.0:
+-----------------------------------
+x Add a class-indexing class option to the metaclass so we can maintain class instances
+  index without any secondary indices or indexed slots 
+x Upgrade solution to 0.6.0, DB properties & version tag for future upgrades 
+x Validate migration
+x Documentation update
+x Indexing tutorial and tutorial review 
 
-0.6.0 - Adding default class/slot indexing; modularize backends
-- Documentation update
-- Tutorial review 
 
 0.6.1 - performance, safety and portability
+--------------------------------------------------
+
+Bugs or Observations:
+- BDB overwrite of values makes DB grow 
+  [So far I can only find that it grows on the 2nd write, but not after that...artifact of
+   page allocation or caching of memory pools?]
+- FEATURE: Investigate BDB record size; it's 2x larger than expected? 
+  [Need a good test for this to follow up]
 
 Multi-threading operation:
 - Make elephant thread bound variables dynamic and modifiable by backends
@@ -21,55 +35,63 @@
   stored only as OIDs, and we should have a separate OID->class table.  This way 
   change-class can be handled correctly.  This also non-trivially compresses storage 
   in the database as we only need to store oids rather than serialized class names.
+  [Ian comment: only problem with this is an extra access to oid table each time a 
+   class is deserialized and overall storage is constant.  Would make it easy to 
+   invalidate objects though!]
 - Delete persistent slot values from the slot store with remove-kv to ensure that
   there's no data left lying around if you define then redefine a class and add
   back a persistent slot name that you thought was deleted and it gets the old
-  value
+  value by default.
 
 Stores: 
+- Think through default *store-controller* vs. explicit parameter passing 
+  referencing all over the APIs 
 - Think about dynamic vs. object based store & transaction resolution?
-  - Error checking when mixed 
+  - Perform error checking when mixed 
   - Current store specific *current-transaction* stack
-- Think through default vs. explicit store referencing all over the APIs (Both)
 - Throw condition when store spec is invalid, etc
 
 Transactionalism:
 - Cleaner failure modes if operations are performed without repository or without
   transaction or auto-commit (Both)
+
+BDB Features:
 - Trace all paths to db-put or db-delete and ensure that there is a check or a 
   default with-transaction around the primitive components - write a document 
   clarifying transaction design & assumptions in the backend]  Add asserts if 
   *auto-index* is false and we're not in a transaction to help users avoid lockups 
   in bdb?  Should be able to turn off for performance but it will help catch 
   missing with-transaction statemetns in user code. (Both)
-- BDB: determine how to detect deadlock conditions as an optional run-safe mode? (?)
-  Does BDB have timeouts enabled on select? (Ian)
+- Determine how to detect deadlock conditions as an optional run-safe mode?
+- Automatically run db_deadlock when opening a bdb backend?  Requires path to
+  functions and ability to launch shell command.  Closing the store stops the
+  sub-process.  
+- Always support locks that timeout?  Tradeoffs?
 
 Performance:
 - Metering and understanding locking issues.  Large transactions seem
   to use a lot of locks.  In general understanding how to use Sleepycat
   efficiently seems like a good thing. (From Ben)
-- Reclaim table storage on index drop (Ian)
 - Add dependency information into secondary index callback functions so that
   we can more easily compute which indices need to be updated to avoid the
   global remove/add in order to maintain consistency (Ian)
+- Improve SQL serializer performance (Robert)
 
 Indexing features:
-- Add a class-indexing class option to the metaclass so we can maintain class instances
-  index without any secondary indices or indexed slots (Ian)
 - Add :inverse-reader to slot options to create a named method that indexes into objects 
   based on slot values.  Is this a GF or defun?  Do we dispatch on class name or bake it in?
 
-Bugs:
-- anything else reported against 0.5.0/0.6.0
-
 0.6.3 - Query & indexing expansion
+--------------------------------------------------
   - simple object query language (Ian - orthogonal, on main branch)
   - Add needed support (if any) for persistent graph structures & queries (Ian on a branch)
   - A wrapper around migration that emulates a stop-and-copy GC
 
 0.6.4 - Compliance & Documentation
+--------------------------------------------------
   - Update to support BDB 4.4
+    - Add ability from within lisp to reclaim DB space after deleting btree key-value pairs
+  - Reclaim table storage on index drop 
   - Tutorial example rethink: update the blog tutorial using indexed 
     objects to create different views as well as integrating something 
     like logging for admin or version control purposes.
@@ -83,12 +105,14 @@
      what state they're in...useful for long-lived repositories)
 
 0.6.5 - Additional datastructures?
+--------------------------------------------------
   - Native BDB persistent hashes (easy; can do on SQL backends?)
   - Support for cheap persistent sets (medium? can do on SQL?)
 
 Some placeholders & dreams features below... :)
 
 0.7+: Major features
+--------------------------------------------------
   - A backend controller for AllegroCache (Ian)
   - Prevalence-like in-memory database system (Robert?)
   - Richer controller modes: 
@@ -101,9 +125,13 @@
       - NoSynch - allow transactions to be lost on failure but maintains consistency instead of performance
 
 0.8 - Lisp Backend?
+--------------------------------------------------
   - A native BTree implementation in CommonLisp (prototype on Allegro's BTree implementation for ACache)
 
 
+========================================================
+========================================================
+
 Resolved issues:
 - On class change, new slots should have their initform values pushed 
   into the slot value as if the slot was being created the first time 
--- /project/elephant/cvsroot/elephant/elephant.asd	2006/02/22 20:18:51	1.16
+++ /project/elephant/cvsroot/elephant/elephant.asd	2006/04/26 17:53:43	1.17
@@ -94,7 +94,7 @@
 		      :serial t)
 	     (:module elephant
 		      :components
-		      ((:file "elephant")
+		      ((:file "package")
 		       (:file "config" :pathname "../../config.lisp")
 		       (:file "variables")
 		       #+cmu (:file "cmu-mop-patches")


