From ieslick at common-lisp.net Sun Apr 1 14:33:24 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 1 Apr 2007 10:33:24 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070401143324.2C70A1600F@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv29083 Modified Files: TODO elephant.asd Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/TODO 2007/03/30 14:34:34 1.76 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/01 14:33:23 1.77 @@ -18,7 +18,9 @@ - Verify db_deadlock for other lisps (launch and kill background program I/F) Bugs: +- Fix awkward serializer API - Support for asdf-install? +- Edi's patches & suggestions for windows Test coverage: - Clean up interface to tests @@ -26,14 +28,13 @@ - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization happening concurrently to make sure that multi-threading is in good shape (Henrik's code) - Unicode tests - - Ensure that variable length UTF-8 reps are automatically stored as UTF-16 + - Ensure that variable length UTF-8 reps are automatically stored as UTF-16? - Class / DB sychronization tests Documentation: ~ License and copyright file headers -- Redo tutorial -- Proper user guide - Update install, build and test procedures +- Proper user guide - Upgrade, migration and other system level issues - Performance and design issues - More notes about transaction performance @@ -42,7 +43,6 @@ - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) -- Add document section about backend interface & developer decisions 0.6.1 - Features COMPLETED to date ---------------------------------- --- /project/elephant/cvsroot/elephant/elephant.asd 2007/03/30 23:36:52 1.40 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/01 14:33:23 1.41 @@ -304,6 +304,7 @@ (:file "serializer2") ;; 0.6.1 db's (:file "unicode2") (:file "migrate") + (:file "query") (:file "backend")) :serial t :depends-on (memutil utils))))) From ieslick at common-lisp.net Sun Apr 1 14:33:34 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 1 Apr 2007 10:33:34 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070401143334.6E0B02A13E@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv29083/doc Modified Files: Makefile data-store-reference.texinfo elephant-design.texinfo elephant.texinfo installation.texinfo reference.texinfo tutorial.texinfo user-guide.texinfo Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6 @@ -9,4 +9,4 @@ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo pdf: includes-stuff - texi2dvi --pdf elphant.texinfo + texi2dvi --pdf elephant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 @@ -3,9 +3,8 @@ @node Data Store API Reference @comment node-name, next, previous, up @chapter Data Store API Reference - at cindex Data Store API Reference @cindex Data Store - at cindex API Reference + at cindex API This reference includes functions that need to be overridden, classes inherited from or other action taken to implement support for a new @@ -26,16 +25,16 @@ relevant to them. @menu -* Registration:: Register the backend to parse controller specifications. -* Store Controllers:: Subclassing the store controller. -* Handling Serialization:: Available facilities for serializing objects. -* C Utilities:: Writing primitive C types. -* Slot access:: Support for metaprotocol slot access. -* Collections:: BTrees and indices. -* Cursors:: Traversing BTrees. -* Transactions:: Transaction implementation. -* Multithreading:: Multithreading considerations. -* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries +* Registration: DSR Registration. Register the backend to parse controller specifications. +* Store Controllers: DSR Store Controllers. Subclassing the store controller. +* Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. +* Memory Utilities: DSR Memory Utilities. Writing primitive C types. +* Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access. +* Collections: DSR Collections. BTrees and indices. +* Cursors: DSR Cursors. Traversing BTrees. +* Transactions: DSR Transactions. Transaction implementation. +* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. +* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu @node DSR Registration @@ -96,17 +95,28 @@ @section Slot Access @cindex Persistent Objects and Slot Access +Persistence is implement with a metaclass and several required base +classes. + + at include includes/class-elephant-persistent-metaclass.texinfo @include includes/class-elephant-persistent.texinfo + at include includes/class-elephant-persistent-object.texinfo + +Persistent objects can be queries for their home store controller so +that functions such as map-btree do not need a store-controller +argument. (NOTE: Should this function be user visible?) + @include includes/fun-elephant-backend-get-con.texinfo - at c @include includes/fun-elephant-backend-oid.texinfo -All objects require a unique id. During new object creation the -backend is asked to produce a unique id. +All objects require a unique object identifier. During new object +creation the backend is asked to produce a unique id. @include includes/fun-elephant-backend-next-oid.texinfo -These functions are called by the metaclass protocol to support -operations on persistent class slots. +These functions are called by the metaclass protocol to implement the +appropriate operations on persistent class slots. Unless protected by +a transaction, the side effects of these functions should be atomic, +persistent and visible to other threads on completion. @include includes/fun-elephant-backend-persistent-slot-writer.texinfo @include includes/fun-elephant-backend-persistent-slot-reader.texinfo @@ -121,7 +131,9 @@ To support collections, the data store must subclass the following classes. - at include includes/class-elephant-btree.texinfo.texinfo + at include includes/class-elephant-persistent-collection.texinfo + + at include includes/class-elephant-btree.texinfo @include includes/class-elephant-btree-index.texinfo @include includes/class-elephant-indexed-btree.texinfo @@ -131,42 +143,48 @@ @include includes/fun-elephant-build-btree.texinfo @include includes/fun-elephant-build-indexed-btree.texinfo -And every btree needs accessors, these must be implemented for btree, -indexed-btree and btree-index. +Most of the user-visible operations over BTrees must be implemented. +Class indexing functions such as @code{map-class} and + at code{get-instances-by-value} and related functions are all +implemented using map-btree and map-index. + + at itemize + at item @ref{Generic-Function elephant:get-value} + at item @ref{Generic-Function (setf elephant:get-value)} + at item @ref{Generic-Function elephant:existsp} + at item @ref{Generic-Function elephant:remove-kv} + at item @ref{Generic-Function elephant:get-index} + at item @ref{Generic-Function elephant:remove-index} + at item @ref{Generic-Function elephant:map-btree} + at item @ref{Generic-Function elephant:map-index} + at end itemize - at include includes/fun-elephant-get-value.texinfo - at include includes/fun-elephant-setf-get-value.texinfo - at include includes/fun-elephant-existsp.texinfo - at include includes/fun-elephant-remove-kv.texinfo +Mapping over the indices of a btree is important to derived facilities +such as class indexing and the query subsystem. @include includes/fun-elephant-map-indices.texinfo - at include includes/fun-elephant-get-index.texinfo - at include includes/fun-elephant-remove-index.texinfo - -Critical to indexing and queries are the map operators for collections - - at include includes/fun-elephant-map-btree.texinfo - at include includes/fun-elephant-map-index.texinfo @node DSR Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors - at include includes/class-cursor.texinfo - at c #:cursor-btree - at c #:cursor-oid - at c #:cursor-initialized-p +Data stores must subclass these cursor classes and implement all the +methods described in @ref{Cursors} except @ref{Macro +elephant:with-btree-cursor}. + + at include includes/class-elephant-cursor.texinfo + at include includes/class-elephant-secondary-cursor.texinfo @node DSR Transactions @comment node-name, next, previous, up @section Transactions @cindex Transactions -These functions must be implemented or stubbed in any -backend. +These functions must be implemented or stubbed by all data stores. @include includes/fun-elephant-backend-execute-transaction.texinfo + @include includes/fun-elephant-backend-controller-start-transaction.texinfo @include includes/fun-elephant-backend-controller-commit-transaction.texinfo @include includes/fun-elephant-backend-controller-abort-transaction.texinfo @@ -179,33 +197,77 @@ @include includes/fun-elephant-backend-transaction-store.texinfo @include includes/fun-elephant-backend-transaction-object.texinfo +;; Designer considerations: +;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction +;; in the parent keyword argument. Backends allowing nested transactions can treat the transaction +;; as a parent, otherwise they can reuse the current transaction by ignoring it (inheriting the dynamic +;; value of *current-transaction*) or rebinding the dynamic context (whatever makes coding easier). +;; - ensure-transaction uses *current-transaction* to determine if there is a current transaction +;; in progress (not null). If so, it jumps to the body directly. Otherwise it executes the body in a +;; new transaction by calling ... +;; - execute-transaction contract: +;; - Backends must dynamically bind *current-transaction* to a meaningful identifier for the +;; transaction in progress and execute the provided closure in that context +;; - All non-local exists result in an abort; only regular return values result in a commit +;; - If a transaction is aborted due to a deadlock or read conflict, execute-transaction should +;; automatically retry with an appropriate default amount +;; - execute-transaction can take any number of backend-defined keywords, although designers should +;; make sure there are no semantic conflicts if there is a name overlap with existing backends +;; - A typical design approach is to make sure that the most primitive interfaces to the backend +;; database look at *current-transaction* to determine whether a transaction is active. Users code can also +;; access this parameter to check whether a transaction is active. + @node DSR Multithreading Considerations @comment node-name, next, previous, up @section Multithreading Considerations @cindex Multithreading - at c utils locks - at c utils thread-vars +Generic locking utility functions + +Variable behavior in multithreading situations @node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization @cindex Serialization - at c #:deserialize #:serialize - at c #:serialize-symbol-complete - at c #:deserialize-from-base64-string - at c #:serialize-to-base64-string +Backends must initialize @ref{Class elephant:store-controller} with +internal serializer functions. Packages @code{elephant-serializer1} +and @code{elephant-serializer2} contains serialize and deserialize +methods on buffer-streams as defined in @code{elephant-memutil}. The +elephant functions @code{serialize} and @code{deserialize} dispatch on +the appropriate slot values of the store-controller. + + at verbatim +NOTE: This should perhaps become entirely the job of the data store to +decide how to serialize values and for a specific version, what +serializer to use. The elphant main package can define serializers +for use by different backends. + at end verbatim + + at include includes/fun-elephant-backend-serialize.texinfo + at include includes/fun-elephant-backend-deserialize.texinfo + +These utility functions are useful if a data store does not have the +ability to store variable length binary data. They are based on the + at code{cl-base64} library. + + at include includes/fun-elephant-backend-serialize-to-base64-string.texinfo + at include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo - at node DSR Memory utilities + at node DSR Memory Utilities @comment node-name, next, previous, up @section Memory utilities @cindex Memory utilities - at node DSR Foreign libraries +Details about memory utilities here. + + at node DSR Foreign Libraries @comment node-name, next, previous, up @section Foreign libraries @cindex Foreign libraries +How foreign libraries are built and used via UFFI. What functions are +in the .asd files or main lisp code to build & load libraries? --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3 @@ -1,9 +1,37 @@ + at c -*-texinfo-*- @node Elephant Design @comment node-name, next, previous, up - at section Elephant Design + at chapter Elephant Design @cindex design +Elephant's early architecture was tightly coupled to the Berkeley DB +API. Over time we've moved towards a more modular architecture to +support easy upgrading, repository migration, shared functionality +between data stores and general hygene. + +The current architecture is modularized into the following pieces: + + at verbatim +[Picture describing] Metaclass, Store controller, persistent objects, +data stores, serializer & memutils. Derived features such as class +indexing, migration, query system and root operations can also be +illustrated? + at end verbatim + +To get a feeling for what is happening inside elephant, it is probably +most useful to walk through the various major protocols: + + at itemize + at item Initialization of a store controller + at item Creating a persistent object + at item Operations on persistent slots + at item Operations on persistent collections + at item Implementing @code{with-transaction} + at end itemize + + at section Initializing a store controller + When the main elephant @code{open-store} function is called, it calls @code{get-controller} which grabs an existing store controller if the spec is identical, or builds a new controller. Building the @@ -13,4 +41,14 @@ that returns a @code{store-controller} subclass instance specific to that backend. -Elephant than calls open-controller +Elephant than calls open-controller to actually establish a connection +to or create the files of the data store. + + at section Persistent Object Creation + at section Persistent Slot Protocol + at section Persistent Slot Protocol + at section Persistent Collection Protocol + at section Implementing Transactions + + + --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7 @@ -38,6 +38,7 @@ @insertcopying @end ifnottex + at ifhtml @menu * Table of Contents:: @end menu @@ -49,7 +50,7 @@ * Tutorial:: A basic ``getting started'' tutorial. * Installation:: Installation and test-suite procedures. * User Guide:: In depth discussion of all Elephant facilities and features. -* Usage scenarios:: Design scenarios for Elephant applications. +* Usage Scenarios:: Design scenarios for Elephant applications. * User API Reference:: Function and class documentation of the user API. * Elephant Design:: An overview of elephant's internal architecture. * Data Store API Reference:: Function level documentation for data store implementors. @@ -66,6 +67,8 @@ * Colophon:: @end menu + at end ifhtml + @node Table of Contents @unnumbered @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 @@ -10,9 +10,9 @@ * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. * Berkeley DB Data Store:: Installing support for the Berkeley DB data store -* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store. +* Berkeley DB Example:: An example of installing and running the Berkeley DB data store. * CL-SQL Data Store:: Install and connecting to the CL-SQL data store -* CL-SQL Examples:: An example of using the CL-SQL data store. +* CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite * Documentation:: Building documentation from texinfo sources. @@ -277,10 +277,6 @@ As of Elephant 0.3, Elephant has been tested to work with both Postgres, and SQLite 3, thanks do Dan Knapp. - at node Extension Status - at comment node-name, next, previous, up - at section Extension Status - As far as is known at this writing, all functionality except nested transaction support and cursor-puts supported by the BerkeleyDB backend is supported by the CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested @@ -302,13 +298,9 @@ the multi-repository version somewhat complicates the underlying persistent object management. - at node PostGres Examples - at comment node-name, next, previous, up - at section Setting up PostGres - - at node Setting up PostGres + at node CL-SQL Example @comment node-name, next, previous, up - at section Setting up PostGres + at section CL-SQL Example To set up a PostGres based back end, you should: @@ -351,6 +343,10 @@ sequences, and so on needed by the Elephant system will be created in the schema named ``test'' automatically. + at node Elephant on Windows + at comment node-name, next, previous, up + at section Elephant on Windows + @node Test Suites @comment node-name, next, previous, up @section Test Suites --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9 @@ -22,14 +22,20 @@ @section Store Controllers @cindex Store Controllers -Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller. +Store controllers provide the persistent storage for CLOS objects and +BTree collections. Any persistent operations must be done in the +context of a store controller. The default store-controller is stored +in a global variable. - at include includes/class-elephant-store-controller.texinfo @include includes/var-elephant-star-store-controller-star.texinfo + at c @include includes/class-elephant-store-controller.texinfo + at ref{Class elephant:store-controller} is associated with the following +user methods and macros: + + at include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-open-store.texinfo @include includes/fun-elephant-close-store.texinfo - at include includes/macro-elephant-with-open-store.texinfo @include includes/fun-elephant-get-from-root.texinfo @include includes/fun-elephant-add-to-root.texinfo @@ -42,12 +48,12 @@ @section Persistent Objects @cindex Persistent Objects - at include includes/class-elephant-persistent-metaclass.texinfo - at include includes/class-elephant-persistent.texinfo - at include includes/class-elephant-persistent-object.texinfo + at ref{Class elephant:persistent-metaclass} can be used as the +:metaclass argument in a defclass form to create a persistent object. +Slots of the metaclass take the :index and :transient keyword +arguments and the class accepts the :index keyword argument. @include includes/macro-elephant-defpclass.texinfo - @include includes/fun-elephant-drop-pobject.texinfo @node Persistent Object Indexing @@ -94,22 +100,26 @@ interface is provided for reference only, a new system is under development for the 0.7 release. - at include includes/fun-elephant-get-query-results.texinfo - at include includes/fun-elephant-map-class-query.texinfo + at c @include includes/fun-elephant-get-query-results.texinfo + at c @include includes/fun-elephant-map-class-query.texinfo @node Collections @comment node-name, next, previous, up @section Collections @cindex Collections - at include includes/class-elephant-persistent-collection.texinfo - at include includes/class-elephant-btree.texinfo - at include includes/class-elephant-indexed-btree.texinfo - at include includes/class-elephant-btree-index.texinfo +Persistent collections inherit from @ref{Class +elephant:persistent-collection} and consist of the @ref{Class +elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class +elephant:btree-index} classes. The following operations are defined +on most of these classes. More information can be found in @ref{Using +BTrees} and @ref{Secondary Indices}. @include includes/fun-elephant-get-value.texinfo @include includes/fun-elephant-setf-get-value.texinfo @include includes/fun-elephant-remove-kv.texinfo + at include includes/fun-elephant-existsp.texinfo + @include includes/fun-elephant-map-btree.texinfo @include includes/fun-elephant-map-index.texinfo @@ -118,17 +128,17 @@ @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo - @node Cursors @comment node-name, next, previous, up @section Cursors @cindex Cursors - at include includes/class-elephant-cursor.texinfo - at include includes/class-elephant-secondary-cursor.texinfo +Cursors are objects of type cursor (@pxref{Class elephant:cursor}) +which provide methods for complex traversals of BTrees. + + at include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo - at include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-cursor-current.texinfo @include includes/fun-elephant-cursor-delete.texinfo --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 @@ -15,6 +15,7 @@ * Persistent collections:: Keep track of collections of values. * Indexing Persistent Classes:: Simple way to keep track of persistent instances. * Using Transactions:: Providing ACID database properties. +* Advanced Topics:: Additional Elephant features covered in the User Guide. @end menu @node Overview @@ -704,14 +705,14 @@ @end lisp The @ref{User Guide} contains a descriptions of the advanced features -of @ref{Class indices} such as ``derived indicies'' that allow you to +of @ref{Class Indices} such as ``derived indicies'' that allow you to order classes according to an arbitrary function, a dynamic API for adding and removing slots and how to set a policy for resolving conflicts between the code image and a database where the indexing specification differs. This same facility is also available for your own use. For more -information @pxref{Using Indexed BTrees}. +information @pxref{Secondary Indices}. @node Using Transactions @@ -914,8 +915,71 @@ complicated. The best strategy at the beginning is a conservative one, break things up into the smallest logical sets of primitive operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction details} -for the full details and @pxref{Usage scenarios} for more examples of +they absolutely have to commit together. See @ref{Transaction Details} +for the full details and @pxref{Usage Scenarios} for more examples of how systems can be designed and tuned using transactions. + at node Advanced Topics + at comment node-name, next, previous, up + at section Advanced Topics + +The tutorial covers the essential topics and concepts for using +Elephant. Many people will find that these features are the ones that +are most often needed and used in ordinary applications. + +More sophisticated uses of Elephant may require additional features +that are covered in the user guide. The following is a list of major +features in the user guide that were not covered in this tutorial. + + at itemize + at item @strong{Class Heirarchies and Queries} + There are some subtle issues to take into account when querying +persistent classes. For example, how do you query a base class of +type people to get subclass instances such as employee, manager, +consultant, etc? + at item @strong{Derived Class Indices} + You can create your own indices for classes that are arbitrary +lisp functions of the persistent object. + at item @strong{Class Definition/Database Conflict Resolution} + When you startup lisp, there are potential conflicts between the +class definition and the indexing records in the database. There are +some constraints to account for and some facilities to manage +how slots, class indices and + at item @strong{Dynamic Class Index Management} + It is possible to add and remove indexes from classes at runtime. + at item @strong{BTree Cursors} + If you need to do more than iterate over a collection, or you need +to delete elements of the collection as you iterate cursors are an +important data structure. They implement a variety of operators for +moving backward and forward over a btree, including ranged operations +and iterating of duplicate or unique values. + at item @strong{Indexed BTrees} + Indexed BTrees are just like BTrees, except it is possible to add +indexes which are BTrees who's values are primary keys in the parent + at code{indexed-btree}. This allows for multiple ordering and groupings +of the values of a BTree. + at item @strong{Using the Map Operators} + Mapping operators can be very efficient if properly utilized. + at item @strong{Handling Errors and Conditions} + There are a variety of errors that can occur in Elephant that need +to be dealt with by applications. + at item @strong{Deadlock Detection in Berkeley DB} + Berkeley DB requires an external process to detect deadlock +conditions among transactions. The :deadlock-detect keyword argument +to open-store for Berkeley DB specs will launch this process on most +lisps. + at item @strong{Using Multiple Stores} + Multiple store controllers can be open simultaneously. However it +does make the code more complex and you need to be careful about how +you use them to avoid crashes and other unpleasant side effects. + at item @strong{Custom Transaction Architecture} + You can implement your own version of with-transaction using the +underlying controller methods for starting, aborting and committing +transactions. You had better know what you are doing, however! + at item @strong{Using Multiple Threads and Processes} + What constraints must be accommodated to use Elephant data stores in +multiple threads? What capabilities are there to share data stores +among multiple processes or machines? + at end itemize +Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 @@ -8,18 +8,19 @@ @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. -* Reachability:: Determining liveness in a store. * Persistent objects:: All the dirt on persistent objects. -* Class indices:: In-depth discussion about indexing persistent indices. +* Class Indices:: In-depth discussion about indexing persistent indices. * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. -* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls. +* Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. +* Multi-threaded Applications:: What considerations are required for safe multi-threading +* Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines? * Repository Migration and Upgrade:: How to move objects from one repository to another. -* Garbage collection:: How to recover storage and OIDs in long-lived repositories. -* Performance tuning:: How to get the most from Elephant. +* Garbage Collection:: How to recover storage and OIDs in long-lived repositories. +* Performance Tuning:: How to get the most from Elephant. @end menu @node Persistent objects @@ -83,9 +84,15 @@ @code{with-open-controller} macro. Opening and closing a controller is very expensive. + at node Serialization details + at comment node-name, next, previous, up + at section Serialization details + +Empty. + @node Class Indices @comment node-name, next, previous, up - at section Class Indicies + at section Class Indices You can enable/disable class indexing for an entire class. When you disable indexing all references to instances of that class are lost. If you re-enable @@ -130,6 +137,10 @@ somewhat user customizable; documentation for this exists in the source file referenced above. + at node Querying persistent instances + at comment node-name, next, previous, up + at section Querying persistent instances + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -147,10 +158,127 @@ blocks of data is relatively inexpensive after a seek and comparisons on objects that are stored in memory is cheap. + at node Secondary Indices + at comment node-name, next, previous, up + at section Secondary Indices + +Empty. + + at node Using Cursors + at comment node-name, next, previous, up + at section Using Cursors + +Empty. - at node Repository Migration + at node Transaction Details @comment node-name, next, previous, up - at section Repository Migration + at section Transaction Details + +;; Transaction architecture: +;; +;; User and designer considerations: +;; - *current-transaction* is reserved for use by dynamic transaction context. The default global +;; value must always be null (no transaction). Each backend can set it to a different parameter +;; within the dynamic context of an execute-transaction. +;; - Any closures returned from within a transaction cannot bind *current-transaction* +;; - Only a normal return value will result in the transaction being committed, any non-local exit +;; results in a transaction abort. If you want to do something more sophisticated, roll your own +;; using controller-start-transaction, etc. +;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc) +;; knowing that the transaction will be aborted +;; + + at node Multi-threaded Applications + at comment node-name, next, previous, up + at section Multi-threaded Applications + +Sleepycat plays well with threads and processes. The store controller +is thread-safe by default, that is, can be shared amongst threads. +This is set by the @code{:thread} key. Transactions may not be shared +amongst threads except serially. One thing which is NOT thread and +process safe is recovery, which should be run when no one is else is +talking to the database environment. Consult the Sleepycat docs for +more information. + +Elephant uses some specials to hold parameters and buffers. If you're +using a natively threaded lisp, you can initialize these specials to +thread-local storage by using the @code{run-elephant-thread} function, +assuming your lisp creates thread-local storage for let-bound +specials. (This functionality is currently broken) + +Persisting ordinary aggregate types (e.g. NOT persistent classes or +btrees) suffers from something called "merge-conflicts." Since +updating one value of an aggregate object requires the entire object +to be written to the database, in heavily threaded situations you may +overwrite changes another thread or process has committed. This is +not protected by transactions! + +Consider two processes operating on the same cons: + + at code{ +-----start---read--update-car--write--commit---------------- +-start------read--update-cdr-----------------write--commit-- +} + +Although the first process successfully committed its transaction, its +work (writing to the car) will be erased by the second transaction +(which writes both the car and cdr.) + +Persistent classes and persistent collections do not suffer from +merge-conflicts, since each slot / entry is a separate database entry. + + at node Multi-repository Operation + at comment node-name, next, previous, up + at section Multi-repository Operation + +Elephant now keeps a small hashtables that maps ``database specifications'' into +actual database connections. + +If a database spec is a string, it is assumed to be a BerkeleyDB path. +If it is a list, it is a assumed to be a CL-SQL connection specification. +For example: + at lisp +ELE-TESTS> *testdb-path* +"/home/read/projects/elephant/elephant/tests/testdb/" +ELE-TESTS> *testpg-path* +(:postgresql "localhost.localdomain" "test" "postgres" "") +ELE-TESTS> + at end lisp + +The tests now have a function @code{do-all-tests-spec} that take a spec and +based on its type attempt to open the correct kind of store controller and +perform the tests. + +The routine @code{get-controller} takes this specifiation. + +The basic strategy is that the ``database specification'' object is stored in +every persistent object and collection so that the repository can be found. + +In this way, objects that reside in different repositories can coexist within +the LISP object space, allowing data migration. + +;; Multiple stores + +;; Multiple store considerations: +;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to +;; work around: how to avoid writing one store with a transaction created in the context of another. +;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct; +;; this requirement may relax in the future +;; - The following macros accomodate multiple stores by requiring that execute-transaction return a +;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller +;; is the store instance it is associated with. A nested or ensured transaction is only indicated +;; in the call to execute transaction if the store controllers match, otherwise a new transaction +;; for that store is created + + at node Multiple Processes and Distributed Applications + at comment node-name, next, previous, up + at section Multiple Processes and Distributed Applications + +Can we do this? What do we have to say about this? + + at node Repository Migration and Upgrade + at comment node-name, next, previous, up + at section Repository Migration and Upgrade This version of Elephant supports migration between store controllers of any backend type. @@ -227,49 +355,17 @@ @code{*inhibit-slot-writes*} in your user method using @code{with-inhibited-slot-copy} a convenience macro. - - at node Threading + at node Garbage Collection @comment node-name, next, previous, up - at section Threading - -Sleepycat plays well with threads and processes. The store controller -is thread-safe by default, that is, can be shared amongst threads. -This is set by the @code{:thread} key. Transactions may not be shared -amongst threads except serially. One thing which is NOT thread and -process safe is recovery, which should be run when no one is else is -talking to the database environment. Consult the Sleepycat docs for -more information. - -Elephant uses some specials to hold parameters and buffers. If you're -using a natively threaded lisp, you can initialize these specials to -thread-local storage by using the @code{run-elephant-thread} function, -assuming your lisp creates thread-local storage for let-bound -specials. (This functionality is currently broken) - -Persisting ordinary aggregate types (e.g. NOT persistent classes or -btrees) suffers from something called "merge-conflicts." Since -updating one value of an aggregate object requires the entire object -to be written to the database, in heavily threaded situations you may -overwrite changes another thread or process has committed. This is -not protected by transactions! - -Consider two processes operating on the same cons: - - at code{ ------start---read--update-car--write--commit---------------- --start------read--update-cdr-----------------write--commit-- -} - -Although the first process successfully committed its transaction, its -work (writing to the car) will be erased by the second transaction -(which writes both the car and cdr.) + at section Garbage Collection -Persistent classes and persistent collections do not suffer from -merge-conflicts, since each slot / entry is a separate database entry. +GC is not implemented, but migration (@pxref{Repository Migration and +Upgrade}) will consolidate storage and recover OIDs which emulates GC. +No online solution is currently supported. - at node Performance Tips + at node Performance Tuning @comment node-name, next, previous, up - at section Performance Tips + at section Performance Performance is usually measured in transactions per second. Database reads are cheap. To get more transactions throughput, consider @@ -290,8 +386,8 @@ things. It is fast but consing with floats and doubles. YMMV with other values, though I've tried to make them fast. -Using @code{*auto-commit*} and not @code{with-transactions} is a great -way to have a huge number of transactions. You'll find that +Use @code{with-transactions} to avoid many automatic transactions, for +example you'll find that this construct @lisp (dotimes (i 1000) (add-to-root "key" "value")) @@ -300,9 +396,8 @@ is much slower than @lisp -(let ((*auto-commit* nil)) - (with-transaction () - (dotimes (i 1000) (add-to-root "key" "value")))) +(with-transaction () + (dotimes (i 1000) (add-to-root "key" "value")))) @end lisp since there's only 1 transaction in the latter. However storing @@ -319,32 +414,3 @@ mode is not suitable for use in web servers or other typically multi-threaded applications. - at node Multi-repository Operation - at comment node-name, next, previous, up - at section Multi-repository Operation - -Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. - -If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CL-SQL connection specification. -For example: - at lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> - at end lisp - -The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration. From ieslick at common-lisp.net Sun Apr 1 14:33:35 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 1 Apr 2007 10:33:35 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070401143335.9E6393201A@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv29083/src/elephant Modified Files: classes.lisp controller.lisp metaclasses.lisp serializer.lisp Log Message: Latest documentation changes --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/03/30 23:36:53 1.25 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/01 14:33:34 1.26 @@ -39,9 +39,11 @@ (defclass persistent-object (persistent) () (:metaclass persistent-metaclass) (:documentation - "Superclass of all user-defined persistent classes. This is + "Superclass for all user-defined persistent classes. This is automatically inherited if you use the persistent-metaclass - metaclass.")) + metaclass. This allows specialization of functions for user + objects that would not be appropriate for Elephant objects + such as persistent collections")) ;; ================================================ ;; METACLASS INITIALIZATION AND CHANGES --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/03/30 23:42:35 1.44 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/01 14:33:34 1.45 @@ -48,11 +48,13 @@ (defvar *dbconnection-spec* (make-hash-table :test 'equal)) (defvar *dbconnection-lock* (ele-make-lock)) -(defmethod get-con ((instance persistent) &optional (sc *store-controller*)) - "This is used to find and validate the connection spec +(defgeneric get-con (instance &optional sc) + (:documentation "This is used to find and validate the connection spec maintained for in-memory persistent objects. Should we re-open the controller from the spec if it's not - cached? That might be dangerous so for now we error" + cached? That might be dangerous so for now we error")) + +(defmethod get-con ((instance persistent) &optional (sc *store-controller*)) (declare (ignore sc)) (let ((con (gethash (dbcn-spc-pst instance) *dbconnection-spec*))) (cond ((not con) --- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/03/30 23:36:53 1.14 +++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/01 14:33:34 1.15 @@ -29,7 +29,7 @@ :documentation "Persistent objects use a spec pointer to identify which store they are connected to")) (:documentation "Abstract superclass for all persistent classes (common - to user-defined classes and collections.)")) + to both user-defined classes and Elephant-defined objects such as collections.)")) (defclass persistent-metaclass (standard-class) ((%persistent-slots :accessor %persistent-slots) --- /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/03/30 14:34:35 1.26 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/04/01 14:33:34 1.27 @@ -47,6 +47,8 @@ ;; (defun serialize-to-base64-string (x sc) + "Encode object using the store controller's serializer format, + but encoded in a base64" (with-buffer-streams (out-buf) (cl-base64::usb8-array-to-base64-string (elephant-memutil::buffer-read-byte-vector @@ -62,6 +64,7 @@ ) (defun deserialize-from-base64-string (x sc) + "Decode a base64-string using the store controller's deserialize method" (with-buffer-streams (other) (deserialize (elephant-memutil::buffer-write-byte-vector From ieslick at common-lisp.net Sun Apr 1 20:22:24 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 1 Apr 2007 16:22:24 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070401202224.CD6904717F@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv22287 Modified Files: tutorial.texinfo user-guide.texinfo Log Message: Documentation changes, mostly to transaction section of tutorial --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12 @@ -732,6 +732,8 @@ transaction that performs all the updates atomically and thus enforcing consistency. + at subsection Why do we need Transactions? + Most real applications will need to use explicit transactions rather than relying on the primitives alone because you will want multiple read-modify-update operations act as an atomic unit. A good example @@ -815,6 +817,8 @@ And presto, we have an ACID compliant, thread-safe, persistent banking system! + at subsection Using @code{with-transaction} + What is @code{with-transaction} really doing for us? It first starts a new transaction, attempts to execute the body, and if successful commit the transaction. If anywhere along the way there is a deadlock @@ -823,14 +827,145 @@ to retry the transaction a fixed number of times by re-executing the whole body. -The other value transactions provide is the capability to delay -flushing dirty data to disk. The most time-intensive part of -persistent operations is flushing newly written data to disk. Using -the default auto-commit behavior requires a flush on every operation -which can become very expensive. Because a transaction caches values, -all the values read or written are cached in memory until the -transaction completes, dramatically decreasing the number of flushes -and the total time taken. +And this brings us to two important caveats: nested transactions and +idempotent side-effects. + + at subsection Nesting Transactions + +In general, you want to avoid nesting @code{with-transaction} +statements. Nested transactions are valid for some data stores +(namely Berkeley DB), but typically only a single transaction can be +active at a time. The purpose of a nested transaction in data stores +that provide it, is break a long transaction into chunks. This way if +there is contention on a given subset of variables, only the inner +transaction is restarted while the larger transaction can continue. +When commit their results, those results become part of the outer +transaction until it in turn commits. + +If you have transaction protected primitive operations (such as + at code{deposit} and @code{withdraw}) and you want to perform a group of +such transactions, for example a transfer between accounts, you can +use the macro @code{ensure-transaction} instead of @code{with-transaction}. + + at lisp +(defun deposit (account amount) + "Wrap the balance read and the setf with the new balance" + (ensure-transaction () + (let ((balance (balance account))) + (setf (balance account) + (+ balance amount))))) + +(defun deposit (account amount) + "A more concise version with decf doing both read and write" + (ensure-transaction () + (decf (balance account) amount))) + +(defun withdraw (account amount) + (ensure-transaction () + (decf (balance account) amount))) + +(defun transfer (src dst amount) + "There are four primitive read/write operations + grouped together in this transaction" + (with-transaction () + (withdraw src amount) + (deposit dst amount))) + at end lisp + + at code{ensure-transaction} is exactly like @code{with-transaction} +except it will reuse an existing transaction, if there is one, or +create a new one. There is no harm, in fact, in using this macro all +the time. + +Notice the use of @code{decf} and @code{incf} above. The primary +reason to use Lisp is that it is good at hiding complexity using +shorthand constructs just like this. This also means it is also going +to be good at hiding data dependencies that should be captured in a +transaction! + + at subsection Idempotent Side Effects + +Within the body of a with-transaction, any non database operations +need to be @emph{idempotent}. That is the side effects of the body +must be the same no matter how many times the body is executed. This +is done automatically for side effects on the database, but not for +side effects like pushing a value on a lisp list, or creating a new +standard object. + + at lisp +(defparameter *transient-objects* nil) + +(defun load-transients (n) + "This is the wrong way!" + (with-transaction () + (loop for i from 0 upto n do + (push (get-from-root i) *transient-objects*)))) + at end lisp + +In this contrived example we are pulling a set of standard objects +from the database using an integer key and pushing them onto a list +for later use. However, if there is a conflict where some other +process writes a key-value pair to a matching key, the whole +transaction will abort and the loop will be run again. In a heavily +contended system you might see results like the following. + + at lisp +(defun test-list () + (setf *transient-objects* nil) + (load-transients) + (length *transient-objects*)) + +(test-list) +=> 3 + +(test-list) +=> 5 + +(test-list) +=> 4 + at end lisp + +So the solution is to make sure that the operation on the lisp +parameters is atomic if the transaction completes. + + at lisp +(defun load-transients () + "This is a better way" + (setq *transient-objects* + (with-transaction () + (loop for i from 0 upto 3 collect + (get-from-root i))))) + at end lisp + +Of course we would need to use @code{nreverse} if we cared about the +order of instances in @code{*transient-objects*}. The best rule of +thumb is that transaction bodies should be purely functional as above, +except for side effects to the persistent store such as persistent +slot writes, adding to btrees, etc). + +If you do need side effects to lisp memory, such as writes to +transient slots, make sure they are idempotent and that other +processes will not be reading the written values until the transaction +completes. + + at subsection Transactions and Performance + +By now transactions almost look like more work than they are worth! +Well there are still some significant benefits to be had. Part of how +transactions are implemented is that they gather together all the +writes that are supposed to made to the database and store them until +the transaction commits, and then writes them atomically. + +The most time-intensive part of persistent operations is flushing +newly written data to disk. Using the default auto-committing +behavior requires a flush for every primitive write operation. This +can become very expensive! Because all the values read or written are +cached in memory until the transaction completes, the number of +flushes can be dramatically reduced. + +But don't take my word for it, run the following statements and see +for yourself the visceral impact transactions can have on system +performance. @lisp (defpclass test () @@ -872,52 +1007,42 @@ thumb is to keep the number of objects touched in a transaction well under 1000. -And this brings us to the last caveat we'll introduce in this -introductory tutorial: nested transactions. - -In general, avoid nesting transactions. Nested transactions are valid -for some data stores (namely Berkeley DB), but typically only a single -transaction is valid at a time. The purpose of a nested transaction -is to allow a long transaction to be broken up into chunks. This way -if there is contention on a given subset of variables, only the -subtransaction is restarted while the larger transaction can continue. -Subtransactions commit their results and they become part of the -outer transaction until it in turn commits. - -If you have transaction protected primitive operations (such as - at code{deposit} and @code{withdraw}) and you want to perform a group of -such transactions, for example a transfer between accounts, you can -use the macro @code{ensure-transaction} instead of @code{with-transaction}. - - at lisp -(defun deposit (account amount) - (ensure-transaction () - (let ((balance (balance account))) - (setf (balance account) - (+ balance amount))))) - -(defun withdraw (account amount) - (ensure-transaction () - (decf (balance account) amount))) - -(defun transfer (src dst amount) - (with-transaction () - (withdraw src amount) - (deposit dst amount))) - at end lisp - - at code{ensure-transaction} is exactly like @code{with-transaction} -except it will reuse an existing transaction, if there is one, or -create a new one. There is no harm, in fact, in using this macro all -the time. + at subsection Transactions and Applications Designing and tuning a transactional architecture can become quite -complicated. The best strategy at the beginning is a conservative -one, break things up into the smallest logical sets of primitive -operations and only wrap higher level functions in transactions when -they absolutely have to commit together. See @ref{Transaction Details} -for the full details and @pxref{Usage Scenarios} for more examples of -how systems can be designed and tuned using transactions. +complex. Moreover, bugs in your system can be very difficult to find +as they only show up when transactions are interleaved within a +larger, multi-threaded application. + +In many cases, however, you can ignore transactions. For example, +when you don't have any other concurrent processes running. In this +case all operations are sequential and there is no chance of +conflicts. You would only want to use transactions for write +performance. + +You can also ignore transactions if your application can guarantee +that concurrency won't generate any conflicts. For example, a web app +that guarantees only one thread will write to objects in a particular +session can avoid transactions altogether. However, it is good to be +careful about making these assumptions. In the above example, a +reporting function that iterates over sessions, users or other objects +may still see partial updates (i.e. a user's id was written prior to +the query, but not the name). However, if you don't care about these +infrequent glitches, this case would still hold. + +If these cases don't apply to your application, or you aren't sure, +you will fare best by programming defensively. Break your system into +the smallest logical sets of primitive operations +(i.e. @code{withdraw} and @code{deposit}) using + at code{ensure-transaction} and then wrap the highest level calls made +to your system in with-transaction when the operations absolutely have +to commit together or you need the extra performance. Try not to have +more than two levels of transactional accesses with the top using +with-transaction and the bottom using ensure-transaction. + + at xref{Transaction Details} for more details and @pxref{Usage +Scenarios} for examples of how systems can be designed and tuned using +transactions. @node Advanced Topics @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6 @@ -23,26 +23,6 @@ * Performance Tuning:: How to get the most from Elephant. @end menu - at node Persistent objects - at comment node-name, next, previous, up - at section Persistent Objects - -Finally, if you for some reason make an instance with a specified OID -which already exists in the database, @code{initargs} take precedence -over values in the database, which take precedences over - at code{initforms}. - -Also currently there is a bug where - at code{initforms} are always evaluated, so beware. -(What is the current model here?) - -Readers, writers, accessors, and @code{slot-value-using-class} are -employed in redirecting slot accesses to the database, so override -these with care. Because @code{slot-value, slot-boundp, -slot-makunbound} are not generic functions, they are not guaranteed by -the specification to work properly with persistent slots. However the -proper behavior has been verified on SBCL, Allegro and Lispworks. - @node The Store Controller @comment node-name, next, previous, up @section The Store Controller @@ -90,6 +70,26 @@ Empty. + at node Persistent objects + at comment node-name, next, previous, up + at section Persistent Objects + +Finally, if you for some reason make an instance with a specified OID +which already exists in the database, @code{initargs} take precedence +over values in the database, which take precedences over + at code{initforms}. + +Also currently there is a bug where + at code{initforms} are always evaluated, so beware. +(What is the current model here?) + +Readers, writers, accessors, and @code{slot-value-using-class} are +employed in redirecting slot accesses to the database, so override +these with care. Because @code{slot-value, slot-boundp, +slot-makunbound} are not generic functions, they are not guaranteed by +the specification to work properly with persistent slots. However the +proper behavior has been verified on SBCL, Allegro and Lispworks. + @node Class Indices @comment node-name, next, previous, up @section Class Indices @@ -141,6 +141,111 @@ @comment node-name, next, previous, up @section Querying persistent instances + + +A SQL select-like interface is in the works, but for now queries are +limited to manual mapping over class instances or doing small queries +with @code{get-instances-*} functions. One advantage of this is that +it is easy to estimate the performance costs of your queries and to +choose standard and derived indices that give you the ordering and +performance you want. + +There is, however, a quick and dirty query API example that is not +officially supported in the release but is intended to invite comment. +This is an example of a full query system that would automatically +perform joins, use the appropriate indices and perhaps even adaptively +suggest or add indices to facilitate better performance on common +queries. + +There are two functions @ref{Function elephant:get-query-instances} +and @ref{Function elephant:map-class-query} which accept a set of +constraints instead of the familiar value or range arguments. + +We'll use the classes @code{person} and @code{department} to +illustrate how to perform queries over a set of objects that may be +constrainted by their relationships to other objects. + + at lisp +(defpclass person () + ((name :initarg :name :index t) + (salary :initarg :salary :index t) + (department :initarg :dept))) + +(defmethod print-object ((p person) stream) + (format stream "#" (slot-value p 'name))) + +(defun print-name (inst) + (format t "Name: ~A~%" (slot-value inst 'name))) + +(defpclass department () + ((name :initarg :name) + (manager :initarg :manager))) + +(defmethod print-object ((d department) stream) + (format stream "#" + (slot-value d 'name) + (when (slot-boundp d 'manager) + (slot-value (slot-value d 'manager) 'name)))) + at end lisp + +Here we have a simple employee database with managers (also of type +person) and departments. This simple system will provide fodder for +some reasonably complex constraints. Let's create a few departments. + + at lisp +(setf marketing (make-instance 'department :name "Marketing")) +(setf engineering (make-instance 'department :name "Engineering")) +(setf sales (make-instance 'department :name "Sales")) + at end lisp + +And manager @code{people} for the departments. + + at lisp +(make-instance 'person :name "George" :salary 140000 :department marketing) +(setf (slot-value marketing 'manager) *) + +(make-instance 'person :name "Sally" :salary 140000 :department engineering) +(setf (slot-value engineering 'manager) *) + +(make-instance 'person :name "Freddy" :salary 180000 :department sales) +(setf (slot-value sales 'manager) *) + at end lisp + +And of course we need some folks to manage + + at lisp +(defparameter *names* + '("Jacob" "Emily" "Michael" "Joshua" "Andrew" "Olivia" "Hannah" "Christopher")) + +(defun random-element (list) + "Choose a random element from the list and return it" + (nth (random (length list)) list)) + +(with-transaction () + (loop for i from 0 upto 40 do + (make-instance 'person + :name (format nil "~A~A" (random-elephant *names*) i) + :salary (floor (+ (* (random 1000) 100) 30000)) + :department (case (random 3) + (0 marketing) + (1 engineering) + (2 sales))))) + at end lisp + +Due to the random allocation of +In the follwoing examples below, the results will be different due to the random +allocation of employee names, etc. However, these examples are +illustrative of what you should see if you run the same code. + + + +For those familiar with SQL, if an instance of @code{person} has a +pointer to an instance of @code{department} then that relation can be +used to perform a join. Of course joins in the object world won't +return a table, instead they will return conjunctions of objects that +satisfy a mutual set of constraints. + + @node Using BTrees @comment node-name, next, previous, up @section Using BTrees @@ -174,6 +279,14 @@ @comment node-name, next, previous, up @section Transaction Details +You can trace @code{elephant::execute-transaction} to see the sequence +of calls to @code{execute-transaction} that occur dynamically and +detect where transactions are and are not happening. We may add some +transaction diagnosis and tracing tools in the future, such as +throwing a condition when @code{with-transaction} forms are nested +dynamically. + + ;; Transaction architecture: ;; ;; User and designer considerations: From ieslick at common-lisp.net Sun Apr 1 20:56:19 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 1 Apr 2007 16:56:19 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070401205619.2DF4C53068@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv27196/doc Modified Files: user-guide.texinfo Log Message: Fixed package bug. Minor tweaks elsewhere --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:22:24 1.6 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7 @@ -141,8 +141,6 @@ @comment node-name, next, previous, up @section Querying persistent instances - - A SQL select-like interface is in the works, but for now queries are limited to manual mapping over class instances or doing small queries with @code{get-instances-*} functions. One advantage of this is that From ieslick at common-lisp.net Sun Apr 1 20:56:19 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 1 Apr 2007 16:56:19 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070401205619.A952F56008@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv27196/src/elephant Modified Files: backend.lisp package.lisp query-example.lisp query.lisp Log Message: Fixed package bug. Minor tweaks elsewhere --- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/03/30 23:36:53 1.15 +++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/04/01 20:56:19 1.16 @@ -56,6 +56,10 @@ #:controller-deserialize #:root #:spec #:class-root + ;; Collections + #:build-btree + #:build-indexed-btree + ;; Serializer tools/api's #:serialize #:deserialize #:deserialize-from-base64-string --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/03/30 14:55:54 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/01 20:56:19 1.28 @@ -206,9 +206,9 @@ #:persistent #:persistent-object #:persistent-metaclass #:defpclass #:persistent-collection #:drop-pobject - #:btree #:build-btree + #:btree #:make-btree #:get-value #:remove-kv #:existsp - #:indexed-btree #:build-indexed-btree + #:indexed-btree #:make-indexed-btree #:btree-index #:add-index #:get-index #:remove-index #:map-indices #:get-primary-key #:primary #:key-form #:key-fn @@ -260,6 +260,9 @@ #:lookup-persistent-symbol-id #:struct-constructor + + #:map-class-query + #:get-query-instances ) ) --- /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/03/01 02:45:45 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/04/01 20:56:19 1.2 @@ -61,7 +61,7 @@ (slot-value (slot-value person 'department) 'name))) (defun example-query1 () - "Performs a query against a single class. Trivial string & integer matchingA" + "Performs a query against a single class. Trivial string & integer matching" (map-class-query #'print-person '((person name = "Greg") (person salary >= 100000)))) --- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/03/30 14:34:35 1.2 +++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/01 20:56:19 1.3 @@ -41,6 +41,7 @@ (defun get-query-instances (constraints) "Get a list of instances according to the query constraints" + (declare (dynamic-extent constraints)) (let ((list nil)) (flet ((collect (inst) (push inst list))) @@ -52,6 +53,7 @@ slot indices (for last query) and stack allocated test closures. This is a minimally optimizing version that uses the first index it finds, and then does a nested loop join on the rest of the parameters." + (declare (dynamic-extent constraints)) (assert (not (null constraints))) (destructuring-bind (class slot relation &rest values) (first constraints) (flet ((filter-by-relation (inst) From rread at common-lisp.net Sun Apr 1 22:01:52 2007 From: rread at common-lisp.net (rread) Date: Sun, 1 Apr 2007 18:01:52 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-clsql Message-ID: <20070401220152.260305538E@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv8188 Modified Files: package.lisp Log Message: Not sure what happened here but this file is definitely needed for the test on the sql side. --- /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/02/07 22:54:12 1.2 +++ /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/01 22:01:51 1.3 @@ -21,7 +21,7 @@ (defpackage db-clsql (:use :common-lisp :uffi :cl-base64 :elephant :elephant-memutil :elephant-backend -;; :elephant-utils + :elephant-utils #+sbcl :sb-thread )) From rread at common-lisp.net Sun Apr 1 23:28:35 2007 From: rread at common-lisp.net (rread) Date: Sun, 1 Apr 2007 19:28:35 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070401232835.6296356008@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv27892 Modified Files: RUNTEST.lisp Log Message: Modernization of this file. --- /project/elephant/cvsroot/elephant/tests/RUNTEST.lisp 2006/02/05 23:46:41 1.1 +++ /project/elephant/cvsroot/elephant/tests/RUNTEST.lisp 2007/04/01 23:28:35 1.2 @@ -4,60 +4,44 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2005,2006 by Robert L. Read +;;; Copyright (c) 2005,2006,2007 by Robert L. Read ;;; ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. - - -;; This file is now obsolete... -;; Please use SQLDB-test.lisp or BerkeleyDB-tests.lisp - +;; If you are only using one back-end, you may prefer: +;; SQLDB-test.lisp or BerkeleyDB-tests.lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) -(asdf:oos 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) -(asdf:operate 'asdf:load-op :elephant-tests) - (asdf:operate 'asdf:load-op :ele-sqlite3) +(asdf:operate 'asdf:load-op :elephant-tests) (in-package "ELEPHANT-TESTS") -(do-all-tests) -(do-all-tests-spec *testpg-path*) -(do-migrate-test-spec *testpg-path*) -(do-all-tests-spec *testdb-path*) -(do-all-tests-spec *testsqlite3-path*) -;; The primary and secondary test-paths are -;; use for the migration tests. -(setq *test-path-primary* *testpg-path*) -(setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) +;; Test Postgres backend +(setq *default-spec* *testpg-spec*) +(do-backend-tests) + +;; Test BDB backend +(setq *default-spec* *testbdb-spec*) +(do-backend-tests) + +;; Test SQLite 3 +(setq *default-spec* *testsqlite3-spec*) +(do-backend-tests) -(setq *test-path-primary* *testdb-path*) -(setq *test-path-secondary* nil) +;; Test a Migration of data from BDB to postgres +(do-migration-tests *testbdb-spec* *testpg-spec*) -(do-all-tests-spec *test-path-primary*) - - -(use-package :sb-profile) - -(profile "CLSQL") -(profile "POSTGRESQL-SOCKET") -(profile "ELEPHANT") - -(use-package "SB-PROFILE") - -(open-store *testpg-path*) -(open-store *testdb-path*) +;; An example usage. +(open-store *testpg-spec*) (add-to-root "x1" "y1") (get-from-root "x1") - (add-to-root "x2" '(a 4 "spud")) (get-from-root "x2") From rread at common-lisp.net Mon Apr 2 00:51:07 2007 From: rread at common-lisp.net (rread) Date: Sun, 1 Apr 2007 20:51:07 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070402005107.3B3E048152@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv13016 Modified Files: data-store-reference.texinfo installation.texinfo intro.texinfo make-ref.lisp tutorial.texinfo user-guide.texinfo Log Message: Sprucing up Chapter 3. --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/02 00:51:06 1.6 @@ -138,10 +138,8 @@ @include includes/class-elephant-indexed-btree.texinfo To create the backend-appropriate type of btree, the backend -implements these methods aginst their store-controller. - - at include includes/fun-elephant-build-btree.texinfo - at include includes/fun-elephant-build-indexed-btree.texinfo +implements this method (and possibly related methods) aginst their store-controller. + at include includes/fun-elephant-backend-build-btree.texinfo Most of the user-visible operations over BTrees must be implemented. Class indexing functions such as @code{map-class} and --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/02 00:51:06 1.7 @@ -66,7 +66,9 @@ test other combinations, but practically these configurations will be the most stable and reliable. Elephant is becoming quite stable in general, so don't be afriad to try an unemphasized combination - -chances are it is just a little more work to bring it up. +chances are it is just a little more work to bring it up. In particular, +Elephant can probably work with MySQL or Oracle with just a little work, +but nobody has asked for this yet. @subsection Library dependencies @@ -223,12 +225,53 @@ @comment node-name, next, previous, up @section Berkeley DB Data Store +The Berkeley DB Data Store started out as a very simple data dictionary in the +Berkeley Unix operating system. There are many ``Xdb'' systems that use the +same API, or a similarly one. A commercial version of the BDB was provided by +Sleepycat Software, and this product has since been purchased by Oracle corporation, +and is currently distributed under a similar licensing arrangement, with commercial +support also available. Please follow the download and installation procedures +defined here: +http://www.oracle.com/technology/products/berkeley-db/db/index.html + +Elephant only works with version 4.5 of BerkeleyDB. @node Berkeley DB Example @comment node-name, next, previous, up @section Setting up Berkeley DB +Beyond ensuring that the file ``my-config.sexp'' points correctly +to your BDB installation, nothing else should be required to configure +the example that uses a local ``testdb'' directory as a dabase (under ``tests'') +in the top-level Elephant directory. + +On one Fedora based system, the ``my-config.sexp'' file looked like this: + + at lisp +((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:pthread-lib . nil) + (:clsql-lib . "/usr/local/share/common-lisp/") + (:compiler . :gcc)) + at end lisp + +The @ref{Test Suites} give a nice example of using BDB by running the test using +the specification: + at lisp +'(:BDB "/home/me/db/testdb/")) + at end lisp + +Once you start working on an application, you will want to change this path +to a directory that make sense for you application, and use that as +the specification passed to + at lisp open-store + at end lisp +in your application. + + @node CL-SQL Data Store @comment node-name, next, previous, up @section CL-SQL Data Store @@ -237,7 +280,7 @@ the original Elephant system has been experimenetally extended to support the use of relational database management systems as the implementation of the persistent store. This relies on Kevin Rosenberg's -CL-SQL interface to relational systems. +CL-SQL interface to a large number of relational systems. Although the BerkeleyDB system is an ideal object store for LISP objects, one might prefer the licensing of a different system. For example, at @@ -246,13 +289,10 @@ http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. -Neither the PostGres DBMS nor SQLite 3 has any such restriction. Elephant itself is released -under the GPL. It is somewhat debatable if the GPL allows one to construct -to construct a non-open-source web application but the preponderance of -opinion appears to be that it does. Thefore using Elephant and the other -GPLed software that it depends upon allows one to host a a non open-source -web application. This might be a reason to use Elephant on PostGres of SQLite rather -than Elephant on BerkeleyDB. +Neither the PostGres DBMS nor SQLite 3, nor Elephant itself, imposes +any such restriction. (Elephant is released under the LLGPL +(http://opensource.franz.com/preamble.html .) Older versions were +released under the GPL.) Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application @@ -260,27 +300,22 @@ the tools associated with those systems, etc. The SQL back-end extention of Elephant provides a function for migrating -data seamlessly between repositories. That is, one can quite easily move +data seamlessly between repositories. One can quite easily move data from a BerkeleyDB repository to a PostGres repository, and vice versa. -In fact, one of the most important aspects of the extention is that it -makes Elephant a multi-repository system, rather than a single repository -system, as addition to allowing different implementation strategies for -those repositories. This offers at least the possiblity than once -can develop using one backend, for example BerkeleyDB, and then later -move to MySQL. +This offers at least the possiblity than once can develop using one backend, +for example BerkeleyDB, and then later move to Postgres, or vice versa. +One could even operate simultaneously out of multiple repositories, if +there were a good reason to do so. At the time of this writing, the basic strategy for the SQL implementation is quite simple. The same serializer used for the Sleepycat implementation is employed, the byte-string is base64 encoded, and placed in a single table which is managed by Elephant. -As of Elephant 0.3, Elephant has been tested to work with both Postgres, and -SQLite 3, thanks do Dan Knapp. - -As far as is known at this writing, all functionality except nested transaction -support and cursor-puts supported by the BerkeleyDB backend is supported by the -CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested -well for the CL-SQL based system. +All functionality except nested transaction support and cursor-puts +supported by the BerkeleyDB backend is supported by the CL-SQL +back-end. Concurrency and transaction atomicity have not been stress +tested well for the CL-SQL based system. Additionally, it is NOT the case that the Elephant system currently provides transaction support across multiple repositories; it provides the transaction @@ -290,13 +325,7 @@ The PostGres backend is as currently employed is about 5 times slower than the BerkeleyDB backend. This could probably change with continued development. -CL-SQL supports a lot of DBMS systems, but only PostGres has been tested. - -The SQL back-end extention has only been tested under SBCL 0.8.18. - -The SQL back-end is as easy to use as the BerkeleyDB back-end. However, -the multi-repository version somewhat complicates the underlying -persistent object management. +CL-SQL supports a lot of DBMS systems, but only PostGres and SqlLite 3 have been tested. @node CL-SQL Example @comment node-name, next, previous, up @@ -306,6 +335,9 @@ @enumerate @item Install postgres and make sure postmaster is running. +Postgres may be installed on your system; you may be able to use a package manager to +install it, or you can install it quite easily from the PostgresSQL site directly +(http://www.postgresql.org/). @item Create a database called ``test'' and set its permissions to be reached by whatever connection specification you intend to use. The @@ -377,34 +409,39 @@ The text of this file is included here to give the casual reader an idea of how elepant test can be run in general: @lisp -;; This file is an example of how to perform the -;; migration tests. You will have to modify it -;; slightly depending on the systems that want to test... -;; You can test migration even between two BDB respositories if you wish +;; If you are only using one back-end, you may prefer: +;; SQLDB-test.lisp or BerkeleyDB-tests.lisp (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) -(asdf:operate 'asdf:load-op :clsql-postgresql-socket) (asdf:operate 'asdf:load-op :ele-bdb) +(asdf:operate 'asdf:load-op :ele-sqlite3) + (asdf:operate 'asdf:load-op :elephant-tests) -;; For sqlite-3.. -;; (asdf:operate 'asdf:load-op :ele-sqlite3) (in-package "ELEPHANT-TESTS") -;; The primary and secondary test-paths are -;; use for the migration tests. - -;; This this configuration for testing between BDB and SQL.... -(setq *test-path-primary* *testpg-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path*) - -;; This this configuration for testing from one BDB repository to another... -(setq *test-path-primary* *testdb-path*) -;; (setq *test-path-primary* *testsqlite3-path*) -(setq *test-path-secondary* *testdb-path2*) +;; Test Postgres backend +(setq *default-spec* *testpg-spec*) +(do-backend-tests) + +;; Test BDB backend +(setq *default-spec* *testbdb-spec*) +(do-backend-tests) + +;; Test SQLite 3 +(setq *default-spec* *testsqlite3-spec*) +(do-backend-tests) + +;; Test a Migration of data from BDB to postgres +(do-migration-tests *testbdb-spec* *testpg-spec*) + +;; An example usage. +(open-store *testpg-spec*) +(add-to-root "x1" "y1") +(get-from-root "x1") -(do-migrate-test-spec *test-path-primary*) +(add-to-root "x2" '(a 4 "spud")) +(get-from-root "x2") @end lisp The appropriate test should execute for you with no errors. @@ -440,5 +477,29 @@ @comment node-name, next, previous, up @section Documentation +If you are getting the documentation as a released tar file, you will probably find +the documenation in .html or .pdf form in the release, or can find it at the +Elephant website. + +If you want to compile the documentation youself, for example, if you can +think of a way to improve this manual, then you will do something similar +to this in a shell or command-line prompt: + at code +cd doc +make +make pdf + at end code + +This process will populate the ``./includes'' directory with references +automatically extracted from the list code. It will then compile the +texinfo documenation source into both HTML and a PDF which will be left +in the ``doc/elephant'' directory. + +Don't edit anything in the ``doc/elephant'' directory or the +``doc/includes'' directories, as everything in these directories is +generated. Instead, edit the ``.texinfo'' files in the doc directory. + + + --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/03/24 12:16:02 1.5 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/02 00:51:06 1.6 @@ -28,7 +28,8 @@ Elephant has been extended to provide support for multiple backends, specifically a relational database backend based on CL-SQL which has -been tested with Postgres and SQLite 3. It supports, with some care, +been tested with Postgres and SQLite 3, and probably support +other relational systems easily. It supports, with some care, multi-repository operation and enables convenient migration of data between repositories. --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/30 23:36:52 1.6 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/02 00:51:06 1.7 @@ -30,7 +30,8 @@ (make-instance 'elephant::persistent-collection :sc sc :from-oid 10) (make-instance 'elephant::secondary-cursor) (make-instance 'elephant::indexed-btree :sc sc :from-oid 10) - (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" +;; (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" + (sb-texinfo:generate-includes #p"." (find-package :elephant) (find-package :elephant-backend) (find-package :elephant-memutil) --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 20:22:24 1.12 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/02 00:51:06 1.13 @@ -32,6 +32,11 @@ Elephant provides a persistent index which maintains an ordered collection of lisp values or persistent object references. +The use of persistent objects makes coding concise, convenient, and +powerful, and makes persistence almost invisible to the programmer. +However, Elephant also allows the same basic data dictionary of +key/value retrieval that BerkeleyDB provides. + When someone says "database," most people think of SQL Relational Data Base Management Systems (e.g. Oracle, Postgresql, MySql). Those systems store data in statically typed tables with unique shared @@ -202,9 +207,11 @@ values and objects can be stored: numbers (except for complexes), symbols, strings, nil, characters, pathnames, conses, hash-tables, arrays, CLOS objects and structs. Nested and circular things are -allowed. You can store basically anything except lambdas, closures, -class objects, packages and streams. (These may eventually get -supported too.) +allowed. Nested and circular things are allowed. You can store +basically anything except compiled functions, closures, class objects, +packages and streams. Functions can be stored as uncompiled lambda +expressions. (Compiled functions and other kinds of objects may +eventually get supported too.) Elephant needs to use a representation of data that is independant of a specific lisp or data store. Therefore all lisp values that are --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 20:56:19 1.7 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/02 00:51:06 1.8 @@ -155,8 +155,8 @@ suggest or add indices to facilitate better performance on common queries. -There are two functions @ref{Function elephant:get-query-instances} -and @ref{Function elephant:map-class-query} which accept a set of +There are two functions @ref{Function elephant:get-query-instances,,,includes/fun-elephant-get-query-instance } +and @ref{Function elephant:map-class-query,,,includes/fun-elephant-map-class-query} which accept a set of constraints instead of the familiar value or range arguments. We'll use the classes @code{person} and @code{department} to From rread at common-lisp.net Mon Apr 2 13:09:46 2007 From: rread at common-lisp.net (rread) Date: Mon, 2 Apr 2007 09:09:46 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070402130946.CE8D4671A6@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv2667 Modified Files: scenarios.texinfo Log Message: Added a blurb about Konsenti. --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 1.1 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/02 13:09:46 1.2 @@ -23,3 +23,21 @@ Object-oriented data storage, large graph traversals + at node Commercial Applications + +Elephant is used by Konsenti(tm), a for-profit company of Robert L. Read, one of the maintainers of Elephant. It can be visited at + at uref{http://konsenti.com}. Konsenti uses the Data Collection Management (DCM) package, which can be +found in the contrib directory, under user rread. DCM provides prevalence-style in-memory write-through caching. +The most enjoyable feature about Elephant for this project is that new Business Layer objects can be created without having to +deal with an Object-Relational Mapping, which has allowed extremely rapid development. All Business objects are persisted via +a director in DCM (which sits on top of Elephant.) Many of these business objects are in fact finite state machines decorated +with functions. The functions are represented by their lambda expression stored in slots on the business objects. A complete +Message Factory and double-entry accounting system are also implemented as DCM objects. Binary objects, such as uploaded +PDFs that can be attached to objects as comments, are treated as simple objects and stored directly in Elephant. Konsenti +is completely based on utf-8, and unicode characters outside of the ISO-9959-1 character set are routinely stored in +Elephant. Konsenti uses Postgres as a backend; but Elephant makes it so easy to migrate between repositories that we +could change this decision at any time. + + + + From ieslick at common-lisp.net Wed Apr 4 15:28:29 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 4 Apr 2007 11:28:29 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-bdb Message-ID: <20070404152829.41B4C7E008@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv9316/src/db-bdb Modified Files: libberkeley-db.c Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/03/18 20:40:50 1.9 +++ /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/04/04 15:28:28 1.10 @@ -1030,6 +1030,7 @@ #define S2_ARRAY 19 #define S2_STRUCT 20 #define S2_CLASS 21 +#define S2_COMPLEX 22 #define S2_NIL 0x3F #define S2_FILL_POINTER_P 0x40 From ieslick at common-lisp.net Wed Apr 4 15:28:29 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 4 Apr 2007 11:28:29 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070404152829.D0B4247143@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv9316/src/elephant Modified Files: migrate.lisp serializer2.lisp Added Files: gc.lisp Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/03/24 10:49:59 1.13 +++ /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/04/04 15:28:29 1.14 @@ -42,8 +42,9 @@ ;; - Migrate keeps a memory-resident hash of all persistent objects; ;; this is not as bad as it sounds as an object is only an oid reference ;; and a pointer to the store controller it belongs to. However, you -;; may eventually run out of heap space for very large DB's. We can use -;; the old DB to store the mappings if this becomes a problem. +;; may eventually run out of heap space for very large DB's. You can use +;; a third store controller to get around this by setting set-oid-spec to +;; a valid, uncreated store specification. ;; ;; - Each top-level call to migration will be good about keeping track ;; of already copied persistent objects. However the hash is not --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/03/30 23:36:53 1.35 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/04 15:28:29 1.36 @@ -80,6 +80,7 @@ (defconstant +array+ 19) (defconstant +struct+ 20) (defconstant +class+ 21) +(defconstant +complex+ 22) ;; Lispworks support (defconstant +short-float+ 30) @@ -251,6 +252,10 @@ (let ((pstring (namestring frob))) (buffer-write-byte +pathname+ bs) (serialize-string pstring bs))) + (complex + (buffer-write-byte +complex+ bs) + (%serialize (realpart frob)) + (%serialize (imagpart frob))) (hash-table (buffer-write-byte +hash-table+ bs) (let ((idp (gethash frob circularity-hash))) @@ -459,6 +464,10 @@ (setf (car c) (%deserialize bs)) (setf (cdr c) (%deserialize bs)) c)))) + ((= tag +complex+) + (let ((rpart (%deserialize bs)) + (ipart (%deserialize bs))) + (complex rpart ipart))) ((= tag +hash-table+) (let* ((id (buffer-read-fixnum bs)) (maybe-hash (lookup-id id))) --- /project/elephant/cvsroot/elephant/src/elephant/gc.lisp 2007/04/04 15:28:29 NONE +++ /project/elephant/cvsroot/elephant/src/elephant/gc.lisp 2007/04/04 15:28:29 1.1 (in-package :elephant) (defgeneric stop-and-copy-gc (sc &key &allow-other-keys) (:documentation "Wrap all the migrate machinery in a simple top-level gc call. This will copy all the data in the source store to the target store")) (defmethod stop-and-copy-gc ((src store-controller) &key target mapspec replace-source delete-source) (when map-spec (set-oid-spec mapspec)) (let ((target (gc-open-target sc target-spec)) (src-spec (controller-spec src))) (migrate target src) (when map-spec (set-oid-spec nil) (delete-spec mapspec)) (when (or replace-source delete-source) (close-store src) (delete-spec src)) (when replace-source (copy-spec source target)))) (defun delete-spec (spec) "Delete the storage associated with spec" ) (defun copy-spec (src targ) "Copy files associated with spec from src to targ" ) From ieslick at common-lisp.net Wed Apr 4 15:28:31 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 4 Apr 2007 11:28:31 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070404152831.3EF9147185@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv9316/tests Modified Files: testserializer.lisp Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/tests/testserializer.lisp 2007/02/25 20:02:32 1.19 +++ /project/elephant/cvsroot/elephant/tests/testserializer.lisp 2007/04/04 15:28:30 1.20 @@ -173,6 +173,18 @@ (in-out-equal (/ (expt 2 200) (expt 3 300))) (in-out-equal (/ (expt 2 200) (- (expt 3 300))))) t t t t t t t) + +(deftest complexes + (are-not-null + (in-out-equal (sqrt -1)) + (in-out-equal (complex 1)) + (in-out-equal (complex 1.0)) + (in-out-equal (complex (/ 1 2) (/ 2 3))) + (in-out-equal #C(1.0 0.0)) + (in-out-equal #C(2 3)) + (in-out-equal (complex most-positive-fixnum most-negative-fixnum)) + (in-out-equal (complex (expt 2 200) (expt 3 201)))) + t t t t t t t t) (deftest base-strings (are-not-null From ieslick at common-lisp.net Wed Apr 4 15:35:44 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 4 Apr 2007 11:35:44 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070404153544.6FA3116038@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv10431 Modified Files: tutorial.texinfo Log Message: Fix to docs after adding complex --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:28:28 1.14 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:35:44 1.15 @@ -204,14 +204,13 @@ @section Serialization What can you put into the store besides strings? Almost all lisp -values and objects can be stored: numbers (except for complexes), -symbols, strings, nil, characters, pathnames, conses, hash-tables, -arrays, CLOS objects and structs. Nested and circular things are -allowed. Nested and circular things are allowed. You can store -basically anything except compiled functions, closures, class objects, -packages and streams. Functions can be stored as uncompiled lambda -expressions. (Compiled functions and other kinds of objects may -eventually get supported too.) +values and objects can be stored: numbers, symbols, strings, nil, +characters, pathnames, conses, hash-tables, arrays, CLOS objects and +structs. Nested and circular things are allowed. Nested and circular +things are allowed. You can store basically anything except compiled +functions, closures, class objects, packages and streams. Functions +can be stored as uncompiled lambda expressions. (Compiled functions +and other kinds of objects may eventually get supported too.) Elephant needs to use a representation of data that is independant of a specific lisp or data store. Therefore all lisp values that are From ieslick at common-lisp.net Wed Apr 4 15:28:28 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 4 Apr 2007 11:28:28 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070404152828.BDEF17E005@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv9316/doc Modified Files: installation.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Added support for complex serialization (no sorting), latest doc changes and a preliminary GC wrapper --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/02 00:51:06 1.7 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/04 15:28:28 1.8 @@ -175,31 +175,32 @@ @subsection Packages Now that Elephant has been loaded, you can call @code{use-package} in -the cl-user package or create a new package that imports the symbols -exported from package :elephant. +the cl-user package, @lisp CL-USER> (use-package :elephant) => T + at end lisp -OR +use a predefined user package, -(defpackage :elephant-user - (:use :common-lisp :elephant)) + at lisp +CL-USER> (in-package :elephant-user) +=> T + +ELE-USER> @end lisp -Beginners can skip to the end of this section. +or import the symbols into your own project package from :elephant. -Elephant has a common package called elephant that exports a set of -generic functions. It also contains a dispatcher based on the first -element of a specification list that calls the relevant backend -version of @code{open-controller}, the internal method that creates a - at code{store-controller}. Each backend has it's own subclass -implementing the abstract interface of @code{store-controller}. + at lisp +(defpackage :my-project + (:use :common-lisp :elephant)) + at end lisp @subsection Opening a Store -As discussed in the tutoral, you can now open a store to begin using +As discussed in the tutoral, you need to open a store to begin using Elephant: @lisp --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/02 13:09:46 1.2 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/04 15:28:28 1.3 @@ -5,39 +5,138 @@ @chapter Usage Scenarios @cindex Usage Scenarios -Sorry, haven't written this section yet. - -Simple file replacement and indexing -- Keep track of ordinary objects, ignore metaprotocol - -Persist system objects -- Intermingle persistent objects and regular objects -- Look up objects using class indices - -Full database system -- storage, rich data models, references, queries, etc - -Multithreaded web applications -- DB + multithreading - -Object-oriented data storage, large graph traversals - - - at node Commercial Applications - -Elephant is used by Konsenti(tm), a for-profit company of Robert L. Read, one of the maintainers of Elephant. It can be visited at - at uref{http://konsenti.com}. Konsenti uses the Data Collection Management (DCM) package, which can be -found in the contrib directory, under user rread. DCM provides prevalence-style in-memory write-through caching. -The most enjoyable feature about Elephant for this project is that new Business Layer objects can be created without having to -deal with an Object-Relational Mapping, which has allowed extremely rapid development. All Business objects are persisted via -a director in DCM (which sits on top of Elephant.) Many of these business objects are in fact finite state machines decorated -with functions. The functions are represented by their lambda expression stored in slots on the business objects. A complete -Message Factory and double-entry accounting system are also implemented as DCM objects. Binary objects, such as uploaded -PDFs that can be attached to objects as comments, are treated as simple objects and stored directly in Elephant. Konsenti -is completely based on utf-8, and unicode characters outside of the ISO-9959-1 character set are routinely stored in -Elephant. Konsenti uses Postgres as a backend; but Elephant makes it so easy to migrate between repositories that we -could change this decision at any time. - - + at menu +* File Replacement:: Simple deployment of Elephant as file replacement +* Persistent System Objects:: Making persistent objects a natural part of your system +* Crash Recovery:: How to recover application state from application or system crashes +* Elephant as Database:: Using Elephant as a database for records and user data instead of using a SQL relational Database +* Multithreaded Web Applications:: Elephant is a natural match for web applications +* Graph-oriented Applications:: Elephant is good, but not optimized, for graph-oriented applications. +* Real-World Application Examples:: See some real-world applications Elephant has been used for and a brief discussion of how it was used and any novel uses of Elephant. + at end menu + + at node File Replacement + at comment node-name, next, previous, up + at section File Replacement + +One of the annoying overheads in writing many programs is persisting +data between lisp sessions or invocations of that program. Elements +such as configuration files, raw data such as graphics and other +formats take time, attention and are a potential source of bugs. +Elephant can ease these concerns and allow you to work directly with +your natural in-memory representations with no work to encode/decode +formats or manage files in the file system. + +The simplest way to accomplish this is to simply open a store +controller and use the root btree as a key-value store instead of a +file system directory. You might hide some of the complexity sort of +like this: + + at lisp +(defmacro def-resource (name initializer) + (assert (symbolp name)) + `(defparameter name (list nil nil ,initializer))) + +(defun call-initializer (init) + (case init-stmt + (symbol (funcall (symbol-function init-stmt))) + (list (apply (first init) (rest init))))) + +(defun get-resource (name) + (if (and (symbol-value name) + (symbol-value-name) + (let ((newval (get-from-root name))) + (if newval + (setq name (add-to-root name newval)) + (setq name (add-to-root name (call-initializer + at end lisp + + at node Persistent System Objects + at comment node-name, next, previous, up + at section Persistent System Objects + +Persist system objects: + + at itemize + at item Intermingle persistent objects and regular objects + at item Look up objects using class indices + at end itemize + + at node Crash Recovery + at comment node-name, next, previous, up + at section Crash Recovery + + at node Elephant as Database + at comment node-name, next, previous, up + at section Elephant as Database + +Full database system: storage, rich data models, references, queries, etc + + at node Multithreaded Web Applications + at comment node-name, next, previous, up + at section Multithreaded Web Applications + +Multithreaded web applications: DB + multithreading + web objects + + at node Graph-oriented Applications + at comment node-name, next, previous, up + at section Graph-oriented Applications + + at node Real-World Applications + at comment node-name, next, previous, up + at section Real-World Application + + at subsection Konsenti + +Elephant is used by Konsenti(tm), a for-profit company of Robert +L. Read, one of the maintainers of Elephant. It can be visited at + at uref{http://konsenti.com}. + +Konsenti uses the Data Collection Management (DCM) package, found in +the @verbatim{src/contrib/rread directory}. DCM provides +prevalence-style in-memory write-through caching. The most enjoyable +feature about Elephant for this project is that new Business Layer +objects can be created without having to deal with an +Object-Relational Mapping, enabling extremely rapid development. + +All Business objects are persisted via a @code{director} in DCM (which +sits on top of Elephant.) Many of these business objects are in fact +finite state machines decorated with functions. The functions are +represented by lambda s-expressions stored in slots on the business +objects. A complete Message Factory and double-entry accounting +system are also implemented as DCM objects. Binary objects, such as +uploaded PDFs, can be attached to objects as comments and are stored +directly in Elephant. Konsenti is based on utf-8, and unicode +characters outside of the ISO-9959-1 character set are routinely +stored in Elephant. Konsenti uses Postgres as a backend for licensing +reasons; but use of other data stores is possible. + + at node Conceptminer + at subsection Conceptminer + +Conceptminer is an Elephant-based web-mining framework developed by +Ian Eslick (@uref{http://www.media.mit.edu/~eslick}) that performs +large-scale text analysis over the web to identify semantic +relationships such as ``PartOf'', ``DesireOf'' and ``EffectOf'' +between English phrases. + +Elephant's persistence capabilities is used to keep full records of +all source material, extracted relationships and search queries so +that it is always possible to trace the source of a learned relation +and to avoid repeated queries to web search engines. Conceptminer +used Elephant 0.6.0 and the development branch of Elephant 0.9 to +perform months of analysis consisting of millions of pages and a +page/query database of over ten gigabytes. + +There are several uses of Elephant in Conceptminer that bear +examination: + + at itemize + at item Process Components + at item Bulk storage of post-processed web data + at item Class indexes on strings + at item Cheap associations + at item Inverted document index + at end itemize --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/02 00:51:06 1.13 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:28:28 1.14 @@ -520,9 +520,9 @@ But what if we want to read out our friends from oldest to youngest? One way is to employ another btree that maps birthdays to names, but -this will require storing values multiple times for each update and -increases the burden on the programmer. Elephant provides a better -way. +this requires multiple @code{get-value} calls for each update, +increasing the burden on the programmer. Elephant provides several +better ways to do this. The next section @ref{Indexing Persistent Classes} shows you how to order and retrieve persistent classes by one or more slot values. @@ -547,20 +547,20 @@ (defmethod print-object ((f friend) stream) (format stream "#<~A>" (name f))) -(defun encode-birthday (dmy) +(defun encode-date (dmy) (apply #'encode-universal-time (append '(0 0 0) dmy))) (defmethod (setf birthday) (dmy (f friend)) (setf (slot-value f 'birthday) - (encode-birthday dmy)) + (encode-date dmy)) dmy) -(defun decode-birthday (utime) +(defun decode-date (utime) (subseq (multiple-value-list (decode-universal-time utime)) 3 6)) (defmethod birthday ((f friend)) - (decode-birthday (slot-value f 'birthday))) + (decode-date (slot-value f 'birthday))) @end lisp Notice the class argument ``:index t''. This tells Elephant to store @@ -579,9 +579,9 @@ (defun print-friend (friend) (format t " name: ~A birthdate: ~A~%" (name friend) (birthday friend))) -(make-instance 'friend :name "Carlos" :birthday (encode-birthday '(1 1 1972))) -(make-instance 'friend :name "Adriana" :birthday (encode-birthday '(24 4 1980))) -(make-instance 'friend :name "Zaid" :birthday (encode-birthday '(14 8 1976))) +(make-instance 'friend :name "Carlos" :birthday (encode-date '(1 1 1972))) +(make-instance 'friend :name "Adriana" :birthday (encode-date '(24 4 1980))) +(make-instance 'friend :name "Zaid" :birthday (encode-date '(14 8 1976))) (get-instances-by-class 'friends) => (# # #) @@ -629,14 +629,14 @@ (birthday :initarg :birthday :index t))) @end lisp -Notice the :index argument to the slots. Also notice that we dropped -the class :index argument. Specifying that a slot is indexed -automatically registers the class as indexed. While slot indices -increase the cost of writes and disk storage, each entry is only -slightly larger than the size of the slot value. Numbers, small -strings and symbols are good candidate types for indexed slots, but -any value may be used, even different types. Once a slot is indexed, -we can use the index to retrieve objects by slot values. +Notice the :index argument to the slots and that we dropped the class +:index argument. Specifying that a slot is indexed automatically +registers the class as indexed. While slot indices increase the cost +of writes and disk storage, each entry is only slightly larger than +the size of the slot value. Numbers, small strings and symbols are +good candidate types for indexed slots, but any value may be used, +even different types. Once a slot is indexed, we can use the index to +retrieve objects by slot values. @code{get-instances-by-value} will retrieve all instances that are equal to the value argument. @@ -652,7 +652,7 @@ (get-instances-by-range 'friends 'name "Adam" "Devin") => (# #) -(get-instances-by-range 'friend 'birthday (encode-birthday '(1 1 1974)) (encode-birthday '(31 12 1984))) +(get-instances-by-range 'friend 'birthday (encode-date '(1 1 1974)) (encode-date '(31 12 1984))) => (# #) (mapc #'print-friend *) @@ -676,8 +676,8 @@ @end lisp There are also functions for mapping over instances of a slot index. -To map over values, use the :value keyword argument. To map by range, -use the :start and :end arguments. +To map over duplicate values, use the :value keyword argument. To map +by range, use the :start and :end arguments. @lisp (map-class-index #'print-friend 'friend 'name :value "Carlos") @@ -690,21 +690,21 @@ => NIL (map-class-index #'print-friend 'friend 'birthday - :start (encode-birthday '(1 1 1974)) - :end (encode-birthday '(31 12 1984))) + :start (encode-date '(1 1 1974)) + :end (encode-date '(31 12 1984))) name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) => NIL (map-class-index #'print-friend 'friend 'birthday :start nil - :end (encode-birthday '(10 10 1978))) + :end (encode-date '(10 10 1978))) name: Carlos birthdate: (1 1 1972) name: Zaid birthdate: (14 8 1976) => NIL (map-class-index #'print-friend 'friend 'birthday - :start (encode-birthday '(10 10 1975)) + :start (encode-date '(10 10 1975)) :end nil) name: Zaid birthdate: (14 8 1976) name: Adriana birthdate: (24 4 1980) @@ -827,27 +827,29 @@ @subsection Using @code{with-transaction} What is @code{with-transaction} really doing for us? It first starts -a new transaction, attempts to execute the body, and if successful -commit the transaction. If anywhere along the way there is a deadlock -with another thread, contention, or an error the transaction is +a new transaction, attempts to execute the body, and commits the +transaction if successful. If anytime during the dynamic extent of +this process there is a conflict with another thread's transaction, an +error, or other non-local transfer of control, the transaction is aborted. If it was aborted due to contention or deadlock, it attempts to retry the transaction a fixed number of times by re-executing the whole body. -And this brings us to two important caveats: nested transactions and -idempotent side-effects. +And this brings us to two important constraints on transaction bodies: +no dynamic nesting and idempotent side-effects. @subsection Nesting Transactions -In general, you want to avoid nesting @code{with-transaction} -statements. Nested transactions are valid for some data stores -(namely Berkeley DB), but typically only a single transaction can be -active at a time. The purpose of a nested transaction in data stores -that provide it, is break a long transaction into chunks. This way if -there is contention on a given subset of variables, only the inner -transaction is restarted while the larger transaction can continue. -When commit their results, those results become part of the outer -transaction until it in turn commits. +In general, you want to avoid nested uses of @code{with-transaction} +statements over multiple functions. Nested transactions are valid for +some data stores (namely Berkeley DB), but typically only a single +transaction can be active at a time. The purpose of a nested +transaction in data stores that support them is to break a long +transaction into subsets. This way if there is contention on a given +subset of variables, only the inner transaction is restarted while the +larger transaction can continue. When the inner transaction commits +its results, those results become part of the outer transaction but +are not written to disk until the outer transaction commits. If you have transaction protected primitive operations (such as @code{deposit} and @code{withdraw}) and you want to perform a group of @@ -922,13 +924,13 @@ (load-transients) (length *transient-objects*)) -(test-list) +(test-list 3) => 3 -(test-list) +(test-list 3) => 5 -(test-list) +(test-list 3) => 4 @end lisp @@ -936,39 +938,40 @@ parameters is atomic if the transaction completes. @lisp -(defun load-transients () +(defun load-transients (n) "This is a better way" (setq *transient-objects* (with-transaction () - (loop for i from 0 upto 3 collect + (loop for i from 0 upto n collect (get-from-root i))))) @end lisp -Of course we would need to use @code{nreverse} if we cared about the -order of instances in @code{*transient-objects*}. The best rule of -thumb is that transaction bodies should be purely functional as above, -except for side effects to the persistent store such as persistent -slot writes, adding to btrees, etc). - -If you do need side effects to lisp memory, such as writes to -transient slots, make sure they are idempotent and that other -processes will not be reading the written values until the transaction +(Of course we would need to use @code{nreverse} if we cared about the +order of instances in @code{*transient-objects*}) + +The best rule-of-thumb is to ensure that transaction bodies are purely +functional as above, except for side effects to persistent objects and +btrees. + +If you really do need to execute side-effects into lisp memory, such +as writes to transient slots, make sure they are idempotent and that +other processes cannot read the written values until the transaction completes. @subsection Transactions and Performance By now transactions almost look like more work than they are worth! -Well there are still some significant benefits to be had. Part of how -transactions are implemented is that they gather together all the -writes that are supposed to made to the database and store them until -the transaction commits, and then writes them atomically. - -The most time-intensive part of persistent operations is flushing -newly written data to disk. Using the default auto-committing -behavior requires a flush for every primitive write operation. This -can become very expensive! Because all the values read or written are -cached in memory until the transaction completes, the number of -flushes can be dramatically reduced. +Fortunately, there are also performance benefits to explicit use of +transactions. Transactions gather together all the writes that are +supposed to made to the database and store them in memory until the +transaction commits, and only then writes them to the disk. + +The most time-intensive component of a transaction is waiting while +flushing newly written data to disk. Using the default +auto-committing behavior requires a disk flush for every primitive +write operation. This is very, very expensive! Because all the +values read or written are cached in memory until the transaction +completes, the number of flushes can be dramatically reduced. But don't take my word for it, run the following statements and see for yourself the visceral impact transactions can have on system @@ -1003,12 +1006,11 @@ When we increase the number of objects within the transaction, the time cost does not go up linearly. This is because the total time to -write a hundred simple objects is still dominated by the final -synchronization step. +write a hundred simple objects is still dominated by the disk writes. These are huge differences in performance! However we cannot have -infinitely sized transactions due to the need to cache values in -working memory. Large operations (such as loading data into a +infinitely sized transactions due to the finite size of the data +store's memory cache. Large operations (such as loading data into a database) need to be split into a sequential set of smaller transactions. When dealing with persistent objects a good rule of thumb is to keep the number of objects touched in a transaction well @@ -1021,11 +1023,11 @@ as they only show up when transactions are interleaved within a larger, multi-threaded application. -In many cases, however, you can ignore transactions. For example, -when you don't have any other concurrent processes running. In this -case all operations are sequential and there is no chance of -conflicts. You would only want to use transactions for write -performance. +In many cases you can simply ignore transactions. For example, when +you don't have any other concurrent processes running. In this case +all operations are sequential and there is no chance of conflicts. +You would only want to use transactions to improve performance on +repeated sets of operations. You can also ignore transactions if your application can guarantee that concurrency won't generate any conflicts. For example, a web app @@ -1064,6 +1066,10 @@ features in the user guide that were not covered in this tutorial. @itemize + at item @strong{Using Multiple Threads and Processes} + What constraints must be accommodated to use Elephant data stores in +multiple threads? What capabilities are there to share data stores +among multiple processes or machines? @item @strong{Class Heirarchies and Queries} There are some subtle issues to take into account when querying persistent classes. For example, how do you query a base class of @@ -1072,34 +1078,26 @@ @item @strong{Derived Class Indices} You can create your own indices for classes that are arbitrary lisp functions of the persistent object. + at item @strong{Dynamic Class Index Management} + It is possible to add and remove indexes from classes at runtime. @item @strong{Class Definition/Database Conflict Resolution} When you startup lisp, there are potential conflicts between the class definition and the indexing records in the database. There are some constraints to account for and some facilities to manage how slots, class indices and - at item @strong{Dynamic Class Index Management} - It is possible to add and remove indexes from classes at runtime. + at item @strong{Indexed BTrees} + Indexed BTrees are just like BTrees, except it is possible to add +indexes which are BTrees who's values are primary keys in the parent + at code{indexed-btree}. This allows for multiple ordering and groupings +of the values of a BTree. @item @strong{BTree Cursors} If you need to do more than iterate over a collection, or you need to delete elements of the collection as you iterate cursors are an important data structure. They implement a variety of operators for moving backward and forward over a btree, including ranged operations and iterating of duplicate or unique values. - at item @strong{Indexed BTrees} - Indexed BTrees are just like BTrees, except it is possible to add -indexes which are BTrees who's values are primary keys in the parent - at code{indexed-btree}. This allows for multiple ordering and groupings -of the values of a BTree. @item @strong{Using the Map Operators} Mapping operators can be very efficient if properly utilized. - at item @strong{Handling Errors and Conditions} - There are a variety of errors that can occur in Elephant that need -to be dealt with by applications. - at item @strong{Deadlock Detection in Berkeley DB} - Berkeley DB requires an external process to detect deadlock -conditions among transactions. The :deadlock-detect keyword argument -to open-store for Berkeley DB specs will launch this process on most -lisps. @item @strong{Using Multiple Stores} Multiple store controllers can be open simultaneously. However it does make the code more complex and you need to be careful about how @@ -1108,10 +1106,14 @@ You can implement your own version of with-transaction using the underlying controller methods for starting, aborting and committing transactions. You had better know what you are doing, however! - at item @strong{Using Multiple Threads and Processes} - What constraints must be accommodated to use Elephant data stores in -multiple threads? What capabilities are there to share data stores -among multiple processes or machines? + at item @strong{Handling Errors and Conditions} + There are a variety of errors that can occur in Elephant that need +to be dealt with by applications. + at item @strong{Deadlock Detection in Berkeley DB} + Berkeley DB requires an external process to detect deadlock +conditions among transactions. The :deadlock-detect keyword argument +to open-store for Berkeley DB specs will launch this process on most +lisps. @end itemize Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/02 00:51:06 1.8 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/04 15:28:28 1.9 @@ -8,9 +8,9 @@ @menu * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. -* Persistent objects:: All the dirt on persistent objects. +* Persistent Classes and Objects:: All the dirt on persistent objects. * Class Indices:: In-depth discussion about indexing persistent indices. -* Querying persistent instances:: Retrieving instances of classes. + at c * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. @@ -21,67 +21,248 @@ * Repository Migration and Upgrade:: How to move objects from one repository to another. * Garbage Collection:: How to recover storage and OIDs in long-lived repositories. * Performance Tuning:: How to get the most from Elephant. +* Berkeley DB Data Store:: Commands and concerns specific to the :BDB data store +* CL-SQL Data Store:: Commands and concerns specific to the :CLSQL data store @end menu @node The Store Controller @comment node-name, next, previous, up @section The Store Controller -What is @code{open-store} doing? It creates a @code{store-controller} -object, and sets the special @code{*store-controller*} to point to it. -The store controller holds the handles to the database environment and -tables, and some other bookkeeping. If for some reason you need to -run recovery on the database (see sleepycat docs) you can specify that -with the @code{:recover} and @code{:recover-fatal} keys. +An instance of the @code{store-controller} class mediates interactions +between Lisp and a data store. All elephant operations are performed +in the context of a store controller. To be more specific, a data +store provides a subclass of @code{store-controller} specialized to +that data store. Typically this object contains pointers to the disk +files, foreign memory regions and any other necessary bookkeeping +information to support Elephant operations such as slot writes and +btree operations. The store also contains the root objects and other +bookeeping common to all data stores. + +To obtain a @code{store-controller} object, call the function + at code{open-store} with a store controller specification. The current +data store specification formats are: + + at itemize + at item Berkeley DB: '(:BDB "/path/to/datastore/directory/") + at item CL-SQL: '(:CLSQL ( )) + at end itemize + +Valid CLSQL database tags for @code{} are + at code{:SQLITE} and @code{:POSTGRESQL}. The @code{} is +what you would pass to CLSQL's @code{connect} command. + +The open store function uses the first symbol in the specification +(i.e. :BDB or :CLSQL) to dispatch instance creation to the specified +data store which returns a specialized instance of + at code{store-controller}. @code{open-store} then initializes the store +using an internal call to @code{open-controller}. + +The final step of @code{open-store} is to set the global variable + at code{*store-controller*}. This special variable is used as a default +value in the optional or keyword arguments to number of operations +such as: + + at itemize + at item @code{make-instance} for persistent objects + at item @code{get-from-root} and @code{add-to-root} for accessing a store's root + at item @code{make-btree} for creating persistent index instances + at end itemize + +Each of these functions also accepts an explicit store controller +argument for use in multiple store environments. Normal applications +should only be aware that this global parameter is used. For further +discussion of @code{*store-controller*} @pxref{Multi-repository Operation}. + +Additionally, @code{open-store} accepts data store specific keyword +arguments. For example, you can force recovery to be run on Berkeley +DB data stores: + + at lisp +(open-store *my-spec* :recover t) + at end lisp + +The data store sections of the user guide (@ref{Berkeley DB Data +Store} and @ref{CL-SQL Data Store}) list all the data-store specific +options to various elephant functions. + +When you finish your application, @code{close-store} will close the +store controller. Failing to do this properly may lead to a need to +run recovery on the data store during the next session. Again, see +the relevant data store sections for more details. -To create one by hand one can do, + at node Serialization details + at comment node-name, next, previous, up + at section Serialization details - at lisp -* (setq *store-controller* (make-instance 'store-controller :path "testdb")) -=> # +This section captures the details of how various types of objects are +serialized and some considerations to keep in mind when storing lisp +objects. + +The high level factors that you need to keep in mind are: + + at itemize + at item Circular References: +The serializer properly handles circular references to/from objects +such as cons cells, standard objects, arrays, etc. It accomplishes +this by assigning an ID to any non-atomic object and keeping a mapping +between previously serialized objects and these ids. + at end itemize + +Here is an introduction to + + at itemize + at item + at end itemize + +We will also review and add to the considerations outlined in the tutorial: -* (open-controller *store-controller*) -=> # + at enumerate + + + at item @strong{Lisp identity can't be preserved}. Since this is a store which +persists across invocations of Lisp, this probably doesn't even make +sense. However if you get an object from the index, store it to a +lisp variable, then get it again - they will not be eq: + + at lisp +(setq foo (cons nil nil)) +=> (NIL) +(add-to-root "my key" foo) +=> (NIL) +(add-to-root "my other key" foo) +=> (NIL) +(eq (get-from-root "my key") + (get-from-root "my other key")) +=> NIL @end lisp -but + at item @strong{Nested aggregates are stored in one buffer}. +If you store an set of objects in a hash table you try to store a hash +table, all of those objects will get stored in one large binary buffer +with the hash keys. This is true for all other aggregates that can +store type T (cons, array, standard object, etc). + + at item @strong{Circular References}. +The serializer properly handles circular references to/from objects +such as cons cells, standard objects, arrays, etc. It accomplishes +this by assigning an ID to any non-atomic object and keeping a mapping +between previously serialized objects and these ids. + + at item @strong{Mutated substructure does not persist}. @lisp -* (open-store "testdb")) +(setf (car foo) T) +=> T +(get-from-root "my key") +=> (NIL) @end lisp -is the preferred mechanism. +This will affect all aggregate types: objects, conses, hash-tables, et +cetera. (You can of course manually re-store the cons.) In this sense +elephant does not automatically provide persistent collections. If you +want to persist every access, you have to use BTrees (@pxref{Using BTrees}). -This opens the environment and database. The @code{persistent-*} objects -reference the @code{*store-controller*} special. (This is in part because -slot accessors can't take additional arguments.) If for some reason -you want to operate on 2 store controllers, you'll have to do that by -flipping the @code{*store-controller*} special. - - at code{close-store} closes the store controller. Alternatively - at code{close-controller} can be called on a handle. Don't forget to do -this or else you may need to run recovery later. There is a - at code{with-open-controller} macro. Opening and closing a controller -is very expensive. + at item @strong{Storage limitations}. +The serializer writes sequentially into a foreign memory byte array +before passing that array to a given data store's API. There are +practical limits to the size of this buffer. Moreoever, in most data +stores there is a practical limit to the size of a transaction. +Either of these considerations should encourage you to plan to limit +the size of objects that you serialize to disk. A good rule of thumb +is to stay under a megabyte. - at node Serialization details - at comment node-name, next, previous, up - at section Serialization details + at item @strong{Serialization and deserialization can be costly}. While +serialization is pretty fast, but it is still expensive to store large +objects wholesale. Also, since object identity is impossible to +maintain, deserialization must re-cons or re-allocate the entire +object every time increasing the number of GCs the system does. This +eager allocation is contrary to how most people want to use a +database: one of the reasons to use a database is if your objects +can't fit into main memory all at once. + + at item @strong{Merge-conflicts in heavily multi-process/threaded situations}. +This is the common read-modify-write problem in all databases. We will talk +more about this in the @ref{Transactions} section. + + at end enumerate -Empty. - at node Persistent objects + at node Persistent Classes and Objects @comment node-name, next, previous, up - at section Persistent Objects + at section Persistent Classes and Objects + +Persistent classes are Elephant's answer to the limitations of +ordinary lisp object serialization, namely support for persistent +references. Any persistent object, when serialized, only serializes a +reference to the object and not the whole object. For example you can +serialize a node in the graph of persistent objects without worrying +about serializing the entire graph. + + at subsection{Persistent Class Definition} + +Other than specifying the metaclass or using @code{defpclass} the only +important differences in the @code{defclass} form is the specification +of a slot storage policy. Slot storage policy can be specified by a +boolean argument to the slot initargs @code{:persistent} or + at code{:transient}. Slots are @code{:persistent} by default + + at lisp +(defclass my-pclass () + ((pslot1 :accessor pslot1 :initarg :pslot1 :initform 'one) + (pslot2 :accessor pslot2 :initarg :pslot2 :initform 'two :persistent t) + (tslot1 :accessor tslot3 :initarg :tslot3 :initform nil :transient t)) + (:metaclass persistent-metaclass)) + at end lisp + +The :index options to persistent classes are discussed in persistent +indices. + +Slot storage class implications are straightforward. Persistent slot +writes are durably stored to disk in an automatic or encompassing +transaction. Transient slots are initialized on instance creation +according to initforms or to initargs. They are never stored to nor +loaded from the database. + + + at subsection{Instance Creation} + +Persistent objects are instances of the persistent classes defined +above. All persistent objects inherit from the class + at code{persistent} and share two properties: a unique ID and a +reference to the specification of the @code{store-controller} in which +they reside. This is ensured by the instance creation protocol +implemented by @code{persistent-metaclass}. + +Instances are created as normal, with a call to make-instance and +appropriate initargs. + + +The two properties of @code{persistent} can be specified explicitly +during instance creation: + + at lisp +(make-instance 'my-pclass :from-oid 100 :sc *store-controller*) + at end lisp + +These three elements, class, oid and store controller is all that is +needed to create a new instance + +If you do make an instance with a specified OID which already exists +in the database, @code{initargs} to @code{make-instanc} take +precedence over values in the database, which take precedences over +any @code{initforms} defined in the class. + +* Default store controller & instance creation +* What happens to persistent objects when store-controller is closed? + + -Finally, if you for some reason make an instance with a specified OID -which already exists in the database, @code{initargs} take precedence -over values in the database, which take precedences over - at code{initforms}. - -Also currently there is a bug where - at code{initforms} are always evaluated, so beware. -(What is the current model here?) +:: User-defined persistent objects + +* slot types +* caching +* slot access protocol Readers, writers, accessors, and @code{slot-value-using-class} are employed in redirecting slot accesses to the database, so override @@ -90,6 +271,20 @@ the specification to work properly with persistent slots. However the proper behavior has been verified on SBCL, Allegro and Lispworks. +:: Initialization + +Also currently there is a bug where @code{initforms} are always +evaluated, so beware. (What is the current model here?) + +:: Class Redefinition and Evolution + +* What happens when you redefine a class online? +* Drop & add slots? Change slot status? +* What if you connect to an old database with a new class specification? + (ref to class indicies behavior) + +:: Storage and Performance Considerations + @node Class Indices @comment node-name, next, previous, up @section Class Indices @@ -137,112 +332,111 @@ somewhat user customizable; documentation for this exists in the source file referenced above. - at node Querying persistent instances - at comment node-name, next, previous, up - at section Querying persistent instances - -A SQL select-like interface is in the works, but for now queries are -limited to manual mapping over class instances or doing small queries -with @code{get-instances-*} functions. One advantage of this is that -it is easy to estimate the performance costs of your queries and to -choose standard and derived indices that give you the ordering and -performance you want. - -There is, however, a quick and dirty query API example that is not -officially supported in the release but is intended to invite comment. -This is an example of a full query system that would automatically -perform joins, use the appropriate indices and perhaps even adaptively -suggest or add indices to facilitate better performance on common -queries. - -There are two functions @ref{Function elephant:get-query-instances,,,includes/fun-elephant-get-query-instance } -and @ref{Function elephant:map-class-query,,,includes/fun-elephant-map-class-query} which accept a set of -constraints instead of the familiar value or range arguments. - -We'll use the classes @code{person} and @code{department} to -illustrate how to perform queries over a set of objects that may be -constrainted by their relationships to other objects. - - at lisp -(defpclass person () - ((name :initarg :name :index t) - (salary :initarg :salary :index t) - (department :initarg :dept))) - -(defmethod print-object ((p person) stream) - (format stream "#" (slot-value p 'name))) - -(defun print-name (inst) - (format t "Name: ~A~%" (slot-value inst 'name))) - -(defpclass department () - ((name :initarg :name) - (manager :initarg :manager))) - -(defmethod print-object ((d department) stream) - (format stream "#" - (slot-value d 'name) - (when (slot-boundp d 'manager) - (slot-value (slot-value d 'manager) 'name)))) - at end lisp - -Here we have a simple employee database with managers (also of type -person) and departments. This simple system will provide fodder for -some reasonably complex constraints. Let's create a few departments. - - at lisp -(setf marketing (make-instance 'department :name "Marketing")) -(setf engineering (make-instance 'department :name "Engineering")) -(setf sales (make-instance 'department :name "Sales")) - at end lisp - -And manager @code{people} for the departments. - - at lisp -(make-instance 'person :name "George" :salary 140000 :department marketing) -(setf (slot-value marketing 'manager) *) - -(make-instance 'person :name "Sally" :salary 140000 :department engineering) -(setf (slot-value engineering 'manager) *) - -(make-instance 'person :name "Freddy" :salary 180000 :department sales) -(setf (slot-value sales 'manager) *) - at end lisp - [156 lines skipped] From ieslick at common-lisp.net Fri Apr 6 02:51:47 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 5 Apr 2007 22:51:47 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070406025147.0E2B32E1BF@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv21893 Modified Files: TODO elephant.asd Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/TODO 2007/04/01 14:33:23 1.77 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/06 02:51:46 1.78 @@ -18,7 +18,7 @@ - Verify db_deadlock for other lisps (launch and kill background program I/F) Bugs: -- Fix awkward serializer API +- Fix awkward serializer API - Support for asdf-install? - Edi's patches & suggestions for windows @@ -44,6 +44,11 @@ - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) +Additional features to document in Trac: +- Add error/restart conditions when lose data on: class redef, index drop, etc. + +:association slot option + 0.6.1 - Features COMPLETED to date ---------------------------------- --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/01 14:33:23 1.41 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/06 02:51:46 1.42 @@ -298,6 +298,7 @@ (:file "serializer") (:file "controller") (:file "collections") + (:file "pset") (:file "classindex-utils") (:file "classindex") (:file "serializer1") ;; 0.6.0 db's From ieslick at common-lisp.net Fri Apr 6 02:51:47 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 5 Apr 2007 22:51:47 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070406025147.B88933A01F@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv21893/doc Modified Files: user-guide.texinfo Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/04 15:28:28 1.9 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/06 02:51:47 1.10 @@ -89,41 +89,33 @@ When you finish your application, @code{close-store} will close the store controller. Failing to do this properly may lead to a need to run recovery on the data store during the next session. Again, see -the relevant data store sections for more details. +the relevant data store sections for more detail. + @node Serialization details @comment node-name, next, previous, up @section Serialization details -This section captures the details of how various types of objects are -serialized and some considerations to keep in mind when storing lisp -objects. - -The high level factors that you need to keep in mind are: - - at itemize - at item Circular References: -The serializer properly handles circular references to/from objects -such as cons cells, standard objects, arrays, etc. It accomplishes -this by assigning an ID to any non-atomic object and keeping a mapping -between previously serialized objects and these ids. - at end itemize +There are consequences to trying to move values from lisp memory onto +disk in order to persist them. The first consequence is that that +pointers cannot be guaranteed to be valid and so references to lisp +objects cannot be maintained. This is very similar to the problems +with passing references in foreign function interfaces. The second, +and more frustrating limitation is that lisp operations that commit +side effects on aggregate objects, such as objects, arrays, etc, +cannot be trapped and replicated on the disk representation. This +leads up to a very important consequence: all lisp objects are stored +by @emph{value}. This policy has a number of consequences which are +detailed below. -Here is an introduction to - - at itemize - at item - at end itemize - -We will also review and add to the considerations outlined in the tutorial: + at subsection{Restrictions of Store-by-Value} @enumerate - - - at item @strong{Lisp identity can't be preserved}. Since this is a store which -persists across invocations of Lisp, this probably doesn't even make -sense. However if you get an object from the index, store it to a -lisp variable, then get it again - they will not be eq: + at item @strong{Lisp identity can't be preserved}. + Since this is a store which persists across invocations of Lisp, +this probably doesn't even make sense. However if you get an object +from the index, store it to a lisp variable, then get it again - they +will not be eq: @lisp (setq foo (cons nil nil)) @@ -137,17 +129,35 @@ => NIL @end lisp - at item @strong{Nested aggregates are stored in one buffer}. + at item @strong{Nested aggregates are serialized recursively into a single buffer}. If you store an set of objects in a hash table you try to store a hash table, all of those objects will get stored in one large binary buffer -with the hash keys. This is true for all other aggregates that can -store type T (cons, array, standard object, etc). +with the hash keys. This is true for all aggregates that can store +type T (cons, array, standard object, etc). @item @strong{Circular References}. -The serializer properly handles circular references to/from objects -such as cons cells, standard objects, arrays, etc. It accomplishes +One benefit provided by the serializer is that the recursive +serialization process does not lead to infinite loops when they +encounter circular references among aggregate types. It accomplishes this by assigning an ID to any non-atomic object and keeping a mapping -between previously serialized objects and these ids. +between previously serialized objects and these ids. This same +mapping is used to reconstruct references in lisp memory on +deserialization such that the original structure is properly +reproduced. + + at item @strong{Storage limitations}. +The serializer writes sequentially into a contiguous foreign byte +array before passing that array to a given data store's API. There +are practical limits to the size of the foreign buffer that lisp can +allocate (usually somewhere on the order of 10-100MB due to address +space fragmentation). Moreoever, most data stores will have a +practical limit to the size of a transaction or the size of key or +value they will store. Either of these considerations should +encourage you to plan to limit the size of objects that you serialize +to disk. A good rule of thumb is to stay under a handful of +megabytes. We have successfully serialized arrays over 100MB in the +past, but have not tested the robustness of these large values over +time. @item @strong{Mutated substructure does not persist}. @@ -163,15 +173,6 @@ elephant does not automatically provide persistent collections. If you want to persist every access, you have to use BTrees (@pxref{Using BTrees}). - at item @strong{Storage limitations}. -The serializer writes sequentially into a foreign memory byte array -before passing that array to a given data store's API. There are -practical limits to the size of this buffer. Moreoever, in most data -stores there is a practical limit to the size of a transaction. -Either of these considerations should encourage you to plan to limit -the size of objects that you serialize to disk. A good rule of thumb -is to stay under a megabyte. - @item @strong{Serialization and deserialization can be costly}. While serialization is pretty fast, but it is still expensive to store large objects wholesale. Also, since object identity is impossible to @@ -185,8 +186,147 @@ This is the common read-modify-write problem in all databases. We will talk more about this in the @ref{Transactions} section. + at item @strong{Byte Ordering}. + The primitive elements such as integers are written to disk in +the native byte ordering of the machine on which the lisp runs. This +means that little endian machines cannot read values written by big +endian machines and vice a versa. + + at item @strong{Unicode codes and Serialized Strings}. + The characters and strings stored to disk can store and recover +lisp character codes that implement unicode, but the character maps +are the lisp character maps (produced by @code{char-code}) and not +strict unicode codes so lisps may not be able to interoperably read +characters unless they have identical character code maps for the +character sets you are reading and writing. All standard ASCII +strings should be portable. Here is what we know about specific +lisps, but this should not be taken as gospel. + at itemize + at item SBCL: In versions with the :sb-unicode feature (after 0.8.17) @code{char-code} + produces proper Unicode codes + at item Allegro: In the interational version, @code{char-code} produces proper Unicode codes for codes < 2^16 + at item OpenMCL: OpenMCL 1.1 supports unicode, we are unsure about earlier versions + at item Lispworks: Lispworks 5 does not, to our knowledge, produce proper Unicode characters. +(@emph{This can be fixed on request iff users ask for it and are willing to pay the performance hit}) + at end itemize + @end enumerate + at subsection{Atomic Types} + +Atomic types have no recursive substructure. That is they cannot +contain arbitrary objects and are of a bounded size. (Bignums are an +exception, but they have a predictable structure and cannot reference +or otherwise encapsulate other objects). The following is a list of +atoms and a discussion of how they are serialized. + + at itemize + at item @strong{@code{nil}}: + nil has it's own special tag in the serializer so it is easily +identifiable. @code{nil} is an awkward value as it is also a boolean. +The boolean value @code{t} is stored as the symbol 'T. + at item @strong{fixnums}: + The serializer will store both 32-bit and 64-bit fixnums. Both +types of fixnums are readable by a 32-bit or 64-bit lisp, but 64-bit +fixnums are only written if the underlying lisp is supports fixnums +between 32 and 64 bits. + at item @strong{bignums}: + Bignums are broken into a sequence of fixnum-sized chunks and +assembled by masking words onto the bignum. This is awfully +expensive, but it's always correct and fully portable. + at item @strong{small-float}: + Supported only on Lispworks 5 where type @code{small-float} is +not equivalent to type @code{single-float} as it is on all other +supported platforms. Written to disk and deserialized as a single +float so any memory footprint savings of @code{small-float} is lost. + at item @strong{single-float}: + 32-bit floating point numbers + at item @strong{double-float}: + 64-bit floating point numbers + at item @strong{rational}: + A rational is merely a ratio of two integers stored as fixnums or bignums. + at item @strong{complex}: + A complex is a pair of floating point values, rationals or integers. + at item @strong{char}: + Standalone chars are represented by their char-code and are +stored in 32-bit format to ensure that all lisps are stored correctly. + at item @strong{strings}: + Strings can be represented as 8, 16 or 32 bit sequences +depending on the character sizes used in the underlying lisp. Because +strings can be such a large percentage of on-disk space, Elephant uses +a peculiar method of encoding strings. Strings are converted from +their in-memory representation using @code{char-code}. The size of +the first character dictates the word width used for encoding. If a +character violates the word width, the string encoding is aborted and +the next larger width is chosen. The rationale here is that many +strings consist of Latin characters with codes less than 256. Strings +stored in other character sets tend to all be of codes > 256. +Therefore it is likely that the first character will properly +determine the word size of the string. (@emph{On request, we can easily make +a configuration option to fix the word width for encoding}) + at item @strong{pathname}: + A pathname is merely the @code{namestring} of the path object +stored as a string. The path object is reconstructed from the +namestring using @code{parse-namestring} during deserialization. + at item @strong{symbol}: + Symbols are stored as two strings, the package name and the symbol name in that package. When deserialized, the target package is searched for and the symbol is interned in that package. + at end itemize + + at subsection{Aggregate Types} + +The next list are @emph{aggregate} types, meaning that elements of +that type can contain references to elements of type @code{T}. That +means, in theory, that storing an aggregate type to disk that refers +to other objects can copy every reachable object! This is a direct +and dire consequence of the ``store-by-value'' restriction. +(@xref{Persistent Classes and Objects} for how to design around the +store-by-value restriction}). + +This list describes how aggregates are handled by the serializer. + + at itemize + at item @strong{cons}: + Cons is simply stored as a cons record containing two nested +elements. Linear lists are not treated specially (i.e. no cdr-coding) +by the serializer. + at item @strong{array}: + Arrays are stored as sequences of nested, serialized elements. +The array parameters are also stored so that arrays with fill +pointers, adjustable arrays can be stored and reconstructed. The only +arrays that cannot be reproduced are displaced arrays, which are +copied by value and reconstructed as standard arrays during +deserialization. + at item @strong{hash-table}: + Hash tables are stored as a sequence of key-value pairs, where +the key and value can be any serializable value. On deserialization, +the reconstructed key and value quantities are written incrementally +into the hash table. The hash table does remember it's test, rehash +size and threshold and it's total count. The final size of the new +hash table is set to @code{(* (/ size reshash-threshold) rehash-size)}. + at item @strong{struct}: + Structure objects are serialized using the metaprotocol. Each +slot where the value is bound is serialized by serializing the slot +name and the value in sequence. The underlying lisp must support the + at code{struct-constructor} method so that a new, empty instance of the +structure can be created and then populated by the stored keys and +values. + at item @strong{object}: + Instances of subclasses of standard-object are stored almost +identically to structs. The type of the object is stored and the +object slots with bound values are serialized as slotname-value pairs. +To read an object of this type, the lisp image must have the class +defined and it must have at least the slots that are stored on disk. +There is no good method for schema evolution (redefining objects to +have less slots) of ordinary classes. + at end itemize + + +One final strategic consideration is whether you plan on sharing the binary +database between machines or between different lisp platforms on the +same machine. This is almost possible today, but there are some +restrictions. In the section @ref{Repository Migration and Upgrade} +we will discuss possible ways of migrating an existing database across +platforms and lisps. @node Persistent Classes and Objects @comment node-name, next, previous, up From ieslick at common-lisp.net Fri Apr 6 02:51:48 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 5 Apr 2007 22:51:48 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/eslick/db-lisp Message-ID: <20070406025148.3F10A3D00A@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/eslick/db-lisp In directory clnet:/tmp/cvs-serv21893/src/contrib/eslick/db-lisp Modified Files: TODO transactions.lisp Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/src/contrib/eslick/db-lisp/TODO 2007/02/14 04:36:09 1.4 +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/db-lisp/TODO 2007/04/06 02:51:47 1.5 @@ -18,6 +18,22 @@ ========================= +Transaction arch for lisp backend: +- store values during xaction in private lisp structure instead of buffer pool? + then multiple xactions share buffer pool until write & flush and no page dups + fast map oid+slotname to value on slot access and update on write +- btree reads cached as: oid+key? + btree writes update map or add if not cached + btree delete is queued and checked on access + can bypass cache btree vals on map ops (linear traversal) + quick conflict check with other xactions! + how to handle combining writes to btree? +- cursor ops cache all +- single thread optimization? + cache values globally + update on commit + + High level lisp backend design: - Page storage, layout policy; lisp array or foreign data? - key length limits --- /project/elephant/cvsroot/elephant/src/contrib/eslick/db-lisp/transactions.lisp 2007/03/30 15:03:46 1.1 +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/db-lisp/transactions.lisp 2007/04/06 02:51:47 1.2 @@ -9,3 +9,4 @@ ((log :accessor btree-log :initarg :log :documentation "The transaction log"))) + From ieslick at common-lisp.net Fri Apr 6 02:51:48 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 5 Apr 2007 22:51:48 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070406025148.B25C63D00A@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv21893/src/elephant Modified Files: package.lisp serializer2.lisp Added Files: pset.lisp Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/01 20:56:19 1.28 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/06 02:51:48 1.29 @@ -206,6 +206,8 @@ #:persistent #:persistent-object #:persistent-metaclass #:defpclass #:persistent-collection #:drop-pobject + #:pset #:make-pset #:insert-item #:remove-item #:map-pset #:find-item #:pset-list + #:btree #:make-btree #:get-value #:remove-kv #:existsp #:indexed-btree #:make-indexed-btree --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/04 15:28:29 1.36 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/06 02:51:48 1.37 @@ -368,7 +368,8 @@ (,+object+ . "standard object") (,+array+ . "array") (,+struct+ . "struct") - (,+class+ . "class"))) + (,+class+ . "class") + (,+complex+ . "complex"))) (defun enable-deserializer-tracing () (setf *trace-deserializer* t)) --- /project/elephant/cvsroot/elephant/src/elephant/pset.lisp 2007/04/06 02:51:48 NONE +++ /project/elephant/cvsroot/elephant/src/elephant/pset.lisp 2007/04/06 02:51:48 1.1 (in-package :elephant) ;; ;; Simple implementation of persistent sets ;; ;; ;; Public API ;; (defclass pset (persistent-collection) () (:documentation "An unordered persistent collection of unique elements according to serializer equal comparison")) (defgeneric insert-item (item pset) (:documentation "Insert a new item into the pset")) (defgeneric remove-item (item pset) (:documentation "Remove specified item from pset")) (defgeneric map-pset (fn pset) (:documentation "Map operator for psets")) (defgeneric find-item (item pset &key key test) (:documentation "Find a an item in the pset using key and test")) (defgeneric pset-list (pset) (:documentation "Convert items of pset into a list for processing")) (defgeneric build-pset (sc) (:documentation "Construct an empty default pset or backend specific pset. This is an internal function used by make-pset")) ;; NOTE: Other operators? ;; - Efficient union, intersection and difference fn's exploiting an underlying ;; sorted order? ;; - map delete operator? ;; ;; Default implementation based on btrees ;; (defclass default-pset (pset) ((btree :accessor pset-btree :initarg :btree))) (defmethod build-pset ((sc store-controller)) "Default pset method; override if backend has better policy" (let ((btree (make-btree sc))) (make-instance 'default-pset :btree btree :sc sc :from-oid (oid btree)))) (defun make-pset (&key items pset (sc *store-controller*)) (let ((new-pset (build-pset sc))) (when (and items pset) (error "Can only initialize a new pset with item list or pset to copy, not both")) (when items (mapc (lambda (item) (insert-item item new-pset)) items)) (when pset (map-pset (lambda (item) (insert-item item new-pset)) pset)) new-pset)) (defmethod insert-item (item (pset default-pset)) (setf (get-value item (pset-btree pset)) t) item) (defmethod remove-item (item (pset default-pset)) (remove-kv item (pset-btree pset)) item) (defmethod find-item (item (pset default-pset) &key key (test #'equal)) (if (not (or key test)) (get-value item (pset-btree pset)) (map-btree (lambda (elt dc) (declare (ignore dc)) (let ((cmpval (if key (funcall key elt) elt))) (if (funcall test item cmpval) (return-from find-item elt)))) (pset-btree pset)))) (defmethod map-pset (fn (pset default-pset)) (map-btree fn (pset-btree pset)) pset) (defmethod pset-list ((pset default-pset)) (let ((list nil)) (flet ((collect (item) (push item list))) (declare (dynamic-extent collect)) (map-btree (lambda (item dc) (declare (ignore dc)) (push item list)) (pset-btree pset))) list)) From ieslick at common-lisp.net Fri Apr 6 02:51:54 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 5 Apr 2007 22:51:54 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070406025154.0C31783077@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv21893/tests Modified Files: testcollections.lisp Log Message: Trial pset abstraction; fix for debug serialize of complex and more documentation edits --- /project/elephant/cvsroot/elephant/tests/testcollections.lisp 2007/02/21 04:47:47 1.19 +++ /project/elephant/cvsroot/elephant/tests/testcollections.lisp 2007/04/06 02:51:50 1.20 @@ -695,6 +695,23 @@ nil t nil ) +(defparameter test-items '(1 2 3 (1) (2) test1 test2)) + +(deftest pset + (let ((pset1 (make-pset))) + (mapc (lambda (item) + (insert-item item pset1)) + test-items) + (remove-item (list 2) pset1) + (remove-item 'test2 pset1) + (let ((list (pset-list pset1))) + (values + (= (length (pset-list pset1)) 5) + (not (find-item 'test2 pset1)) + (is-not-null (find-item 'test1 pset1)) + (is-not-null (find-item 1 pset1 :key (lambda (x) (when (consp x) (car x))) :test #'eq))))) + t t t t) + ;; This test not only does not work, it appears to ;; hang BDB forcing a recovery!?!?!?! From ieslick at common-lisp.net Thu Apr 12 02:45:09 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:45:09 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/eslick Message-ID: <20070412024509.1A1EB1D0C6@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/eslick In directory clnet:/tmp/cvs-serv28024 Added Files: snapshot-db.lisp Log Message: Cool snapshot hack --- /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 02:45:09 NONE +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 02:45:09 1.1 ;;; ;;; Copyright (c) 2007 Ian Eslick ;;; ;;; Simple snapshot sets. Create a snapshot set for standard objects, ;;; work in-memory and then call snapshot to save the objects to ;;; the underlying store-controller. ;;; ;; ;; Limitations: ;; ;; - Hashes can be registered as indexes of objects. Keys ;; should be simple (numbers, strings, symbols) although ;; arrays in an equalp cache are probably OK too. Values ;; should also be simple or subclasses of standard-object ;; ;; - When a snapshot is taken of a hash, all values that are ;; standard objects are registered. Any refs to registered ;; objects are properly restored on retrieval ;; ;; Easy extensions: ;; ;; - Support arrays of objects as well as hash-tables ;; ;; - Create a method standard-object-dirty-p that defaults to 't' but ;; allows users to implement a method that avoids storing unchanged ;; objects. ;; ;; - Enable versioned or named snapshots ;; ;; - Better interface API (special object metaclass?) to create a more ;; natural abstraction. Could also add object journaling for ;; prevalence dynamics via rucksack log model. For example, writes ;; to slots are saved to a persistent list that gets reused after ;; snapshots (id slotname value). Slot reads are as usual. ;; (in-package :elephant) (defparameter *use-proxy-objects* t "Indicates that the snapshot set should register and write any standard-objects found in slots registered of standard objects during snapshots") (defpclass snapshot-set () ((index :accessor snapshot-set-index :initform (make-btree)) (next-id :accessor snapshot-set-next-id :initform 0) (cache :accessor snapshot-set-cache :initform (make-hash-table) :transient t) (root :accessor snapshot-set-root :initform nil)) (:documentation "Keeps track of a set of standard objects allowing a single snapshot call to update the store controller with the latest state of all objects registered with this set")) (defmethod initialize-instance :after ((set snapshot-set) &rest rest) (declare (ignore rest)) (restore set)) ;; ================= ;; User methods ;; ================= (defmethod register-object ((object standard-object) (set snapshot-set)) "Register a standard object. Not recorded until snapshot is called on db" (if (lookup-cached-id object set) nil (let ((id (incf (snapshot-set-next-id set)))) (cache-snapshot-object id object set) object))) (defmethod register-object ((hash hash-table) (set snapshot-set)) "Adds a hash table to the snapshot set and registers any standard objects stored as values that are not already part of the snapshot. Must call snapshot to save." (if (lookup-cached-id hash set) nil (let ((id (incf (snapshot-set-next-id set)))) (cache-snapshot-object id hash set) hash))) (defmethod set-root ((set snapshot-set)) (if (snapshot-set-root set) (lookup-cached-object (snapshot-set-root set) set) nil)) (defmethod (setf set-root) (value (set snapshot-set)) (setf (snapshot-set-root set) (ensure-registered value))) (defmethod unregister-object (object (set snapshot-set)) "Drops the object from the cache and backing store" (let ((id (gethash object (snapshot-set-cache set)))) (when (null id) (error "Object ~A not registered in ~A" object set)) (drop-cached-object object set) (delete-snapshot-object id set))) (defmethod snapshot ((set snapshot-set)) (maphash (lambda (obj id) (write-object id obj set)) (snapshot-set-cache set))) (defmethod restore ((set snapshot-set)) "Restores a snapshot by setting the snapshot-set state to the last snapshot. If this is used during runtime, the user needs to drop all references to objects and retrieve again from the snapshot set" (clear-cache set) (let ((proxyrecs nil)) (map-btree (lambda (k v) (cond ((hash-table-p v) (push (list k v) proxyrecs)) ((subtypep (type-of v) 'standard-object) (cache-snapshot-object k v set)) (t (error "Invalid type in snapshot-set type ~A for ~A" (type-of v) v)))) (snapshot-set-index set)) ;; All objects should be loaded so object references in hashes are valid (dolist (proxyrec proxyrecs) (destructuring-bind (id proxy) proxyrec (cache-snapshot-object id (restore-proxy-hash proxy set) set))))) (defun map-set (fn set) "Iterates through all values in the set" (maphash (lambda (k v) (declare (ignore v)) (funcall fn k)) (snapshot-set-cache set))) ;; =============== ;; Shorthand ;; =============== ;; Cache ops (defun clear-cache (set) (clrhash (snapshot-set-cache set))) (defun lookup-cached-id (obj set) (gethash obj (snapshot-set-cache set))) (defun lookup-cached-object (id set) (find-hash-key-by-value id (snapshot-set-cache set))) (defun find-hash-key-by-value (value hash) (maphash (lambda (k v) (when (eq v value) (return-from find-hash-key-by-value k))) hash)) (defun cache-snapshot-object (id obj set) (setf (gethash obj (snapshot-set-cache set)) id)) (defun drop-cached-object (obj set) (remhash obj (snapshot-set-cache set))) ;; Save and restore objects (defun read-snapshot-object (id set) (get-value id (snapshot-set-index set))) (defun write-object (id obj set) (setf (get-value id (snapshot-set-index set)) (cond ((subtypep (type-of obj) 'standard-object) (make-proxy-object obj set)) ((eq (type-of obj) 'hash-table) (make-proxy-hash obj set)) (t (error "Cannot only snapshot standard-objects and hash-tables"))))) (defun ensure-registered (obj set) "Return object id by cache lookup or register and write object" (let ((id (lookup-cached-id obj set))) (if id id (progn (register-object obj set) (let ((id (lookup-cached-id obj set))) (write-object id obj set) id))))) (defun delete-snapshot-object (id set) (remove-kv id (snapshot-set-index set))) ;; Snapshot ops (defun reified-class-p (obj) (or (subtypep (type-of obj) 'standard-object) (eq (type-of obj) 'hash-table))) (defclass setref () ((id :accessor snapshot-set-reference-id :initarg :id))) (defun make-proxy-object (obj set) (if (not *use-proxy-objects*) obj (let ((proxy (make-instance (type-of obj)))) (loop for (slotname value) in (subsets 2 (slots-and-values obj)) do (setf (slot-value proxy slotname) (if (reified-class-p value) (make-instance 'setref :id (ensure-registered value set)) value)))))) (defun make-proxy-hash (hash set) (let ((proxy (make-hash-table))) (maphash (lambda (key value) (setf (gethash key proxy) (if (or (subtypep (type-of value) 'standard-object) (subtypep (type-of value) 'hash-table)) (make-instance 'setref :id (ensure-registered value set)) value))) hash) proxy)) (defun restore-proxy-hash (proxy set) "Convert a proxy object to a standard hash, resolving references" (let ((hash (make-hash-table))) (maphash (lambda (k v) (setf (gethash k hash) (if (eq (type-of v) 'setref) (lookup-cached-object (snapshot-set-reference-id v) set) v))) proxy) hash)) From ieslick at common-lisp.net Thu Apr 12 02:47:22 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:47:22 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070412024722.E9B8C2407A@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv28241 Modified Files: LICENSE TODO Log Message: Mostly documentation edits + fixing copyright in all source files --- /project/elephant/cvsroot/elephant/LICENSE 2006/02/04 22:25:09 1.4 +++ /project/elephant/cvsroot/elephant/LICENSE 2007/04/12 02:47:22 1.5 @@ -2,9 +2,13 @@ Copyright (c) 2004 by Andrew Blumberg and Ben Lee -Some parts Copyright (c) 2005,2006 Robert L. Read + +Portions Copyright (c) 2005-2007 Robert L. Read . +Portions Copyright (c) 2006-207 Ian S. Eslick + + Elephant users are granted the rights to distribute and use this software as governed by the terms of the Lisp Lesser GNU Public License (http://opensource.franz.com/preamble.html), also known as the LLGPL. @@ -15,8 +19,8 @@ thought of as particularly appropriate for LISP. Version of Elephant before version 0.5.0 where released under GPL. -With the permission of Andrew Blumberg, Ben Lee and Robert L. Read, -the primary contributors of elephant code up until this time, the -0.5.0 release is made under the LLGPL. +With the permission of Andrew Blumberg, Ben Lee, Robert L. Read and +Ian S. Eslick, the primary contributors of elephant code up until this +time, versions 0.5.0 to 0.9.0 are made under the LLGPL. --- /project/elephant/cvsroot/elephant/TODO 2007/04/06 02:51:46 1.78 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/12 02:47:22 1.79 @@ -11,30 +11,26 @@ TASKS TO GET TO FINAL RELEASE: -Lisp Support: -- Validate OpenMCL pre-1.1 on Mac OS X -- Validate OpenMCL 1.1 and/or 64-bit on Mac OS X? - - 64-bit lisp verification -- Verify db_deadlock for other lisps (launch and kill background program I/F) - Bugs: - Fix awkward serializer API - Support for asdf-install? - Edi's patches & suggestions for windows +Lisp Support: +- Validate OpenMCL pre-1.1 on Mac OS X +- Validate OpenMCL 1.1 and/or 64-bit on Mac OS X? +- Verify db_deadlock for other lisps (launch and kill background program I/F) + Test coverage: - Clean up interface to tests - Test for optimize storage method (just add probe-file methods to get file size?) - Multi-threading stress tests? Ensure that there are conflicts and lots of serialization happening concurrently to make sure that multi-threading is in good shape (Henrik's code) -- Unicode tests - - Ensure that variable length UTF-8 reps are automatically stored as UTF-16? - Class / DB sychronization tests Documentation: -~ License and copyright file headers -- Update install, build and test procedures - Proper user guide +- Update install, build and test procedures - Upgrade, migration and other system level issues - Performance and design issues - More notes about transaction performance @@ -43,11 +39,14 @@ - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) +- Demo file for simple snapshot system + (add standard objects to prevalence DB) + link structure is added to db + in-memory pobjs point to Additional features to document in Trac: - Add error/restart conditions when lose data on: class redef, index drop, etc. - -:association slot option +- :association slot option? 0.6.1 - Features COMPLETED to date ---------------------------------- From ieslick at common-lisp.net Thu Apr 12 02:47:23 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:47:23 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070412024723.913EC7B536@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv28241/doc Modified Files: elephant.texinfo installation.texinfo make-ref.lisp reference.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Mostly documentation edits + fixing copyright in all source files --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/12 02:47:23 1.8 @@ -123,5 +123,4 @@ @uref{http://www.common-lisp.net/project/elephant/}) and submit a patch. - @bye --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/04 15:28:28 1.8 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/12 02:47:23 1.9 @@ -198,6 +198,8 @@ (:use :common-lisp :elephant)) @end lisp +The symbols that are imported are captured in @ref{User API Reference} + @subsection Opening a Store As discussed in the tutoral, you need to open a store to begin using @@ -485,11 +487,11 @@ If you want to compile the documentation youself, for example, if you can think of a way to improve this manual, then you will do something similar to this in a shell or command-line prompt: - at code + at lisp cd doc make make pdf - at end code + at end lisp This process will populate the ``./includes'' directory with references automatically extracted from the list code. It will then compile the --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/02 00:51:06 1.7 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/12 02:47:23 1.8 @@ -1,5 +1,5 @@ (require 'asdf) -(asdf:operate 'asdf:load-op 'elephant) +(asdf:operate 'asdf:load-op 'elephant :force t) (load (merge-pathnames #p"src/elephant/query" (asdf:component-pathname (asdf:find-system 'elephant)))) @@ -29,12 +29,13 @@ (setf (controller-spec sc) nil) (make-instance 'elephant::persistent-collection :sc sc :from-oid 10) (make-instance 'elephant::secondary-cursor) - (make-instance 'elephant::indexed-btree :sc sc :from-oid 10) + (make-instance 'elephant::indexed-btree :sc sc :from-oid 11) + (make-instance 'elephant::pset :sc sc :from-oid 12) ;; (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" - (sb-texinfo:generate-includes #p"." + (sb-texinfo:generate-includes include-dir-path (find-package :elephant) (find-package :elephant-backend) - (find-package :elephant-memutil) + (find-package :elephant-memutil) (find-package :elephant-system)))) (make-docs) --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/12 02:47:23 1.10 @@ -8,10 +8,11 @@ @menu * Store Controllers:: Connecting to a data store. -* Persistent Objects:: Creating and using persistent objects. -* Persistent Object Indexing:: Convenient indexing. -* Query Interfaces:: Finding instances. -* Collections:: BTrees and indices. +* Persistent Class:: Defining persistent classes and creating and manipulating persistent indices. +* Class Indexing:: Convenient indexing of persistent classes. +* Persistent Sets:: Maintaining persistent collections the easy way. + at c * Query Interfaces:: Finding instances. +* BTrees:: BTrees and indices, a low level persistent data structure. * Cursors:: Traversing BTrees. * Transactions:: Transactions. * Migration and Upgrading:: Migration and upgrading. @@ -91,29 +92,49 @@ @include includes/fun-elephant-add-class-derived-index.texinfo @include includes/fun-elephant-remove-class-derived-index.texinfo - at node Query Interfaces + at node Persistent Sets @comment node-name, next, previous, up - at section Query Interfaces - at cindex Query Interfaces + at section Persistent Sets + at cindex Persistent Sets -Query interfaces are currently unimplemented. An example query -interface is provided for reference only, a new system is under -development for the 0.7 release. +Persistent sets are a simple persistent collection abstraction. They +maintain an unordered collection of objects. Unlike the normal +list-oriented sets of Lisp, persistent sets use the equivalent of + at code{pushnew} such that only one copy of any object or value is +maintained using the serializer's @code{equal} implementation. + + at include includes/class-elephant-pset.texinfo + at include includes/fun-elephant-insert-item.texinfo + at include includes/fun-elephant-remove-item.texinfo + at include includes/fun-elephant-find-item.texinfo + at include includes/fun-elephant-map-pset.texinfo + at include includes/fun-elephant-pset-list.texinfo + + at c @node Query Interfaces + at c @comment node-name, next, previous, up + at c @section Query Interfaces + at c @cindex Query Interfaces + + at c The query interface is currently unsupported. See @code{query.lisp} + at c and @code{query-example.lisp} for a sketch of what the query interface + at c will eventually look like. The upcoming version 0.9.1 will include + at c the query system and associated documentation system. @c @include includes/fun-elephant-get-query-results.texinfo @c @include includes/fun-elephant-map-class-query.texinfo - at node Collections + at node BTrees @comment node-name, next, previous, up - at section Collections - at cindex Collections + at section BTrees + at cindex BTrees -Persistent collections inherit from @ref{Class -elephant:persistent-collection} and consist of the @ref{Class -elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class -elephant:btree-index} classes. The following operations are defined -on most of these classes. More information can be found in @ref{Using -BTrees} and @ref{Secondary Indices}. +Persistent collections inherit from @ref{Class elephant:persistent-collection} +and consist of the @ref{Class elephant:btree}, @ref{Class elephant:indexed-btree} and + @ref{Class elephant:btree-index} classes. The following operations are defined +on most of these classes. More information can be found in @ref{Using BTrees} +and @ref{Secondary Indices}. + + at include includes/fun-elephant-make-btree.texinfo @include includes/fun-elephant-get-value.texinfo @include includes/fun-elephant-setf-get-value.texinfo @@ -175,10 +196,26 @@ @include includes/macro-elephant-with-transaction.texinfo +The following functions are an advanced use of the transaction system. +They may be useful if, or example, you want to integrate Elephant +transactions with non-Elephant side-effects that you explicitely make +transactional. + + at include includes/fun-elephant-controller-start-transaction.texinfo + at include includes/fun-elephant-controller-abort-transaction.texinfo + at include includes/fun-elephant-controller-commit-transaction.texinfo + @node Migration and Upgrading @comment node-name, next, previous, up @section Migration and Upgrading @cindex Migration and Upgrading - at include includes/fun-elephant-migrate.texinfo +Upgrade is a call to Migrate with checks for compatability. The +migrate methods are included here in case you wish to develop a more +specific ``partial upgrade'' or ``partial migrate'' of data from one +store to another instead of using the top-level copy which migrates all +live objects. + @include includes/fun-elephant-upgrade.texinfo + at include includes/fun-elephant-migrate.texinfo + --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/04 15:28:28 1.3 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/12 02:47:23 1.4 @@ -48,7 +48,7 @@ (let ((newval (get-from-root name))) (if newval (setq name (add-to-root name newval)) - (setq name (add-to-root name (call-initializer + (setq name (add-to-root name (call-initializer @end lisp @node Persistent System Objects @@ -93,7 +93,7 @@ @uref{http://konsenti.com}. Konsenti uses the Data Collection Management (DCM) package, found in -the @verbatim{src/contrib/rread directory}. DCM provides +the @code{src/contrib/rread directory}. DCM provides prevalence-style in-memory write-through caching. The most enjoyable feature about Elephant for this project is that new Business Layer objects can be created without having to deal with an --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/04 15:35:44 1.15 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/12 02:47:23 1.16 @@ -1048,9 +1048,8 @@ more than two levels of transactional accesses with the top using with-transaction and the bottom using ensure-transaction. - at xref{Transaction Details} for more details and @pxref{Usage -Scenarios} for examples of how systems can be designed and tuned using -transactions. +See @ref{Transaction Details} for more details and @ref{Usage Scenarios} +for examples of how systems can be designed and tuned using transactions. @node Advanced Topics @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/06 02:51:47 1.10 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/12 02:47:23 1.11 @@ -108,7 +108,7 @@ by @emph{value}. This policy has a number of consequences which are detailed below. - at subsection{Restrictions of Store-by-Value} + at subsection Restrictions of Store-by-Value @enumerate @item @strong{Lisp identity can't be preserved}. @@ -212,7 +212,7 @@ @end enumerate - at subsection{Atomic Types} + at subsection Atomic Types Atomic types have no recursive substructure. That is they cannot contain arbitrary objects and are of a bounded size. (Bignums are an @@ -272,14 +272,14 @@ Symbols are stored as two strings, the package name and the symbol name in that package. When deserialized, the target package is searched for and the symbol is interned in that package. @end itemize - at subsection{Aggregate Types} + at subsection Aggregate Types The next list are @emph{aggregate} types, meaning that elements of that type can contain references to elements of type @code{T}. That means, in theory, that storing an aggregate type to disk that refers to other objects can copy every reachable object! This is a direct and dire consequence of the ``store-by-value'' restriction. -(@xref{Persistent Classes and Objects} for how to design around the +(@pxref{Persistent Classes and Objects} for how to design around the store-by-value restriction}). This list describes how aggregates are handled by the serializer. @@ -320,8 +320,7 @@ have less slots) of ordinary classes. @end itemize - -One final strategic consideration is whether you plan on sharing the binary +One final strategic consideration is to whether you plan on sharing the binary database between machines or between different lisp platforms on the same machine. This is almost possible today, but there are some restrictions. In the section @ref{Repository Migration and Upgrade} @@ -332,40 +331,88 @@ @comment node-name, next, previous, up @section Persistent Classes and Objects -Persistent classes are Elephant's answer to the limitations of -ordinary lisp object serialization, namely support for persistent -references. Any persistent object, when serialized, only serializes a -reference to the object and not the whole object. For example you can -serialize a node in the graph of persistent objects without worrying -about serializing the entire graph. - - at subsection{Persistent Class Definition} - -Other than specifying the metaclass or using @code{defpclass} the only -important differences in the @code{defclass} form is the specification -of a slot storage policy. Slot storage policy can be specified by a -boolean argument to the slot initargs @code{:persistent} or - at code{:transient}. Slots are @code{:persistent} by default +Persistent classes are instances of the @code{persistent-metaclass} +metaclass. All instances of this persistent class have a unique ID +and a pointer to the @code{store-controller} specification they are +associated with. Accesses to slot values become direct reads and +writes to the data store and are thus always persisted. When the +instance itself is writtent to the database, for example as a key or +value in a @code{btree}, only the unique ID is stored. + +Thus serialization of persistent objects is exceedingly cheap compared +to standard objects, but slot access can be much more expensive. +Persistent objects are excellent at, for example, storing the link +structure of a graph. A graph of persistent objects can itself be +persisted by storing the head node of a fully-connected graph to the +store controller root. + + at subsection Persistent Class Definition + +To create persistent classes, the user needs to specify the + at code{persistent-metaclass} to the class initarg @code{:metaclass}. +The only differences between standard and persistent class definitions +is the specification of a slot storage policy and an index policy. + +Storage policies are specified by a boolean argument to the slot +initargs @code{:persistent} or @code{:transient}. Slots are + at code{:persistent} by default @lisp (defclass my-pclass () ((pslot1 :accessor pslot1 :initarg :pslot1 :initform 'one) (pslot2 :accessor pslot2 :initarg :pslot2 :initform 'two :persistent t) - (tslot1 :accessor tslot3 :initarg :tslot3 :initform nil :transient t)) + (tslot1 :accessor tslot1 :initarg :tslot1 :initform 'three :transient t)) (:metaclass persistent-metaclass)) @end lisp -The :index options to persistent classes are discussed in persistent -indices. - Slot storage class implications are straightforward. Persistent slot -writes are durably stored to disk in an automatic or encompassing -transaction. Transient slots are initialized on instance creation -according to initforms or to initargs. They are never stored to nor -loaded from the database. +writes are durably stored to disk. Transient slots are initialized on +instance creation according to initforms or to initargs. Transient +slot values are never stored to nor loaded from the database. + +During a given lisp session transient values will be cached as long as +they are not collected by the Lisps GC. After GC, if you retrieve an +object from the store its transient slots will be reset to the slot +initforms from the class definition. + + at lisp +(setf pobj1 (make-instance 'my-pclass :pslot1 1 :tslot3 3)) +=> # + +(pslot1 pobj1) => 1 +(pslot2 pobj1) => 'two +(tslot1 pobj1) => 3 + +(add-to-root 'pobj1 pobj1) + +(setf pobj2 (get-from-root 'pobj1)) +=> # + +(pslot1 pobj2) => 1 +(pslot2 pobj2) => 'two +(tslot1 pobj2) => 3 + +(setf pobj1 nil) +(setf pobj2 nil) +(gc) + +(setf pobj3 (get-from-root 'pobj1)) +(pslot1 pobj2) => 1 +(pslot2 pobj2) => 'two +(tslot1 pobj2) => 'three + at end lisp +The implications of this behavior is that you need to think carefully +about using transient values. Primarily you cannot make assumptions +about the state of transient values in objects loaded from the store +unless you know that they were loaded and cannot be GC'ed. + +Finally, the index policy tells the data store whether to maintain an +inverted index that maps slot values back to their parent objects. +The :index options and behaviors of persistent classes are discussed in +depth in @ref{Class Indices}. - at subsection{Instance Creation} + at subsection Instance Creation Persistent objects are instances of the persistent classes defined above. All persistent objects inherit from the class From ieslick at common-lisp.net Thu Apr 12 02:47:24 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:47:24 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-bdb Message-ID: <20070412024724.A71D22823A@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv28241/src/db-bdb Modified Files: bdb-collections.lisp bdb-controller.lisp bdb-slots.lisp bdb-transactions.lisp berkeley-db.lisp libberkeley-db.c package.lisp Log Message: Mostly documentation edits + fixing copyright in all source files --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-collections.lisp 2007/02/20 19:12:58 1.21 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-collections.lisp 2007/04/12 02:47:23 1.22 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/03/30 14:34:35 1.32 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/04/12 02:47:23 1.33 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-slots.lisp 2007/02/17 12:13:19 1.3 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-slots.lisp 2007/04/12 02:47:23 1.4 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/03/21 14:29:30 1.12 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/12 02:47:23 1.13 @@ -2,8 +2,7 @@ ;;; ;;; bdb-transactions.lisp -- Transaction support for Berkeley DB ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; By Ian Eslick, ;;; ;;; part of ;;; @@ -12,6 +11,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/db-bdb/berkeley-db.lisp 2007/03/03 17:24:58 1.10 +++ /project/elephant/cvsroot/elephant/src/db-bdb/berkeley-db.lisp 2007/04/12 02:47:23 1.11 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/04/04 15:28:28 1.10 +++ /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/04/12 02:47:23 1.11 @@ -12,46 +12,12 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; -;;; This program is released under the following license -;;; ("GPL"). For differenct licensing terms, contact the -;;; copyright holders. +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; ;;; -;;; This program is free software; you can redistribute it -;;; and/or modify it under the terms of the GNU General -;;; Public License as published by the Free Software -;;; Foundation; either version 2 of the License, or (at -;;; your option) any later version. -;;; -;;; This program is distributed in the hope that it will be -;;; useful, but WITHOUT ANY WARRANTY; without even the -;;; implied warranty of MERCHANTABILITY or FITNESS FOR A -;;; PARTICULAR PURPOSE. See the GNU General Public License -;;; for more details. -;;; -;;; The GNU General Public License can be found in the file -;;; LICENSE which should have been distributed with this -;;; code. It can also be found at -;;; -;;; http://www.opensource.org/licenses/gpl-license.php -;;; -;;; You should have received a copy of the GNU General -;;; Public License along with this program; if not, write -;;; to the Free Software Foundation, Inc., 59 Temple Place, -;;; Suite 330, Boston, MA 02111-1307 USA -;;; -;;; Portions of this program (namely the C unicode string -;;; sorter) are derived from IBM's ICU: -;;; -;;; http://oss.software.ibm.com/icu/ -;;; -;;; Copyright (c) 1995-2003 International Business Machines -;;; Corporation and others All rights reserved. -;;; -;;; ICU's copyright, license and warranty can be found at -;;; -;;; http://oss.software.ibm.com/cvs/icu/~checkout~/icu/license.html -;;; -;;; or in the file LICENSE. +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;; */ --- /project/elephant/cvsroot/elephant/src/db-bdb/package.lisp 2007/03/30 14:55:54 1.8 +++ /project/elephant/cvsroot/elephant/src/db-bdb/package.lisp 2007/04/12 02:47:23 1.9 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. From ieslick at common-lisp.net Thu Apr 12 02:47:31 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:47:31 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-clsql Message-ID: <20070412024731.6069233082@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv28241/src/db-clsql Modified Files: package.lisp sql-collections.lisp sql-controller.lisp sql-transaction.lisp Log Message: Mostly documentation edits + fixing copyright in all source files --- /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/01 22:01:51 1.3 +++ /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/12 02:47:24 1.4 @@ -1,6 +1,6 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; sql-controller.lisp -- Interface to a CLSQL based object store. +;;; src/db-clsql/package.lisp - imports for the CLSQL data store ;;; ;;; Initial version 10/12/2005 by Robert L. Read ;;; --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/02/26 19:55:11 1.12 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/12 02:47:24 1.13 @@ -9,7 +9,7 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2005 by Robert L. Read +;;; Copyright (c) 2005-2007 by Robert L. Read ;;; ;;; ;;; Elephant users are granted the rights to distribute and use this software --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-controller.lisp 2007/03/23 16:06:43 1.23 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-controller.lisp 2007/04/12 02:47:24 1.24 @@ -9,7 +9,7 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2005 by Robert L. Read +;;; Copyright (c) 2005-2007 by Robert L. Read ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-transaction.lisp 2007/02/20 02:33:59 1.5 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-transaction.lisp 2007/04/12 02:47:24 1.6 @@ -2,15 +2,15 @@ ;;; ;;; sql-transactions.lisp -- Top level transaction support for elephant sql ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; Initial version 10/12/2005 by Robert L. Read +;;; ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee -;;; +;;; Copyright (c) 2005-2007 by Robert L. Read +;;; Portions Copyright (c) 2006-2007 by Ian Eslick ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License From ieslick at common-lisp.net Thu Apr 12 02:47:51 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:47:51 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070412024751.7A6604D051@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv28241/src/elephant Modified Files: backend.lisp cache.lisp classes.lisp classindex-utils.lisp classindex.lisp cmu-mop-patches.lisp collections.lisp controller.lisp gc.lisp metaclasses.lisp migrate.lisp package.lisp pset.lisp query-example.lisp query.lisp serializer.lisp serializer1.lisp serializer2.lisp transactions.lisp unicode2.lisp variables.lisp Log Message: Mostly documentation edits + fixing copyright in all source files --- /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/04/01 20:56:19 1.16 +++ /project/elephant/cvsroot/elephant/src/elephant/backend.lisp 2007/04/12 02:47:31 1.17 @@ -1,9 +1,8 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; backend.lisp -- Namespace support for backends +;;; backend.lisp -- Namespace support for data store packages ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; By Ian Eslick ;;; ;;; part of ;;; @@ -12,6 +11,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/cache.lisp 2006/02/19 04:53:00 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/cache.lisp 2007/04/12 02:47:31 1.2 @@ -1,5 +1,3 @@ - - ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; ;;; migrate.lisp -- Migrate between repositories @@ -14,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/01 14:33:34 1.26 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/12 02:47:31 1.27 @@ -9,9 +9,12 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; -;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; Original Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/classindex-utils.lisp 2007/02/24 14:51:59 1.6 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex-utils.lisp 2007/04/12 02:47:32 1.7 @@ -1,13 +1,23 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; ;;; classindex-untils.lisp -- support for classindex.lisp and -;;; class re-definition synchronization +;;; class re-definition synchronization ;;; -;;; Initial version 1/24/2006 Ian Eslick -;;; eslick at alum mit edu +;;; By Ian Eslick ;;; -;;; License: Lisp Limited General Public License -;;; http://www.franz.com/preamble.html +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;; (in-package :elephant) --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/03/30 17:45:41 1.35 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/12 02:47:32 1.36 @@ -3,8 +3,17 @@ ;;; classindex.lisp -- use btree collections to track objects by slot values ;;; via metaclass options or accessor :after methods ;;; -;;; Copyright (c) 2006,2007 Ian Eslick -;;; +;;; By Ian Eslick +;;; +;;; part of +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant: an object-oriented database for Common Lisp ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Limited General Public License --- /project/elephant/cvsroot/elephant/src/elephant/cmu-mop-patches.lisp 2006/02/19 04:53:00 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/cmu-mop-patches.lisp 2007/04/12 02:47:32 1.2 @@ -7,6 +7,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/03/30 23:36:53 1.20 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/12 02:47:32 1.21 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. @@ -34,6 +37,9 @@ ;; (defun make-btree (&optional (sc *store-controller*)) + "Constructs a new BTree instance for use by the user. Each backend + returns its own internal type as appropriate and ensures that the + btree is associated with the store-controller that created it." (build-btree sc)) (defgeneric build-btree (sc) @@ -65,6 +71,10 @@ ;; (defun make-indexed-btree (&optional (sc *store-controller*)) + "Constructs a new indexed BTree instance for use by the user. + Each backend returns its own internal type as appropriate and + ensures that the btree is associated with the store-controller + that created it." (build-indexed-btree sc)) (defgeneric build-indexed-btree (sc) --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/01 14:33:34 1.45 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/12 02:47:32 1.46 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/gc.lisp 2007/04/04 15:28:29 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/gc.lisp 2007/04/12 02:47:32 1.2 @@ -1,3 +1,25 @@ +;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- +;;; +;;; gc.lisp - A wrapper around the migrate interface to support +;;; stop-and-copy GC at the repository level +;;; +;;; By Ian Eslick +;;; +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. +;;; + (in-package :elephant) (defgeneric stop-and-copy-gc (sc &key &allow-other-keys) --- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/01 14:33:34 1.15 +++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/12 02:47:32 1.16 @@ -11,13 +11,14 @@ ;;; ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; -;;; (Some changes by Robert L. Read, 2006) +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;; - (in-package "ELEPHANT") (declaim #-elephant-without-optimize (optimize (speed 3) (safety 1))) @@ -61,24 +62,23 @@ ;; Persistent slot maintenance ;; +(defmethod persistent-slots ((class standard-class)) + nil) + (defmethod persistent-slots ((class persistent-metaclass)) (if (slot-boundp class '%persistent-slots) (car (%persistent-slots class)) nil)) -(defmethod persistent-slots ((class standard-class)) - nil) - (defmethod old-persistent-slots ((class persistent-metaclass)) (cdr (%persistent-slots class))) (defmethod update-persistent-slots ((class persistent-metaclass) new-slot-list) -;; (setf (%persistent-slots class) (cons new-slot-list (car (%persistent-slots class))))) - (setf (%persistent-slots class) (cons new-slot-list - (if (slot-boundp class '%persistent-slots) - (car (%persistent-slots class)) - nil) - ))) + (setf (%persistent-slots class) + (cons new-slot-list + (if (slot-boundp class '%persistent-slots) + (car (%persistent-slots class)) + nil)))) (defclass persistent-slot-definition (standard-slot-definition) ((indexed :accessor indexed :initarg :index :initform nil :allocation :instance))) @@ -89,6 +89,7 @@ (defclass persistent-effective-slot-definition (standard-effective-slot-definition persistent-slot-definition) ()) + (defclass transient-slot-definition (standard-slot-definition) ((transient :initform t :initarg :transient :allocation :class))) @@ -98,13 +99,9 @@ (defclass transient-effective-slot-definition (standard-effective-slot-definition transient-slot-definition) ()) -(defgeneric transient (slot)) - -(defmethod transient ((slot standard-direct-slot-definition)) - t) - -(defmethod transient ((slot persistent-direct-slot-definition)) - nil) +(defgeneric transient (slot) + (:method ((slot standard-direct-slot-definition)) t) + (:method ((slot persistent-direct-slot-definition)) nil)) ;; ;; Indexed slots maintenance @@ -279,16 +276,10 @@ "Ordinary classes may NOT inherit from persistent classes." nil) -(defgeneric persistent-p (class)) - -(defmethod persistent-p ((class t)) - nil) - -(defmethod persistent-p ((class persistent-metaclass)) - t) - -(defmethod persistent-p ((class persistent-slot-definition)) - t) +(defgeneric persistent-p (class) + (:method ((class t)) nil) + (:method ((class persistent-metaclass)) t) + (:method ((class persistent-slot-definition)) t)) (defmethod effective-slot-definition-class ((class persistent-metaclass) &rest initargs) "Chooses the persistent or transient effective slot --- /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/04/04 15:28:29 1.14 +++ /project/elephant/cvsroot/elephant/src/elephant/migrate.lisp 2007/04/12 02:47:33 1.15 @@ -2,7 +2,8 @@ ;;; ;;; migrate.lisp -- Migrate between repositories ;;; -;;; New Version 2/19/2006 by Ian Eslick +;;; Original version 2005 by Robert Read +;;; New version 2/19/2006 by Ian Eslick ;;; ;;; ;;; part of @@ -12,6 +13,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/06 02:51:48 1.29 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/12 02:47:33 1.30 @@ -1,9 +1,8 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; package.lisp -- package definition +;;; migrate.lisp - ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; Initial version by Robert Read ;;; ;;; part of ;;; @@ -12,6 +11,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/pset.lisp 2007/04/06 02:51:48 1.1 +++ /project/elephant/cvsroot/elephant/src/elephant/pset.lisp 2007/04/12 02:47:33 1.2 @@ -1,7 +1,30 @@ +;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- +;;; +;;; pset.lisp - A generic implementation of persistent sets that can be +;;; overridden for a more efficient implementation by a data +;;; store. Uses btrees to emulate simple sets +;;; +;;; By Ian Eslick +;;; +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. +;;; + (in-package :elephant) ;; -;; Simple implementation of persistent sets +;; Default implementation of persistent sets ;; ;; --- /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/04/01 20:56:19 1.2 +++ /project/elephant/cvsroot/elephant/src/elephant/query-example.lisp 2007/04/12 02:47:33 1.3 @@ -1,3 +1,23 @@ +;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- +;;; +;;; query-example.lisp -- Simple example of conjuctive query interface +;;; +;;; By Ian Eslick +;;; +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. +;;; (in-package :elephant) --- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/01 20:56:19 1.3 +++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/12 02:47:33 1.4 @@ -1,12 +1,21 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; query.lisp -- Implement syntax for the elephant query engine +;;; query.lisp -- Implement conjunctive syntax as example for elephant query interface ;;; -;;; Copyright (c) 2007 by Ian S. Eslick -;;; +;;; By Ian S. Eslick, +;;; +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; ;;; ;;; Elephant users are granted the rights to distribute and use this software -;;; as governed by the terms of the Lisp Limited General Public License +;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;; --- /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/04/01 14:33:34 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer.lisp 2007/04/12 02:47:33 1.28 @@ -9,6 +9,12 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/serializer1.lisp 2007/03/30 14:34:35 1.13 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer1.lisp 2007/04/12 02:47:33 1.14 @@ -9,6 +9,12 @@ ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/06 02:51:48 1.37 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/12 02:47:33 1.38 @@ -1,14 +1,20 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; serializer.lisp -- convert Lisp data to/from byte arrays +;;; serializer2.lisp -- convert Lisp data to/from byte arrays ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; By Ian Eslick, +;;; Adapted from serializer1 by Ben Lee ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/02/18 10:58:58 1.10 +++ /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/12 02:47:33 1.11 @@ -2,8 +2,7 @@ ;;; ;;; transactions.lisp -- Top level transaction support for elephant ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; By Ian Eslick ;;; ;;; part of ;;; @@ -12,6 +11,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/elephant/unicode2.lisp 2007/03/18 20:40:50 1.7 +++ /project/elephant/cvsroot/elephant/src/elephant/unicode2.lisp 2007/04/12 02:47:33 1.8 @@ -1,24 +1,32 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; serializer.lisp -- convert Lisp data to/from byte arrays +;;; unicode2.lisp -- binary encoding/decoding for strings with codes > 8 bits ;;; -;;; Initial version 11/11/2006 by Ian Eslick -;;; +;;; By Ian Eslick, ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;; -;;; Here UTF8 means 8-bit < #x79 only for efficiency, UTF16 means UTF16 format -;;; but only for values < #xFFFF. In the odd cases of non-0 unicode planes we just -;;; use UTF-32 to avoid the time cost of translation. Only when converting from a -;;; serialized UTF-X do we worry about encoding (UTF32->UTF16). If an ascii/utf8 lisp -;;; encounters a UTF-16 we know it uses more than 127 values and thus cannot be -;;; represented in the reading lisp so we assert an error. + +;;; +;;; In this file UTF8 means 8-bit < #x79 only for efficiency, UTF16 +;;; means UTF16 format but only for values < #xFFFF. In the odd cases +;;; of non-0 unicode planes we just use UTF-32 to avoid the time cost +;;; of translation. Only when converting from a serialized UTF-X do +;;; we worry about encoding (UTF32->UTF16). If an ascii/utf8 lisp +;;; encounters a UTF-16 we know it uses more than 127 values and thus +;;; cannot be represented in the reading lisp so we assert an error. ;;; (in-package :elephant-serializer2) --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/03/30 23:36:53 1.14 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/12 02:47:33 1.15 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. From ieslick at common-lisp.net Thu Apr 12 02:47:52 2007 From: ieslick at common-lisp.net (ieslick) Date: Wed, 11 Apr 2007 22:47:52 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/utils Message-ID: <20070412024752.3ED0F53066@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/utils In directory clnet:/tmp/cvs-serv28241/src/utils Modified Files: convenience.lisp locks.lisp os.lisp package.lisp Log Message: Mostly documentation edits + fixing copyright in all source files --- /project/elephant/cvsroot/elephant/src/utils/convenience.lisp 2007/03/30 17:45:42 1.6 +++ /project/elephant/cvsroot/elephant/src/utils/convenience.lisp 2007/04/12 02:47:51 1.7 @@ -1,9 +1,23 @@ - -;; Copyright Ian Eslick -;; License: LGPL -;; -;; A collection of handy utilities for compacting code complexity in elephant -;; +;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- +;;; +;;; convenience.lisp -- A set of convenience functions for Elephant +;;; +;;; By Ian Eslick, +;;; +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. +;;; (in-package :elephant-utils) --- /project/elephant/cvsroot/elephant/src/utils/locks.lisp 2007/02/09 17:11:53 1.2 +++ /project/elephant/cvsroot/elephant/src/utils/locks.lisp 2007/04/12 02:47:51 1.3 @@ -1,14 +1,19 @@ ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; -;;; cross-platform.lisp -- convert Lisp data to/from byte arrays +;;; locks.lisp -- A generic interface to lisp/os locks ;;; -;;; Initial version 8/26/2004 by Ben Lee -;;; +;;; By Ian Eslick, ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. --- /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/02/16 03:43:48 1.2 +++ /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/12 02:47:51 1.3 @@ -1,3 +1,24 @@ +;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- +;;; +;;; os.lisp -- A set of convenience functions for cross-platform os functions +;;; that elephant requires +;;; +;;; By Ian Eslick, +;;; +;;; part of +;;; +;;; Elephant: an object-oriented database for Common Lisp +;;; +;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee +;;; +;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; +;;; Elephant users are granted the rights to distribute and use this software +;;; as governed by the terms of the Lisp Lesser GNU Public License +;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. +;;; (in-package :elephant-utils) --- /project/elephant/cvsroot/elephant/src/utils/package.lisp 2007/03/30 17:45:42 1.6 +++ /project/elephant/cvsroot/elephant/src/utils/package.lisp 2007/04/12 02:47:51 1.7 @@ -12,6 +12,9 @@ ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; +;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick +;;; +;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. From ieslick at common-lisp.net Thu Apr 12 17:09:56 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 12 Apr 2007 13:09:56 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/eslick Message-ID: <20070412170956.9010E1124@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/eslick In directory clnet:/tmp/cvs-serv19106/src/contrib/eslick Modified Files: snapshot-db.lisp Log Message: Final hacks for snapshot-set --- /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 02:45:08 1.1 +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 17:09:56 1.2 @@ -38,24 +38,18 @@ (in-package :elephant) -(defparameter *use-proxy-objects* t - "Indicates that the snapshot set should register - and write any standard-objects found in slots registered - of standard objects during snapshots") - (defpclass snapshot-set () ((index :accessor snapshot-set-index :initform (make-btree)) (next-id :accessor snapshot-set-next-id :initform 0) - (cache :accessor snapshot-set-cache :initform (make-hash-table) :transient t) - (root :accessor snapshot-set-root :initform nil)) + (root :accessor snapshot-set-root :initform nil) + (cache :accessor snapshot-set-cache :initform (make-hash-table) :transient t)) (:documentation "Keeps track of a set of standard objects allowing a single snapshot call to update the store controller with the latest state of all objects registered with this set")) -(defmethod initialize-instance :after ((set snapshot-set) &rest rest) - (declare (ignore rest)) - (restore set)) +(defmethod initialize-instance :after ((set snapshot-set) &key lazy-load &allow-other-keys) + (unless lazy-load (restore set))) ;; ================= ;; User methods @@ -63,28 +57,21 @@ (defmethod register-object ((object standard-object) (set snapshot-set)) "Register a standard object. Not recorded until snapshot is called on db" - (if (lookup-cached-id object set) nil - (let ((id (incf (snapshot-set-next-id set)))) - (cache-snapshot-object id object set) - object))) + (aif (lookup-cached-id object set) + (values object it) + (let ((id (incf (snapshot-set-next-id set)))) + (cache-snapshot-object id object set) + (values object id)))) (defmethod register-object ((hash hash-table) (set snapshot-set)) "Adds a hash table to the snapshot set and registers any standard objects stored as values that are not already part of the snapshot. Must call snapshot to save." - (if (lookup-cached-id hash set) nil - (let ((id (incf (snapshot-set-next-id set)))) - (cache-snapshot-object id hash set) - hash))) - -(defmethod set-root ((set snapshot-set)) - (if (snapshot-set-root set) - (lookup-cached-object (snapshot-set-root set) set) - nil)) - -(defmethod (setf set-root) (value (set snapshot-set)) - (setf (snapshot-set-root set) - (ensure-registered value))) + (aif (lookup-cached-id hash set) + (values hash it) + (let ((id (incf (snapshot-set-next-id set)))) + (cache-snapshot-object id hash set) + (values hash id)))) (defmethod unregister-object (object (set snapshot-set)) "Drops the object from the cache and backing store" @@ -94,36 +81,47 @@ (drop-cached-object object set) (delete-snapshot-object id set))) -(defmethod snapshot ((set snapshot-set)) - (maphash (lambda (obj id) - (write-object id obj set)) - (snapshot-set-cache set))) - -(defmethod restore ((set snapshot-set)) - "Restores a snapshot by setting the snapshot-set state to the last snapshot. - If this is used during runtime, the user needs to drop all references - to objects and retrieve again from the snapshot set" - (clear-cache set) - (let ((proxyrecs nil)) - (map-btree (lambda (k v) - (cond ((hash-table-p v) - (push (list k v) proxyrecs)) - ((subtypep (type-of v) 'standard-object) - (cache-snapshot-object k v set)) - (t (error "Invalid type in snapshot-set type ~A for ~A" (type-of v) v)))) - (snapshot-set-index set)) - ;; All objects should be loaded so object references in hashes are valid - (dolist (proxyrec proxyrecs) - (destructuring-bind (id proxy) proxyrec - (cache-snapshot-object id (restore-proxy-hash proxy set) set))))) +(defmethod snapshot-root ((set snapshot-set)) + "Get the snapshot root object" + (when (snapshot-set-root set) + (lookup-cached-object (snapshot-set-root set) set))) + +(defmethod (setf snapshot-root) (value (set snapshot-set)) + "Specify a root object for the set. There is only 1 + so it should be a hash or the root node of a graph" + (setf (snapshot-set-root set) + (multiple-value-bind (obj id) + (register-object value set) + id)) + value) (defun map-set (fn set) - "Iterates through all values in the set" + "Iterates through all values in the active set, not the + saved snapshot" (maphash (lambda (k v) (declare (ignore v)) (funcall fn k)) (snapshot-set-cache set))) +(defmethod snapshot ((set snapshot-set)) + "Saves all objects in the set (and any objects reachable from the + current set of objects) to the persistent store" + (with-transaction (:store-controller (get-con (snapshot-set-index set))) + (maphash (lambda (obj id) + (save-snapshot-object id obj set)) + (snapshot-set-cache set)))) + +(defmethod restore ((set snapshot-set)) + "Restores a snapshot by setting the snapshot-set state to the last snapshot. + If this is used during runtime, the user needs to drop all references + to objects and retrieve again from the snapshot set. Also used to initialize + the set state when a set is created, for example pulled from the root of a + store-controller, unless :lazy-load is specified" + (clear-cache set) + (map-btree (lambda (id object) + (load-snapshot-object id object set)) + (snapshot-set-index set))) + ;; =============== ;; Shorthand ;; =============== @@ -133,6 +131,9 @@ (defun clear-cache (set) (clrhash (snapshot-set-cache set))) +(defun cache-snapshot-object (id obj set) + (setf (gethash obj (snapshot-set-cache set)) id)) + (defun lookup-cached-id (obj set) (gethash obj (snapshot-set-cache set))) @@ -145,77 +146,140 @@ (return-from find-hash-key-by-value k))) hash)) -(defun cache-snapshot-object (id obj set) - (setf (gethash obj (snapshot-set-cache set)) id)) - (defun drop-cached-object (obj set) (remhash obj (snapshot-set-cache set))) -;; Save and restore objects - -(defun read-snapshot-object (id set) - (get-value id (snapshot-set-index set))) +;; Save objects -(defun write-object (id obj set) - (setf (get-value id (snapshot-set-index set)) - (cond ((subtypep (type-of obj) 'standard-object) - (make-proxy-object obj set)) - ((eq (type-of obj) 'hash-table) - (make-proxy-hash obj set)) - (t (error "Cannot only snapshot standard-objects and hash-tables"))))) - -(defun ensure-registered (obj set) - "Return object id by cache lookup or register and write object" - (let ((id (lookup-cached-id obj set))) - (if id id - (progn - (register-object obj set) - (let ((id (lookup-cached-id obj set))) - (write-object id obj set) - id))))) - -(defun delete-snapshot-object (id set) - (remove-kv id (snapshot-set-index set))) +(defclass setref () + ((id :accessor snapshot-set-reference-id :initarg :id))) -;; Snapshot ops +(defun setrefp (obj) + (eq (type-of obj) 'setref)) -(defun reified-class-p (obj) - (or (subtypep (type-of obj) 'standard-object) - (eq (type-of obj) 'hash-table))) +(defun standard-object-subclass-p (obj) + (subtypep (type-of obj) 'standard-object)) -(defclass setref () - ((id :accessor snapshot-set-reference-id :initarg :id))) +(defun save-snapshot-object (id obj set) + (setf (get-value id (snapshot-set-index set)) + (cond ((standard-object-subclass-p obj) + (save-proxy-object obj set)) + ((hash-table-p obj) + (save-proxy-hash obj set)) + (t (error "Cannot only snapshot standard-objects and hash-tables")))) + id) + +(defun save-proxy-object (obj set) + (let ((svs (subsets 2 (slots-and-values obj)))) + (if (some #'reified-class-p (mapcar #'second svs)) + (let ((proxy (make-instance (type-of obj)))) + (loop for (slotname value) in svs do + (setf (slot-value proxy slotname) + (if (reify-class-p value) + (reify-object value set) + value))) + proxy) + obj))) -(defun make-proxy-object (obj set) - (if (not *use-proxy-objects*) - obj - (let ((proxy (make-instance (type-of obj)))) - (loop for (slotname value) in (subsets 2 (slots-and-values obj)) do - (setf (slot-value proxy slotname) - (if (reified-class-p value) - (make-instance 'setref :id (ensure-registered value set)) - value)))))) -(defun make-proxy-hash (hash set) +(defun save-proxy-hash (hash set) (let ((proxy (make-hash-table))) (maphash (lambda (key value) (setf (gethash key proxy) - (if (or (subtypep (type-of value) 'standard-object) - (subtypep (type-of value) 'hash-table)) - (make-instance 'setref :id (ensure-registered value set)) + (if (reify-class-p value) + (reify-object value set) value))) hash) proxy)) -(defun restore-proxy-hash (proxy set) - "Convert a proxy object to a standard hash, resolving references" - (let ((hash (make-hash-table))) - (maphash (lambda (k v) - (setf (gethash k hash) - (if (eq (type-of v) 'setref) - (lookup-cached-object (snapshot-set-reference-id v) set) - v))) - proxy) - hash)) +(defun reify-class-p (obj) + (or (standard-object-subclass-p obj) + (hash-table-p obj))) + +(defun reify-object (obj set) + (multiple-value-bind (obj id) + (register-object obj set) + (make-instance 'setref :id (save-snapshot-object id obj set)))) + +;; Load objects + +(defun load-snapshot-object (id object set) + (let ((object (ifret object (get-value id (snapshot-set-index set))))) + (cond ((standard-object-subclass-p object) + (load-proxy-object id object set)) + ((hash-table-p object) + (load-proxy-hash id object set)) + (t (error "Unrecognized type ~A for id ~A in set ~A" (type-of object) id set))))) + +;; Need to create placeholder, then populate slots + +(defun load-proxy-object (id obj set) + (ifret (lookup-cached-object id set) + (progn + (cache-snapshot-object id obj set) + (let ((svs (subsets 2 (slots-and-values obj)))) + (loop for (slotname value) in svs do + (when (setrefp value) + (setf (slot-value obj slotname) + (load-snapshot-object (snapshot-set-reference-id value) nil set))))) + obj))) + +(defun load-proxy-hash (id hash set) + (ifret (lookup-cached-object id set) + (progn + (cache-snapshot-object id hash set) + (maphash (lambda (key value) + (when (setrefp value) + (setf (gethash key hash) + (load-snapshot-object (snapshot-set-reference-id value) nil set)))) + hash) + hash))) + + +;; Delete from snapshot + +(defun delete-snapshot-object (id set) + (remove-kv id (snapshot-set-index set))) - \ No newline at end of file +;; ============================== +;; Tests +;; ============================== + +(defclass snapshot-test () + ((slot1 :accessor slot1 :initarg :slot1) + (slot2 :accessor slot2 :initarg :slot2))) + +(defun make-stest (slot1 slot2) + (make-instance 'snapshot-test :slot1 slot1 :slot2 slot2)) + +(defun test-snapshot () + "Requires open store" + (let* ((set (make-instance 'snapshot-set)) + (hash (make-hash-table)) + (test1 (make-stest 1 2)) + (test2 (make-stest 10 20)) + (test3 (make-stest (make-stest 'one 'two) (make-stest 'three 'four))) + (test4 (make-stest (slot1 test3) (slot2 test3)))) + (loop for num from 1 + for obj in (list test1 test2 test3 test4) do + (setf (gethash num hash) obj)) + (setf (snapshot-root set) hash) + (add-to-root 'set set) + (snapshot set) + ;; Clear + (setf set nil) + (setf hash nil) + (elephant::flush-instance-cache *store-controller*) + (cl-user::gc) + ;; Reload + (setf set (get-from-root 'set)) + (setf hash (snapshot-root set)) + (let ((t1 (gethash 1 hash)) + (t2 (gethash 2 hash)) + (t3 (gethash 3 hash)) + (t4 (gethash 4 hash))) + (values + (eq 1 (slot1 t1)) + (eq 20 (slot2 t2)) + (eq (slot2 t3) + (slot2 t4)))))) From ieslick at common-lisp.net Thu Apr 12 17:16:59 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 12 Apr 2007 13:16:59 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/eslick Message-ID: <20070412171659.D73971A0A2@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/eslick In directory clnet:/tmp/cvs-serv19536/src/contrib/eslick Modified Files: snapshot-db.lisp Log Message: Error when registering unsupported types --- /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 17:09:56 1.2 +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 17:16:59 1.3 @@ -10,10 +10,10 @@ ;; ;; Limitations: ;; -;; - Hashes can be registered as indexes of objects. Keys -;; should be simple (numbers, strings, symbols) although -;; arrays in an equalp cache are probably OK too. Values -;; should also be simple or subclasses of standard-object +;; - Hashes can be registered as indexes of objects. Keys should be +;; simple (numbers, strings, symbols) although arrays are probably +;; OK too. Values should also be simple or subclasses of +;; standard-object. ;; ;; - When a snapshot is taken of a hash, all values that are ;; standard objects are registered. Any refs to registered @@ -73,6 +73,9 @@ (cache-snapshot-object id hash set) (values hash id)))) +(defmethod register-object ((default t) (set snapshot-set)) + (error "Cannot register objects of type ~A" (type-of default))) + (defmethod unregister-object (object (set snapshot-set)) "Drops the object from the cache and backing store" (let ((id (gethash object (snapshot-set-cache set)))) From rread at common-lisp.net Thu Apr 19 03:20:41 2007 From: rread at common-lisp.net (rread) Date: Wed, 18 Apr 2007 23:20:41 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070419032041.B6B5748144@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv9885 Added Files: ArchFigure.svg Log Message: Adding this figure for reference --- /project/elephant/cvsroot/elephant/doc/ArchFigure.svg 2007/04/19 03:20:41 NONE +++ /project/elephant/cvsroot/elephant/doc/ArchFigure.svg 2007/04/19 03:20:41 1.1 image/svg+xml Elephant Architecture April 2007 Robert L. Read Public Domain English Elephant LISP Robert L. Read This shows my idea of the Elephant architecture. Berkeley-DB Postgres Postmodern LISP Native BDB-API CL-SQL BTree API Persistent CLOS DCM Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv1234 Modified Files: ele-bdb.asd elephant.asd Log Message: Documentation edits; edi's lispworks patch to ele build --- /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/03/24 12:16:02 1.25 +++ /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/04/19 05:24:37 1.26 @@ -31,7 +31,7 @@ #+(or windows mswindows) (defun path-for-cygwin (path) "DOS pathname -> cygwin pathname. Replace backslashes with slashes and drive letter with directory. -e.g. \"C:\\dir\\\" -> \"/C/dir/\" (drive C: must be mounted as /C/ in cgwin)" +e.g. \"C:\\dir\\\" -> \"/cygdrive/C/dir/\" " (let* ((result (namestring path)) (colon-pos (position #\: result)) (drive-letter (char result (1- colon-pos)))) @@ -69,7 +69,7 @@ (defmethod foreign-libraries-to-load-first ((c bdb-c-source)) (remove-if #'(lambda (x) (null (car x))) (list (cons (get-config-option :pthread-lib c) "pthread") - (cons (get-config-option :berkeley-db-lib c) + (cons (get-config-option :berkeley-db-lib c) (get-config-option :berkeley-db-lib c))))) ;; --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/06 02:51:46 1.42 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/19 05:24:37 1.43 @@ -108,7 +108,7 @@ :library nil)))) #+allegro (multiple-value-setq (stdout-lines stderr-lines exit-status) (excl.osi:command-output command :directory directory)) - #+lispworks (setf exit-status (system:call-system command :current-directory directory)) + #+lispworks (setf exit-status (system:call-system-showing-output command :current-directory directory)) (unless (zerop exit-status) (error 'operation-error :component c :operation o))) @@ -118,7 +118,7 @@ (format nil "\"~A\"" (namestring (make-pathname :type "o" :defaults pathname)))))) #+allegro (multiple-value-setq (stdout-lines stderr-lines exit-status) (excl.osi:command-output command :directory directory)) - #+lispworks (setf exit-status (system:call-system command :current-directory directory)) + #+lispworks (setf exit-status (system:call-system-showing-output command :current-directory directory)) (unless (zerop exit-status) (error 'operation-error :component c :operation o))) @@ -133,7 +133,7 @@ :library t)))) #+allegro (multiple-value-setq (stdout-lines stderr-lines exit-status) (excl.osi:command-output command :directory directory)) - #+lispworks (setf exit-status (system:call-system command :current-directory directory)) + #+lispworks (setf exit-status (system:call-system-showing-output command :current-directory directory)) (unless (zerop exit-status) (error 'operation-error :component c :operation o))))) From ieslick at common-lisp.net Thu Apr 19 05:24:37 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 01:24:37 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070419052437.2B2BE1120@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv1234/doc Modified Files: scenarios.texinfo Log Message: Documentation edits; edi's lispworks patch to ele build --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/12 02:47:23 1.4 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/19 05:24:37 1.5 @@ -1,55 +1,292 @@ @c -*-texinfo-*- - at node Usage Scenarios + at node Design Patterns @comment node-name, next, previous, up - at chapter Usage Scenarios - at cindex Usage Scenarios + at chapter Design Patterns + at cindex Design Patterns @menu -* File Replacement:: Simple deployment of Elephant as file replacement +* File System Replacement:: Deployment of Elephant as file replacement +* Checkpointing Program State:: How to recover the application state as recorded in a set of interdependant standard classes for purposes of undo, crash recovery and session persistence. * Persistent System Objects:: Making persistent objects a natural part of your system -* Crash Recovery:: How to recover application state from application or system crashes * Elephant as Database:: Using Elephant as a database for records and user data instead of using a SQL relational Database * Multithreaded Web Applications:: Elephant is a natural match for web applications * Graph-oriented Applications:: Elephant is good, but not optimized, for graph-oriented applications. * Real-World Application Examples:: See some real-world applications Elephant has been used for and a brief discussion of how it was used and any novel uses of Elephant. @end menu - at node File Replacement +This chapter explores different ways that Elephant can be used to +solve common problems in user programs. The term ``Design Pattern'' +may be overkill as there is no formal specification of patterns. +However the goals is similar to classical design patterns: provide a +coherent description of how to approach ceratain common problems using +Elephant as an enabling tool. + +Most of this chapter falls short of a tutorial in the application of a +pattern. Instead it provides a conceptual guide to implementing the +pattern along with some code examples to show how Elephant features +are invoked to support the pattern. + +The authors hope that users of Elephant will find this a good source +of inspiration for how to apply Elephant to their own programs and +that they will be motivated to contribute design patterns of their own. + + + at node File System Replacement @comment node-name, next, previous, up - at section File Replacement + at section File System Replacement -One of the annoying overheads in writing many programs is persisting -data between lisp sessions or invocations of that program. Elements -such as configuration files, raw data such as graphics and other -formats take time, attention and are a potential source of bugs. -Elephant can ease these concerns and allow you to work directly with -your natural in-memory representations with no work to encode/decode -formats or manage files in the file system. +One of the more annoying time-wasting activities in programming is +saving and restoring data from disk. Data in configuration files, +static data such as graphics and other formats take time and attention +away from solving the main problem and are additional sources of bugs. +Because Elephant's serializer supports most lisp types, Elephant can +greatly simplify ease these concerns and allow you to work directly +with your natural in-memory representations with almost no work to +encode/decode formats or manage files in the file system. The simplest way to accomplish this is to simply open a store controller and use the root btree as a key-value store instead of a -file system directory. You might hide some of the complexity sort of -like this: +file system directory. You might hide some of the details like this: @lisp -(defmacro def-resource (name initializer) - (assert (symbolp name)) - `(defparameter name (list nil nil ,initializer))) - -(defun call-initializer (init) - (case init-stmt - (symbol (funcall (symbol-function init-stmt))) - (list (apply (first init) (rest init))))) +(defvar *resources* (make-hash-table)) (defun get-resource (name) - (if (and (symbol-value name) - (symbol-value-name) - (let ((newval (get-from-root name))) - (if newval - (setq name (add-to-root name newval)) - (setq name (add-to-root name (call-initializer - at end lisp + (multiple-value-bind (value foundp) (gethash name *resources*) + (if foundp + value + (multiple-value-bind (value foundp) (get-from-root name) + (if foundp + value + (error "Resource named ~A was not initialized" name)))))) + +(defun set-resource (value name) + (add-to-root name value) + (setf (gethash name *resources*) value)) + +(defsetf get-resource set-resource) + at end lisp + +Another simple metaphor is to use Elephant btrees as persistent hash +tables that persist key-value pairs for you. We'll wrap the Elephant +btree in a simple class to provide a little conceptual isolation. + + at lisp +(defclass phash () + ((btree :accessor phash-btree :initarg :btree :initform (make-btree)))) + +(defun make-persistent-hash (name) + (let ((btree (get-from-root name))) + (if btree + (make-instance 'phash :btree btree) + (let ((phash (make-instance 'phash))) + (add-to-root name (phash-btree phash)) + phash)))) + +(defun getphash (key phash) + (get-value key (phash-btree phash))) + +(defun setphash (value key phash) + (setf (get-value key (phash-btree phash)) value)) + +(defsetf getphash setphash) + at end lisp + +Of course to make a proper abstraction we'd want to provide some +conditions that allowed restarts that initialized values or allowed +users to update the hash in the background and continue computation. + + at footnote{Example provided by Ian Eslick, April 2007} + + at node Checkpointing Program State + at comment node-name, next, previous, up + at section Checkpointing Program State + +Another challenge for many programs is saving some subset of program +state. This could involve checkpointing an evolving computation, +keeping track of state for the purposes of 'undo' or enabling crash +recovery at key points in the program's execution. + +One approach is to transform all our program state into persistent +objects. However if the use of program state is slot-access +intensive, this can have a significant performance impact. To improve +the performance of the application, careful use of transactions is +needed which further complicates program design and operation. + +Can Elephant be used to provide a simple solution that retains the +in-memory performance that we want? Can we do all this without having +to put a ton of persistence assumptions into our main program code? +The answer is yes, assuming you are willing to explicitly checkpoint +your code and adhere to some simple constraints in accessing your +program objects. + + at subsection Assumptions + +To get speed, we want all our objects to be standard lisp objects that +are in memory and have no special harnesses that would interfere with +using the full power of lisp. At some point in execution, we want to +store the current state of a bunch of objects to disk, but make it +easy to reproduce the exact state at a later point in time. For +simplicity, we'll assume that we are talking about collections of CLOS +objects. + +An additional complication is that many programs have sets of +interdependant objects. These could be complex program graphs, the +state of an active search process or a standard OO system that uses a +bunch of program objects to function. This means that we need to +persist not just object state, but also references and any object that +is referred to. + +Using CLOS reflection we can provide a general solution to capturing +objects, slot values and references. However to reproduce references, +we'll need to be able to find the object referenced and the only +general way to do that is to store it as well. Thus a snapshot is a +closed set of self-referential objects. + +The assumptions required to implement the simple checkpointing +implemented here is: + + at itemize + at item @strong{Use standard CLOS objects and references to other CLOS objects.} +We need reflection to + at item @strong{Use standard hash tables to keep track of sets of objects.} +Your program should use the hash table as an entry point to find +objects. When objects are restored, just replace an existing hash +table with the new one and access your objects that way. Any parts of +your program that have pointers into your objects but are not +themselves snapshotted, will need to be able to refresh their pointers +in some way. + at item @strong{Find your root object (s) and know what is ``reachable'' from them.} +Ensure that you aren't referring to standard objects outside those you +want to store as they will be stored too (persistent object references +are fine though). Make sure your root refers to objects that refers +to other objects and so on such that all objects you want to store can +be reached by some set of pointer traversals. Looping references are +fine. + at end itemize + + at subsection Implementation: The Snapshot Set + +To generalize all this behavior, we will define a new class called a +snapshot set. The set itself is a persistent object that wraps a +btree, but provides all the automation to store and recover sets of +objects. + + at subsection Isolating snapshot sets + +A brief note on how to separate out the objects you want to store from +those you don't may be useful. We want to snapshot groups of +inter-referential objects without sucking in the whole system in one +snapshot. These object sets must be closed and fully connected. If +the program consists of a set of subgraphs, a root element of each +graph should be stored in a hash table that is then treated as the +snapshot root. + + at itemize + at item @strong{Manual registration:} +Objects without external references are easy, just @code{register} or + at code{unregister} them from the @code{snapshot-set} as needed and then +map over them to get them back. + at item @strong{Implicit registration:} +Just store objects in a hash that is the root of a @code{snapshot-set} +and you are good to go. + at item @strong{Graphs:} +Graphs are easy to store as they naturally consist of a closed set of +objects. If the graph nodes reference other system objects that you +don't want to store, you'll need to implement something akin to the +indirection provided here. Just store the root of the graph in the +snapshot set root and go from there. + at item @strong{All instances of a type:} +Another easy way to create sets is to overload @code{make-instance} to +store all new objects in a weak hash table that is treated as the root +of a @code{snapshot-set} (NOTE: I have not verified that weak hashes +are properly serialized and reproduced - I suspect they are not so you +might have to copy after a @code{restore}). + at end itemize + +For more complex applications, you can isolate these closed sets of +objects by using @code{snapshot-set} root hash tables as an +indirection mechanism. Instead of storing direct references in an +object slot or hash value, isolation is ensured by storing keys and +indirecting through a hash table to get the target object. This can +be hidden from the programmer in multiple ways. The easiest way is +just to make sure that when you store references you store a key and +overload the slot accessor. A sketch of this follows: + + at lisp +(defparameter *island1-hash* (make-hash-table)) +(defparameter *island2-hash* (make-hash-table)) +(defvar *unique-id* 0) + +(defclass island1-object () + ((pointer-to-island1 :accessor child :initform nil) + (pointer-to-island2 :accessor neighbor :initform nil))) + +(defmethod neighbor :around ((obj island1-object)) + (let ((key (call-next-method))) + (when key (gethash key *island2-hash*)))) + +(defmethod (setf neighbor) :around (ref (obj island1-object)) + (cond ((subtypep (type-of ref) 'island2-object) + (let ((key (find-object ref *island2-hash*))) + (if key + (progn + (call-next-method key obj) + obj) + (progn + (setf (gethash (incf *unique-id*) *island2-hash*) ref) + (call-next-method *unique-id* obj) + obj)))) + (t (call-next-method)))) + +(defun find-object (obj hash) + (map-hash (lambda (k v) + (declare (ignore k)) + (if (eq obj v) + (return-from find-object obj))) + hash)) + at end lisp + +The same template would apply to @code{island2} references to + at code{island1} objects. You could further simplify creating these +hash table indirections with a little macro: + + at lisp +(defmacro def-snapshot-reference-wrapper (accessor-name (source-classname target-classname hashname uid)) + (with-gensysms (obj key ref) + `(progn + (defmethod ,accessorname :around ((,obj ,source-classname)) + (let ((,key (call-next-method))) + (when ,key (gethash ,key ,hashname)))) + (defmethod (setf ,accessorname) :around (,ref (,obj ,source-classname)) + (cond ((subtypep (type-of ,ref) ,target-classname) + (let ((,key (find-object ,ref ,hashname))) + (if ,key + (progn + (call-next-method ,key ,obj) + ,obj) + (progn + (setf (gethash (incf ,uid) ,hashname) ,ref) + (call-next-method ,uid ,obj) + ,obj)))) + (t (call-next-method))))))) + +(defclass island2-object () + ((pointer-to-island2 :accessor child :initform nil) + (pointer-to-island1 :accessor neighbor :initform nil))) + +(def-snapshot-reference-wrapper neighbor (island2 island1 *island1-hash* *unique-id*)) + at end lisp + +Of course this doesn't work for multi-threaded environments, or for +separating more complex collections of types. I am also sure that +more elegant solutions could be generated. + +In most cases, we assume the user will have a natural collection of objects +that is closed + + at footnote{Example provided by Ian Eslick, April 2007} @node Persistent System Objects @comment node-name, next, previous, up @@ -62,10 +299,6 @@ @item Look up objects using class indices @end itemize - at node Crash Recovery - at comment node-name, next, previous, up - at section Crash Recovery - @node Elephant as Database @comment node-name, next, previous, up @section Elephant as Database @@ -135,7 +368,7 @@ @item Process Components @item Bulk storage of post-processed web data @item Class indexes on strings - at item Cheap associations + at item User associations @item Inverted document index @end itemize From ieslick at common-lisp.net Thu Apr 19 05:24:38 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 01:24:38 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070419052438.417741122@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv1234/src/elephant Modified Files: collections.lisp Log Message: Documentation edits; edi's lispworks patch to ele build --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/12 02:47:32 1.21 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/19 05:24:37 1.22 @@ -344,7 +344,8 @@ (etypecase a (number (<= a b)) (string (string<= a b)) - (persistent (<= (oid a) (oid b))))) + (persistent (<= (oid a) (oid b))) + (symbol (string<= (symbol-name a) (symbol-name b))))) (defun lisp-compare-equal (a b) (equal a b)) @@ -381,7 +382,7 @@ (funcall fn k v) (return nil))))))))) -(defgeneric map-index (fn btree &rest args &key start end value) +(defgeneric map-index (fn index &rest args &key start end value) (:documentation "Map-index is like map-btree but for secondary indices, it takes a function of three arguments: key, value and primary key. As with map-btree the keyword arguments start and end From ieslick at common-lisp.net Thu Apr 19 05:51:33 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 01:51:33 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070419055133.53DB47113B@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv4865 Modified Files: config.sexp elephant.asd Log Message: The last of Edi's tweaks --- /project/elephant/cvsroot/elephant/config.sexp 2007/03/24 12:16:02 1.9 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/04/19 05:51:33 1.10 @@ -15,7 +15,7 @@ (:clsql-lib . nil) (:compiler . :gcc)) -#+(and lispworks (not windows)) +#+(and lispworks (not mswindows)) ((:berkeley-db-include-dir . "/opt/local/include/db45/") (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/19 05:24:37 1.43 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/19 05:51:33 1.44 @@ -127,7 +127,7 @@ (compiler-options (c-compiler c) c :input-file (list (format nil "\"~A\"" (namestring - (make-pathname :type "o" :defaults patohname))) + (make-pathname :type "o" :defaults pathname))) "exports.o") :output-file (format nil "\"~A\"" (first (output-files o c))) :library t)))) From ieslick at common-lisp.net Thu Apr 19 22:25:51 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 18:25:51 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070419222551.8F50F1E007@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv25569 Modified Files: TODO Log Message: final snapshot scenario and code changes --- /project/elephant/cvsroot/elephant/TODO 2007/04/12 02:47:22 1.79 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/19 22:25:50 1.80 @@ -14,22 +14,14 @@ Bugs: - Fix awkward serializer API - Support for asdf-install? -- Edi's patches & suggestions for windows + - problem with other .asd files in root Lisp Support: - Validate OpenMCL pre-1.1 on Mac OS X - Validate OpenMCL 1.1 and/or 64-bit on Mac OS X? - Verify db_deadlock for other lisps (launch and kill background program I/F) -Test coverage: -- Clean up interface to tests -- Test for optimize storage method (just add probe-file methods to get file size?) -- Multi-threading stress tests? Ensure that there are conflicts and lots of serialization - happening concurrently to make sure that multi-threading is in good shape (Henrik's code) -- Class / DB sychronization tests - Documentation: -- Proper user guide - Update install, build and test procedures - Upgrade, migration and other system level issues - Performance and design issues @@ -39,10 +31,6 @@ - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) -- Demo file for simple snapshot system - (add standard objects to prevalence DB) - link structure is added to db - in-memory pobjs point to Additional features to document in Trac: - Add error/restart conditions when lose data on: class redef, index drop, etc. From ieslick at common-lisp.net Thu Apr 19 22:25:52 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 18:25:52 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070419222552.E5EE71E003@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv25569/doc Modified Files: scenarios.texinfo Log Message: final snapshot scenario and code changes --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/19 05:24:37 1.5 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/19 22:25:51 1.6 @@ -7,7 +7,7 @@ @menu * File System Replacement:: Deployment of Elephant as file replacement -* Checkpointing Program State:: How to recover the application state as recorded in a set of interdependant standard classes for purposes of undo, crash recovery and session persistence. +* Checkpointing Conventional Program State:: How to recover the application state as recorded in a set of interdependant standard classes for purposes of undo, crash recovery and session persistence. * Persistent System Objects:: Making persistent objects a natural part of your system * Elephant as Database:: Using Elephant as a database for records and user data instead of using a SQL relational Database * Multithreaded Web Applications:: Elephant is a natural match for web applications @@ -99,9 +99,9 @@ @footnote{Example provided by Ian Eslick, April 2007} - at node Checkpointing Program State + at node Checkpointing Conventional Program State @comment node-name, next, previous, up - at section Checkpointing Program State + at section Checkpointing Conventional Program State Another challenge for many programs is saving some subset of program state. This could involve checkpointing an evolving computation, @@ -168,12 +168,368 @@ @subsection Implementation: The Snapshot Set -To generalize all this behavior, we will define a new class called a -snapshot set. The set itself is a persistent object that wraps a -btree, but provides all the automation to store and recover sets of -objects. +In this section we walk through the implementation of the snapshot set +in detail as it provides: + + at itemize + at item Insight into constraints in serialization and lisp object identity + at item How to leverage Elephant for some more sophisticated applications than + persistent indices and class slots. + at item Helps you understand a useful utility (that we may add to an extensions + release in the future) + at end itemize + +To generalize the behavior discussed above, we will define a new +persistent class called a snapshot set. The set itself is a wrapper +around the btree, but provides all the automation to store and recover +sets of standard objects. + + at lisp +(defpclass snapshot-set () + ((index :accessor snapshot-set-index :initform (make-btree)) + (next-id :accessor snapshot-set-next-id :initform 0) + (root :accessor snapshot-set-root :initform nil) + (cache :accessor snapshot-set-cache + :initform (make-hash-table :weak-keys t) + :transient t) + (touched :accessor snapshot-set-touched + :initform (make-array 20 :element-type 'fixnum + :initial-element 0 :fill-pointer t :adjustable t) + :transient t)) + (:documentation "Keeps track of a set of standard objects + allowing a single snapshot call to update the store + controller with the latest state of all objects registered with + this set")) + at end lisp + +The set class keeps track of IDs, a set of cached objects in memory, +the on-disk btree for storing instances by uid and the current uid +variable value. Notice the use of the transient keyword argument for +the cache. + +There are two major operations supported by sets @code{snapshot} and + at code{restore}. These save objects to disk and restore objects to +memory, along with proper recovery of multiple references to the same +object. + +Additional operations are: + + at itemize + at item Registration: Adding and removing objects from a set + at item Root operations: Easy access to a single root hash table or object + at item Mapping: Walk over all objects in a set + at end itemize + +To enable snapshots, we have to register a set of root objects with +the set. This function ignores objects that are already cached, +otherwise allocates a new ID and caches the object. + + at lisp +(defmethod register-object ((object standard-object) (set snapshot-set)) + "Register a standard object. Not recorded until snapshot is called on db" + (aif (lookup-cached-id object set) + (values object it) + (let ((id (incf (snapshot-set-next-id set)))) + (cache-snapshot-object id object set) + (values object id)))) + +(defun lookup-cached-id (obj set) + (gethash obj (snapshot-set-cache set))) + +(defun cache-snapshot-object (id obj set) + (setf (gethash obj (snapshot-set-cache set)) id)) + at end lisp + +A parallel function registers hash tables. One very important +invariant implied here is that the cache always contains objects that +are eq and mapped back to a serialized object in the backing btree. +There is no need, however, to immediately write objects to the store +and this gives us some transactional properties: snapshots are atomic, +consistent and durable. Isolation is not enforced by snapshots. + +This means that the transient cache has to be valid immediately after +the snapshot set is loaded from the data store. + + at lisp +(defmethod initialize-instance :after ((set snapshot-set) &key lazy-load &allow-other-keys) + (unless lazy-load (restore set))) + at end lisp + +This also has consequences for unregistration. Removing a root object +should also result in the removal of all objects that are unreachable +from other roots. However, since side effects are not permanent until +a snapshot operation, we merely have to garbage collect id's that were +not touched during a snapshot operation. This makes unregistration +simple. + + at lisp +(defmethod unregister-object (object (set snapshot-set)) + "Drops the object from the cache and backing store" + (let ((id (gethash object (snapshot-set-cache set)))) + (when (null id) + (error "Object ~A not registered in ~A" object set)) + (drop-cached-object object set))) + at end lisp + +But snapshots are a little bit more work. + + at lisp +(defmethod snapshot ((set snapshot-set)) + "Saves all objects in the set (and any objects reachable from the + current set of objects) to the persistent store" + (with-transaction (:store-controller (get-con (snapshot-set-index set))) + (loop for (obj . id) in (get-cache-entries (snapshot-set-cache set)) do + (save-snapshot-object id obj set)) + (collect-untouched set))) + +(defun save-snapshot-object (id obj set) + (unless (touched id set) + (setf (get-value id (snapshot-set-index set)) + (cond ((standard-object-subclass-p obj) + (save-proxy-object obj set)) + ((hash-table-p obj) + (save-proxy-hash obj set)) + (t (error "Cannot only snapshot standard-objects and hash-tables")))) + (touch id set)) + id) + +(defun collect-untouched (set) + (map-btree (lambda (k v) + (unless (touched k set) + (remove-kv k (snapshot-set-index set)))) + (snapshot-set-index set)) + (clear-touched set)) + at end lisp + +We go through all objects in the cache, storing objects as we go via + at code{save-snapshot-object}. This function is responsible for storing +objects and hash tables and recursing on any instances that are +referenced. Any object that is saved is added to a touch list so they +are not stored again and we can mark stored instances for the + at code{collect-untouched} call which ensures that newly unreachable +objects are deleted from the persistent store. Any newly found +objects are added to the in-memory cache which, being a weak array, +should eventually drop references to objects that are not referred to +elsewhere. + +It should be noted that garbage objects not garbage collected from the +weak-array based cache may be stored to and restored from the +persistent store. However this is merely a storage overhead as they +will eventually be dropped across sessions as there are no saved +references to them. + +Now when we serialize a standard object, all the slot values are +stored inline. This means that by default, a slot that refers to a +standard object would get an immediately serialized version rather +than a reference. This of course makes it impossible to restore +multiple references to a single object. The approach taken here is to +instantiate a @emphasize{proxy} object which is a copy of the original +class and stores references to normal values in its slots. Any +references to hashes or standard classes are replaced with a reference +object that records the unique id of the object so it can be properly +restored. + + at lisp +(defun save-proxy-object (obj set) + (let ((svs (subsets 2 (slots-and-values obj)))) + (if (some #'reified-class-p (mapcar #'second svs)) + (let ((proxy (make-instance (type-of obj)))) + (loop for (slotname value) in svs do + (setf (slot-value proxy slotname) + (if (reify-class-p value) + (reify-value value set) + value))) + proxy) + obj))) + at end lisp + +The function checks whether any slot value can be reified (represented +by a unique id) and if so, makes a new proxy instance and properly +instantiates its slots, returning it to the main store function which +writes the proxy object to the btree. + +On restore, we simply load all objects into memory. + + at lisp +(defmethod restore ((set snapshot-set)) + "Restores a snapshot by setting the snapshot-set state to the last snapshot. + If this is used during runtime, the user needs to drop all references + to objects and retrieve again from the snapshot set. Also used to initialize + the set state when a set is created, for example pulled from the root of a + store-controller, unless :lazy-load is specified" + (clear-cache set) + (map-btree (lambda (id object) + (load-snapshot-object id object set)) + (snapshot-set-index set))) + +(defun load-snapshot-object (id object set) + (let ((object (ifret object (get-value id (snapshot-set-index set))))) + (cond ((standard-object-subclass-p object) + (load-proxy-object id object set)) + ((hash-table-p object) + (load-proxy-hash id object set)) + (t (error "Unrecognized type ~A for id ~A in set ~A" (type-of object) id set))))) + at end lisp + +If an object has a reference object in a slot, then we simply restore +that object as well. @code{load-snapshot-object} accepts null for an +object so it can be used recursively when a reference object refers to +an object (via the unique id) that is not yet cached. The @code{load} +functions return an object so that they can used directly to create +values for writing slots or hash entries. + + at lisp +(defun load-proxy-object (id obj set) + (ifret (lookup-cached-object id set) + (progn + (cache-snapshot-object id obj set) + (let ((svs (subsets 2 (slots-and-values obj)))) + (loop for (slotname value) in svs do + (when (setrefp value) + (setf (slot-value obj slotname) + (load-snapshot-object (snapshot-set-reference-id value) nil set))))) + obj))) + at end lisp + +A full set of source code for @code{snapshot-sets} can be found in the +Elephant source tree under @code{src/conrib/eslick/snapshot-set.lisp}. + + at subsection Using Snapshot Sets + +A snapshot set is quite easy to use. Load the complete code and play +with this simple walk through. First we need to create a set object, + + at lisp +(setf my-set (make-instance 'snapshot-set)) + at end lisp + +and add it to the root so we don't lose track of it. - at subsection Isolating snapshot sets + at lisp +(add-to-root 'my-set my-set) + at end lisp + +Then we need some objects to play with. + + at lisp +(defclass my-test-class () + ((value :accessor test-value :initarg :value) + (reference :accessor test-reference :initarg :reference))) + +(setf obj1 (make-instance 'my-test-class :value 1 :reference nil)) +(setf obj2 (make-instance 'my-test-class :value 2 :reference obj1)) +(setf obj3 (make-instance 'my-test-class :value 3 :reference obj2)) + +(register-object obj3 my-set) +(snapshot my-set) + at end lisp + +Now your set should have persistent versions of all three classes that +are reachable from @code{obj3}. + + at lisp +(map-set (lambda (x) (print (test-value x))) my-set) +=> +3 +2 +1 + at end lisp + +Of course such fully connected objects are not always common, so we'll +demonstrate using hash tables to create root indexes into our objects +and sidestep registration calls entirely. We'll create a fresh set to +work with. + + at lisp +(setf my-set (make-instance 'snapshot-set)) +(add-to-root 'my-set my-set) + +(setf obj4 (make-instance 'my-test-class :value 4 :reference obj1)) +(setf obj5 (make-instance 'my-test-class :value 5 :reference nil)) + +(setf hash (make-hash-table)) +(setf (snapshot-root my-set) hash) + +(setf (gethash 'obj3 hash) obj3) +(setf (gethash 'obj4 hash) obj4) +(setf (gethash 'obj5 hash) obj5) + +(snapshot my-set) + at end lisp + +To properly simulate restoring objects, we need to drop our old hash +table as well as clear the persistent object cache so the snapshot set +transient object is reset. + + at lisp +(setf my-set nil) +(setf hash nil) +(elephant::flush-instance-cache *store-controller*) + at end lisp + +Now we'll pretend we're startup up a new session. + + at lisp +(setf my-set (get-from-root 'my-set)) +(setf hash (snapshot-root my-set)) + at end lisp + +The cache is automatically populated by the implicit @code{restore} +call during snapshot-set initialization, and our hash table should now +have all the proper references. We'll pull out a few. + + at lisp +(setf o4 (gethash 'obj4 hash)) +(setf o3 (gethash 'obj3 hash)) +(setf o2 (test-reference o3)) + +(not (or (eq o4 obj4) + (eq o3 obj3) + (eq o2 obj2))) +=> t + at end lisp + +The new objects should not be eq the old ones as we have restored +fresh copies from the disk. + +If you review the setup above, @code{obj3} references @code{obj2} +which references @code{obj1} and @code{obj4} also references + at code{obj1}. So if the objects were properly restored, these +references should be @code{eq}. + + at lisp +(eq (test-reference o2) (test-reference o4)) +=> t + at end lisp + +And finally we can demonstrate the restorative power of snapshot sets. + + at lisp +(remhash 'obj5 hash) + +(gethash 'obj5 hash) +=> nil nil + +(restore my-set) +(setf hash (snapshot-root my-set)) + +(gethash 'obj5 hash) +=> # t + +(test-value *) +=> 5 + at lisp + +This means that while our set object was not reset, the restore +operation properly restored the old reference structure of our root +hash object. Unfortunately, in this implementation you have to reset +your lisp pointers to get access to the restored objects. + +A future version could traverse the existing object cache, dropping +new references and restoring old ones so that in-memory lisp pointers +were still valid. + + at subsection Isolating multiple snapshot sets A brief note on how to separate out the objects you want to store from those you don't may be useful. We want to snapshot groups of @@ -281,10 +637,9 @@ Of course this doesn't work for multi-threaded environments, or for [11 lines skipped] From ieslick at common-lisp.net Thu Apr 19 22:25:53 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 18:25:53 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/eslick Message-ID: <20070419222553.BA8012817F@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/eslick In directory clnet:/tmp/cvs-serv25569/src/contrib/eslick Modified Files: snapshot-db.lisp Log Message: final snapshot scenario and code changes --- /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/12 17:16:59 1.3 +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-db.lisp 2007/04/19 22:25:52 1.4 @@ -35,6 +35,17 @@ ;; to slots are saved to a persistent list that gets reused after ;; snapshots (id slotname value). Slot reads are as usual. ;; +;; - Avoid stack use during recursions. Push new objects onto a stack +;; for later processing so stack depth is constant. +;; +;; - In place restores. A future version could traverse the existing +;; object cache, dropping new references and restoring old ones +;; according to the state of the snapshot-set on disk such that the +;; existing in-memory lisp pointers were still valid..as long as there +;; were not external pointers into objects that are dropped leading to +;; an inconsistency. +;; + (in-package :elephant) @@ -42,7 +53,10 @@ ((index :accessor snapshot-set-index :initform (make-btree)) (next-id :accessor snapshot-set-next-id :initform 0) (root :accessor snapshot-set-root :initform nil) - (cache :accessor snapshot-set-cache :initform (make-hash-table) :transient t)) + (cache :accessor snapshot-set-cache :initform (make-hash-table :weak-keys t) :transient t) + (touched :accessor snapshot-set-touched + :initform (make-array 20 :element-type 'fixnum :initial-element 0 :fill-pointer t :adjustable t) + :transient t)) (:documentation "Keeps track of a set of standard objects allowing a single snapshot call to update the store controller with the latest state of all objects registered with @@ -95,6 +109,7 @@ (setf (snapshot-set-root set) (multiple-value-bind (obj id) (register-object value set) + (declare (ignore obj)) id)) value) @@ -110,9 +125,10 @@ "Saves all objects in the set (and any objects reachable from the current set of objects) to the persistent store" (with-transaction (:store-controller (get-con (snapshot-set-index set))) - (maphash (lambda (obj id) - (save-snapshot-object id obj set)) - (snapshot-set-cache set)))) + (loop for (obj . id) in (get-cache-entries (snapshot-set-cache set)) do + (save-snapshot-object id obj set)) + (collect-untouched set)) + (values set t)) (defmethod restore ((set snapshot-set)) "Restores a snapshot by setting the snapshot-set state to the last snapshot. @@ -123,7 +139,8 @@ (clear-cache set) (map-btree (lambda (id object) (load-snapshot-object id object set)) - (snapshot-set-index set))) + (snapshot-set-index set)) + (values set t)) ;; =============== ;; Shorthand @@ -152,6 +169,13 @@ (defun drop-cached-object (obj set) (remhash obj (snapshot-set-cache set))) +(defun get-cache-entries (hash) + (let ((result nil)) + (maphash (lambda (obj id) + (push (cons obj id) result)) + hash) + result)) + ;; Save objects (defclass setref () @@ -163,34 +187,45 @@ (defun standard-object-subclass-p (obj) (subtypep (type-of obj) 'standard-object)) +(defun touch (id set) + (vector-push-extend id (snapshot-set-touched set) 50)) + +(defun touched (id set) + (find id (snapshot-set-touched set))) + +(defun clear-touched (set) + (loop for i fixnum from 0 upto (1- (length (snapshot-set-touched set))) do + (setf (aref (snapshot-set-touched set) i) 0))) + (defun save-snapshot-object (id obj set) - (setf (get-value id (snapshot-set-index set)) - (cond ((standard-object-subclass-p obj) - (save-proxy-object obj set)) - ((hash-table-p obj) - (save-proxy-hash obj set)) - (t (error "Cannot only snapshot standard-objects and hash-tables")))) + (unless (touched id set) + (setf (get-value id (snapshot-set-index set)) + (cond ((standard-object-subclass-p obj) + (save-proxy-object obj set)) + ((hash-table-p obj) + (save-proxy-hash obj set)) + (t (error "Cannot only snapshot standard-objects and hash-tables")))) + (touch id set)) id) (defun save-proxy-object (obj set) (let ((svs (subsets 2 (slots-and-values obj)))) - (if (some #'reified-class-p (mapcar #'second svs)) + (if (some #'reify-class-p (mapcar #'second svs)) (let ((proxy (make-instance (type-of obj)))) (loop for (slotname value) in svs do (setf (slot-value proxy slotname) (if (reify-class-p value) - (reify-object value set) + (reify-value value set) value))) proxy) obj))) - (defun save-proxy-hash (hash set) (let ((proxy (make-hash-table))) (maphash (lambda (key value) (setf (gethash key proxy) (if (reify-class-p value) - (reify-object value set) + (reify-value value set) value))) hash) proxy)) @@ -199,11 +234,19 @@ (or (standard-object-subclass-p obj) (hash-table-p obj))) -(defun reify-object (obj set) +(defun reify-value (obj set) (multiple-value-bind (obj id) (register-object obj set) (make-instance 'setref :id (save-snapshot-object id obj set)))) +(defun collect-untouched (set) + (map-btree (lambda (k v) + (declare (ignore v)) + (unless (touched k set) + (remove-kv k (snapshot-set-index set)))) + (snapshot-set-index set)) + (clear-touched set)) + ;; Load objects (defun load-snapshot-object (id object set) @@ -273,7 +316,8 @@ (setf set nil) (setf hash nil) (elephant::flush-instance-cache *store-controller*) - (cl-user::gc) + #+allegro (excl:gc) + #+sbcl (cl-user::gc) ;; Reload (setf set (get-from-root 'set)) (setf hash (snapshot-root set)) From ieslick at common-lisp.net Thu Apr 19 22:26:45 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 19 Apr 2007 18:26:45 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/eslick Message-ID: <20070419222645.399CF4B025@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/eslick In directory clnet:/tmp/cvs-serv25813 Added Files: snapshot-set.lisp Removed Files: snapshot-db.lisp Log Message: Change filename --- /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-set.lisp 2007/04/19 22:26:45 NONE +++ /project/elephant/cvsroot/elephant/src/contrib/eslick/snapshot-set.lisp 2007/04/19 22:26:45 1.1 ;;; ;;; Copyright (c) 2007 Ian Eslick ;;; ;;; Simple snapshot sets. Create a snapshot set for standard objects, ;;; work in-memory and then call snapshot to save the objects to ;;; the underlying store-controller. ;;; ;; ;; Limitations: ;; ;; - Hashes can be registered as indexes of objects. Keys should be ;; simple (numbers, strings, symbols) although arrays are probably ;; OK too. Values should also be simple or subclasses of ;; standard-object. ;; ;; - When a snapshot is taken of a hash, all values that are ;; standard objects are registered. Any refs to registered ;; objects are properly restored on retrieval ;; ;; Easy extensions: ;; ;; - Support arrays of objects as well as hash-tables ;; ;; - Create a method standard-object-dirty-p that defaults to 't' but ;; allows users to implement a method that avoids storing unchanged ;; objects. ;; ;; - Enable versioned or named snapshots ;; ;; - Better interface API (special object metaclass?) to create a more ;; natural abstraction. Could also add object journaling for ;; prevalence dynamics via rucksack log model. For example, writes ;; to slots are saved to a persistent list that gets reused after ;; snapshots (id slotname value). Slot reads are as usual. ;; ;; - Avoid stack use during recursions. Push new objects onto a stack ;; for later processing so stack depth is constant. ;; ;; - In place restores. A future version could traverse the existing ;; object cache, dropping new references and restoring old ones ;; according to the state of the snapshot-set on disk such that the ;; existing in-memory lisp pointers were still valid..as long as there ;; were not external pointers into objects that are dropped leading to ;; an inconsistency. ;; (in-package :elephant) (defpclass snapshot-set () ((index :accessor snapshot-set-index :initform (make-btree)) (next-id :accessor snapshot-set-next-id :initform 0) (root :accessor snapshot-set-root :initform nil) (cache :accessor snapshot-set-cache :initform (make-hash-table :weak-keys t) :transient t) (touched :accessor snapshot-set-touched :initform (make-array 20 :element-type 'fixnum :initial-element 0 :fill-pointer t :adjustable t) :transient t)) (:documentation "Keeps track of a set of standard objects allowing a single snapshot call to update the store controller with the latest state of all objects registered with this set")) (defmethod initialize-instance :after ((set snapshot-set) &key lazy-load &allow-other-keys) (unless lazy-load (restore set))) ;; ================= ;; User methods ;; ================= (defmethod register-object ((object standard-object) (set snapshot-set)) "Register a standard object. Not recorded until snapshot is called on db" (aif (lookup-cached-id object set) (values object it) (let ((id (incf (snapshot-set-next-id set)))) (cache-snapshot-object id object set) (values object id)))) (defmethod register-object ((hash hash-table) (set snapshot-set)) "Adds a hash table to the snapshot set and registers any standard objects stored as values that are not already part of the snapshot. Must call snapshot to save." (aif (lookup-cached-id hash set) (values hash it) (let ((id (incf (snapshot-set-next-id set)))) (cache-snapshot-object id hash set) (values hash id)))) (defmethod register-object ((default t) (set snapshot-set)) (error "Cannot register objects of type ~A" (type-of default))) (defmethod unregister-object (object (set snapshot-set)) "Drops the object from the cache and backing store" (let ((id (gethash object (snapshot-set-cache set)))) (when (null id) (error "Object ~A not registered in ~A" object set)) (drop-cached-object object set) (delete-snapshot-object id set))) (defmethod snapshot-root ((set snapshot-set)) "Get the snapshot root object" (when (snapshot-set-root set) (lookup-cached-object (snapshot-set-root set) set))) (defmethod (setf snapshot-root) (value (set snapshot-set)) "Specify a root object for the set. There is only 1 so it should be a hash or the root node of a graph" (setf (snapshot-set-root set) (multiple-value-bind (obj id) (register-object value set) (declare (ignore obj)) id)) value) (defun map-set (fn set) "Iterates through all values in the active set, not the saved snapshot" (maphash (lambda (k v) (declare (ignore v)) (funcall fn k)) (snapshot-set-cache set))) (defmethod snapshot ((set snapshot-set)) "Saves all objects in the set (and any objects reachable from the current set of objects) to the persistent store" (with-transaction (:store-controller (get-con (snapshot-set-index set))) (loop for (obj . id) in (get-cache-entries (snapshot-set-cache set)) do (save-snapshot-object id obj set)) (collect-untouched set)) (values set t)) (defmethod restore ((set snapshot-set)) "Restores a snapshot by setting the snapshot-set state to the last snapshot. If this is used during runtime, the user needs to drop all references to objects and retrieve again from the snapshot set. Also used to initialize the set state when a set is created, for example pulled from the root of a store-controller, unless :lazy-load is specified" (clear-cache set) (map-btree (lambda (id object) (load-snapshot-object id object set)) (snapshot-set-index set)) (values set t)) ;; =============== ;; Shorthand ;; =============== ;; Cache ops (defun clear-cache (set) (clrhash (snapshot-set-cache set))) (defun cache-snapshot-object (id obj set) (setf (gethash obj (snapshot-set-cache set)) id)) (defun lookup-cached-id (obj set) (gethash obj (snapshot-set-cache set))) (defun lookup-cached-object (id set) (find-hash-key-by-value id (snapshot-set-cache set))) (defun find-hash-key-by-value (value hash) (maphash (lambda (k v) (when (eq v value) (return-from find-hash-key-by-value k))) hash)) (defun drop-cached-object (obj set) (remhash obj (snapshot-set-cache set))) (defun get-cache-entries (hash) (let ((result nil)) (maphash (lambda (obj id) (push (cons obj id) result)) hash) result)) ;; Save objects (defclass setref () ((id :accessor snapshot-set-reference-id :initarg :id))) (defun setrefp (obj) (eq (type-of obj) 'setref)) (defun standard-object-subclass-p (obj) (subtypep (type-of obj) 'standard-object)) (defun touch (id set) (vector-push-extend id (snapshot-set-touched set) 50)) (defun touched (id set) (find id (snapshot-set-touched set))) (defun clear-touched (set) (loop for i fixnum from 0 upto (1- (length (snapshot-set-touched set))) do (setf (aref (snapshot-set-touched set) i) 0))) (defun save-snapshot-object (id obj set) (unless (touched id set) (setf (get-value id (snapshot-set-index set)) (cond ((standard-object-subclass-p obj) (save-proxy-object obj set)) ((hash-table-p obj) (save-proxy-hash obj set)) (t (error "Cannot only snapshot standard-objects and hash-tables")))) (touch id set)) id) (defun save-proxy-object (obj set) (let ((svs (subsets 2 (slots-and-values obj)))) (if (some #'reify-class-p (mapcar #'second svs)) (let ((proxy (make-instance (type-of obj)))) (loop for (slotname value) in svs do (setf (slot-value proxy slotname) (if (reify-class-p value) (reify-value value set) value))) proxy) obj))) (defun save-proxy-hash (hash set) (let ((proxy (make-hash-table))) (maphash (lambda (key value) (setf (gethash key proxy) (if (reify-class-p value) (reify-value value set) value))) hash) proxy)) (defun reify-class-p (obj) (or (standard-object-subclass-p obj) (hash-table-p obj))) (defun reify-value (obj set) (multiple-value-bind (obj id) (register-object obj set) (make-instance 'setref :id (save-snapshot-object id obj set)))) (defun collect-untouched (set) (map-btree (lambda (k v) (declare (ignore v)) (unless (touched k set) (remove-kv k (snapshot-set-index set)))) (snapshot-set-index set)) (clear-touched set)) ;; Load objects (defun load-snapshot-object (id object set) (let ((object (ifret object (get-value id (snapshot-set-index set))))) (cond ((standard-object-subclass-p object) (load-proxy-object id object set)) ((hash-table-p object) (load-proxy-hash id object set)) (t (error "Unrecognized type ~A for id ~A in set ~A" (type-of object) id set))))) ;; Need to create placeholder, then populate slots (defun load-proxy-object (id obj set) (ifret (lookup-cached-object id set) (progn (cache-snapshot-object id obj set) (let ((svs (subsets 2 (slots-and-values obj)))) (loop for (slotname value) in svs do (when (setrefp value) (setf (slot-value obj slotname) (load-snapshot-object (snapshot-set-reference-id value) nil set))))) obj))) (defun load-proxy-hash (id hash set) (ifret (lookup-cached-object id set) (progn (cache-snapshot-object id hash set) (maphash (lambda (key value) (when (setrefp value) (setf (gethash key hash) (load-snapshot-object (snapshot-set-reference-id value) nil set)))) hash) hash))) ;; Delete from snapshot (defun delete-snapshot-object (id set) (remove-kv id (snapshot-set-index set))) ;; ============================== ;; Tests ;; ============================== (defclass snapshot-test () ((slot1 :accessor slot1 :initarg :slot1) (slot2 :accessor slot2 :initarg :slot2))) (defun make-stest (slot1 slot2) (make-instance 'snapshot-test :slot1 slot1 :slot2 slot2)) (defun test-snapshot () "Requires open store" (let* ((set (make-instance 'snapshot-set)) (hash (make-hash-table)) (test1 (make-stest 1 2)) (test2 (make-stest 10 20)) (test3 (make-stest (make-stest 'one 'two) (make-stest 'three 'four))) (test4 (make-stest (slot1 test3) (slot2 test3)))) (loop for num from 1 for obj in (list test1 test2 test3 test4) do (setf (gethash num hash) obj)) (setf (snapshot-root set) hash) (add-to-root 'set set) (snapshot set) ;; Clear (setf set nil) (setf hash nil) (elephant::flush-instance-cache *store-controller*) #+allegro (excl:gc) #+sbcl (cl-user::gc) ;; Reload (setf set (get-from-root 'set)) (setf hash (snapshot-root set)) (let ((t1 (gethash 1 hash)) (t2 (gethash 2 hash)) (t3 (gethash 3 hash)) (t4 (gethash 4 hash))) (values (eq 1 (slot1 t1)) (eq 20 (slot2 t2)) (eq (slot2 t3) (slot2 t4)))))) From ieslick at common-lisp.net Sat Apr 21 17:22:35 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:22:35 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070421172235.9D8BB30D3@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv3920 Modified Files: elephant-tests.asd elephant.asd Log Message: Check for cross-store loading errors in multi-store operation; more documentation; backend language to data store language --- /project/elephant/cvsroot/elephant/elephant-tests.asd 2006/11/11 18:41:10 1.7 +++ /project/elephant/cvsroot/elephant/elephant-tests.asd 2007/04/21 17:22:35 1.8 @@ -55,6 +55,7 @@ :components ((:file "elephant-tests") (:file "testserializer") + (:file "testconditions") (:file "mop-tests") (:file "testcollections") (:file "testindexing") --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/19 05:51:33 1.44 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/21 17:22:35 1.45 @@ -306,7 +306,7 @@ (:file "unicode2") (:file "migrate") (:file "query") - (:file "backend")) + (:file "data-store-api")) :serial t :depends-on (memutil utils))))) :serial t From ieslick at common-lisp.net Sat Apr 21 17:22:36 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:22:36 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070421172236.A2F904142@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv3920/doc Modified Files: elephant-design.texinfo elephant.texinfo installation.texinfo make-ref.lisp reference.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Check for cross-store loading errors in multi-store operation; more documentation; backend language to data store language --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/21 17:22:35 1.4 @@ -1,8 +1,8 @@ @c -*-texinfo-*- - at node Elephant Design + at node Elephant Architecture @comment node-name, next, previous, up - at chapter Elephant Design + at chapter Elephant Architecture @cindex design Elephant's early architecture was tightly coupled to the Berkeley DB --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/12 02:47:23 1.8 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/21 17:22:35 1.9 @@ -20,8 +20,8 @@ @titlepage @title Elephant User Manual - at subtitle Elephant version 0.6.1 - at author Ben Lee and Ian Eslick + at subtitle Elephant version 0.9 + at author Ian Eslick and Ben Lee @c The following two commands @c start the copyright page. @@ -50,9 +50,9 @@ * Tutorial:: A basic ``getting started'' tutorial. * Installation:: Installation and test-suite procedures. * User Guide:: In depth discussion of all Elephant facilities and features. -* Usage Scenarios:: Design scenarios for Elephant applications. * User API Reference:: Function and class documentation of the user API. -* Elephant Design:: An overview of elephant's internal architecture. +* Design Patterns:: Design scenarios for Elephant applications. +* Elephant Architecture:: An overview of elephant's internal architecture. * Data Store API Reference:: Function level documentation for data store implementors. * Copyright and License:: Your rights and freedoms. @end menu @@ -78,8 +78,8 @@ @include tutorial.texinfo @include installation.texinfo @include user-guide.texinfo - at include scenarios.texinfo @include reference.texinfo + at include scenarios.texinfo @include elephant-design.texinfo @include data-store-reference.texinfo @include copying.texinfo --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/12 02:47:23 1.9 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/21 17:22:35 1.10 @@ -9,9 +9,9 @@ * Requirements:: Supported lisps and required libraries. * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. -* Berkeley DB Data Store:: Installing support for the Berkeley DB data store +* Berkeley DB:: Installing support for the Berkeley DB data store * Berkeley DB Example:: An example of installing and running the Berkeley DB data store. -* CL-SQL Data Store:: Install and connecting to the CL-SQL data store +* CL-SQL:: Install and connecting to the CL-SQL data store * CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite @@ -88,8 +88,8 @@ of you with asdf-install on your system) In addition to these libraries, each data store has their own -dependencies as discussed in @ref{Berkeley DB Data Store} and - at ref{CL-SQL Data Store}. +dependencies as discussed in @ref{Berkeley DB} and + at ref{CL-SQL}. @node Configuring Elephant @comment node-name, next, previous, up @@ -224,9 +224,9 @@ database and return the @code{store-controller} subclass instance for that backend. - at node Berkeley DB Data Store + at node Berkeley DB @comment node-name, next, previous, up - at section Berkeley DB Data Store + at section Berkeley DB The Berkeley DB Data Store started out as a very simple data dictionary in the Berkeley Unix operating system. There are many ``Xdb'' systems that use the @@ -275,9 +275,9 @@ in your application. - at node CL-SQL Data Store + at node CL-SQL @comment node-name, next, previous, up - at section CL-SQL Data Store + at section CL-SQL Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/12 02:47:23 1.8 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/21 17:22:35 1.9 @@ -1,5 +1,7 @@ (require 'asdf) (asdf:operate 'asdf:load-op 'elephant :force t) +(in-package :elephant) + (load (merge-pathnames #p"src/elephant/query" (asdf:component-pathname (asdf:find-system 'elephant)))) @@ -19,8 +21,6 @@ (sb-posix:chdir include-dir-path) (load docstrings-path) -(in-package :elephant) - (defclass simple-store-controller (store-controller) ()) --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/12 02:47:23 1.10 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/21 17:22:35 1.11 @@ -8,8 +8,8 @@ @menu * Store Controllers:: Connecting to a data store. -* Persistent Class:: Defining persistent classes and creating and manipulating persistent indices. -* Class Indexing:: Convenient indexing of persistent classes. +* Persistent Objects:: Defining persistent classes and creating and manipulating persistent indices. +* Persistent Object Indexing:: Convenient indexing of persistent classes. * Persistent Sets:: Maintaining persistent collections the easy way. @c * Query Interfaces:: Finding instances. * BTrees:: BTrees and indices, a low level persistent data structure. --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/19 22:25:51 1.6 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/21 17:22:35 1.7 @@ -11,7 +11,7 @@ * Persistent System Objects:: Making persistent objects a natural part of your system * Elephant as Database:: Using Elephant as a database for records and user data instead of using a SQL relational Database * Multithreaded Web Applications:: Elephant is a natural match for web applications -* Graph-oriented Applications:: Elephant is good, but not optimized, for graph-oriented applications. + at c * Graph-oriented Applications:: Elephant is good, but not optimized, for graph-oriented applications. * Real-World Application Examples:: See some real-world applications Elephant has been used for and a brief discussion of how it was used and any novel uses of Elephant. @end menu @@ -32,6 +32,92 @@ that they will be motivated to contribute design patterns of their own. + at node Persistent System Objects + at comment node-name, next, previous, up + at section Persistent System Objects + +The simplest design pattern supported by Elephant is the use of +persistent objects in the place of standard objects. Typically you +can just modify the old class definition to inherit the + at code{persistent-metaclass}. Depending on your application, objects +may need to have transient slots for performance reasons. We'll +create a dummy class to illustrate: + + at lisp +(defclass system-object () + ((appname :accessor system-appname :initarg :name) + (url :accessor system-url :initarg :url) + (state :accessor system-state :initarg :state :initform 'idle)) + (:metaclass persistent-metaclass)) + at end lisp + +When starting up your application you need to recover references to +any persistent objects that were created in a prior session or +initialize a new one. + +If you are storing system objects in parameters, you can just call +an initialization function on startup: + + at lisp +(defparameter *system* nil) + +(defun initialize-system (appname) + (let ((system-object (get-from-root '*system*))) + (setf *system + (if system-object system-object + (make-instance 'system-object :name appname))))) + +*system* +=> # + at end lisp + +And now you can use your parameter as you did before. If you want to +avoid calling initialization functions, you can just accesss system +objects through functions instead of parameters. + + at lisp +(defparameter *system* nil) + +(defun sys-object () + (unless *system + (let ((appname (get-application-name)) + (url (get-system-url))) + (setf *system* (make-instance 'system-object :name appname :url url)))) + *system*) + +(sys-object) +=> # + at end lisp + +One constraint to keep in mind is that slot access will be slower as +it has to synchronize to disk. This is usually not noticable for +objects that are accessed on the order of seconds instead of +milliseconds. For objects read constantly, but where you want to save +any written values it helps to have a transient slot to cache values. +You can override some methods to ensure that the persistent value is +always updated, but that reads happen from the cached value and that +the cached value is restored whenever the object is loaded. + + at lisp +(defclass system-object () + ((appname :accessor system-appname :initarg :name) + (url :accessor system-url :initarg :url) + (laststate :accessor system-laststate :initarg :state :initform 'idle) + (state :accessor system-state :initarg :state :transient t) + (:metaclass persistent-metaclass)) + +(defmethod (setf system-state) :after (state (sys system-state)) + (setf (system-laststate sys) state)) + +(defmethod initialize-instance :after ((sys system-state) &rest rest) + (declare (ignore rest)) + (when (slot-boundp sys 'laststate) + (setf (system-state sys) (system-laststate sys)))) + at end lisp + +And now you have an instant read cache for a slot value. This pattern +is used several times within the Elephant implementation. + @node File System Replacement @comment node-name, next, previous, up @section File System Replacement @@ -43,11 +129,14 @@ Because Elephant's serializer supports most lisp types, Elephant can greatly simplify ease these concerns and allow you to work directly with your natural in-memory representations with almost no work to -encode/decode formats or manage files in the file system. +encode/decode formats or manage files in the file +system at footnote{Example provided by Ian Eslick, April 2007}. The simplest way to accomplish this is to simply open a store -controller and use the root btree as a key-value store instead of a -file system directory. You might hide some of the details like this: +controller and initialize a key-value pair in the root btree as a +instead of a filename and file data in some system directory. Like +the initialization process described for standard objects, you can +hide some of the details like this: @lisp (defvar *resources* (make-hash-table)) @@ -97,7 +186,7 @@ conditions that allowed restarts that initialized values or allowed users to update the hash in the background and continue computation. - at footnote{Example provided by Ian Eslick, April 2007} + @node Checkpointing Conventional Program State @comment node-name, next, previous, up @@ -123,29 +212,27 @@ @subsection Assumptions -To get speed, we want all our objects to be standard lisp objects that -are in memory and have no special harnesses that would interfere with -using the full power of lisp. At some point in execution, we want to -store the current state of a bunch of objects to disk, but make it -easy to reproduce the exact state at a later point in time. For -simplicity, we'll assume that we are talking about collections of CLOS -objects. - -An additional complication is that many programs have sets of -interdependant objects. These could be complex program graphs, the -state of an active search process or a standard OO system that uses a -bunch of program objects to function. This means that we need to -persist not just object state, but also references and any object that -is referred to. +To maintain processing speed and convenience we would like all our +objects to be standard lisp objects without special harnesses that +would interfere with applying the full power of lisp. At some point +during execution, we want to store the current state of a set of +objects to disk and yet make it easy to reproduce the original state +at a later point in time. For simplicity, we'll limit ourselves +to collections of CLOS objects. + +A complication is that many programs have sets of interdependant +objects. These could be complex program graphs, the state of an +ongoing search process or a standard OO system that uses a bunch of +different program object types to run. This means that we need to +persist not just object state, but also references to other objects. Using CLOS reflection we can provide a general solution to capturing objects, slot values and references. However to reproduce references, -we'll need to be able to find the object referenced and the only -general way to do that is to store it as well. Thus a snapshot is a -closed set of self-referential objects. +we'll need to be able to find the object referenced and the only way +to do that is to store it as well. Thus we want to create a snapshot +of a closed set of self-referential objects. -The assumptions required to implement the simple checkpointing -implemented here is: +The assumptions underlying the snapshot mechanism is: @itemize @item @strong{Use standard CLOS objects and references to other CLOS objects.} @@ -166,7 +253,149 @@ fine. @end itemize - at subsection Implementation: The Snapshot Set + at subsection Snapshot Set + +The snapshot implementation is called a @code{snapshot-set}. The next +section will go into detail, but a walkthrough will help make it +clearer at footnote{Example provided by Ian Eslick, April 2007}. + +A snapshot set is quite easy to use. Load the complete code and play +with this simple walk through. The code can be located in the +Elephant source tree under @code{src/conrib/eslick/snapshot-set.lisp}. + +The first step is to create a @code{snapshot-set} object, + + at lisp +(setf my-set (make-instance 'snapshot-set)) + at end lisp + +and add it to the root so we don't lose track of it. + + at lisp +(add-to-root 'my-set my-set) + at end lisp + +Then we need some objects to play with. + + at lisp +(defclass my-test-class () + ((value :accessor test-value :initarg :value) + (reference :accessor test-reference :initarg :reference))) + +(setf obj1 (make-instance 'my-test-class :value 1 :reference nil)) +(setf obj2 (make-instance 'my-test-class :value 2 :reference obj1)) +(setf obj3 (make-instance 'my-test-class :value 3 :reference obj2)) + +(register-object obj3 my-set) +(snapshot my-set) + at end lisp + +Now your set should have persistent versions of all three classes that +are reachable from @code{obj3}. + + at lisp +(map-set (lambda (x) (print (test-value x))) my-set) +=> +3 +2 +1 + at end lisp + +Of course such fully connected objects are not always common, so we'll +demonstrate using hash tables to create root indexes into our objects +and sidestep registration calls entirely. We'll create a fresh set to +work with. + + at lisp +(setf my-set (make-instance 'snapshot-set)) +(add-to-root 'my-set my-set) + +(setf obj4 (make-instance 'my-test-class :value 4 :reference obj1)) +(setf obj5 (make-instance 'my-test-class :value 5 :reference nil)) + +(setf hash (make-hash-table)) +(setf (snapshot-root my-set) hash) + +(setf (gethash 'obj3 hash) obj3) +(setf (gethash 'obj4 hash) obj4) +(setf (gethash 'obj5 hash) obj5) + +(snapshot my-set) + at end lisp + +To properly simulate restoring objects, we need to drop our old hash +table as well as clear the persistent object cache so the snapshot set +transient object is reset. + + at lisp +(setf my-set nil) +(setf hash nil) +(elephant::flush-instance-cache *store-controller*) + at end lisp + +Now we'll pretend we're startup up a new session. + + at lisp +(setf my-set (get-from-root 'my-set)) +(setf hash (snapshot-root my-set)) + at end lisp + +The cache is automatically populated by the implicit @code{restore} +call during snapshot-set initialization, and our hash table should now +have all the proper references. We'll pull out a few. + + at lisp +(setf o4 (gethash 'obj4 hash)) +(setf o3 (gethash 'obj3 hash)) +(setf o2 (test-reference o3)) + +(not (or (eq o4 obj4) + (eq o3 obj3) + (eq o2 obj2))) +=> t + at end lisp + +The new objects should not be eq the old ones as we have restored +fresh copies from the disk. + +If you review the setup above, @code{obj3} references @code{obj2} +which references @code{obj1} and @code{obj4} also references + at code{obj1}. So if the objects were properly restored, these +references should be @code{eq}. + + at lisp +(eq (test-reference o2) (test-reference o4)) +=> t + at end lisp + +And finally we can demonstrate the restorative power of snapshot sets. + + at lisp +(remhash 'obj5 hash) + +(gethash 'obj5 hash) +=> nil nil + +(restore my-set) +(setf hash (snapshot-root my-set)) + +(gethash 'obj5 hash) +=> # t + +(test-value *) +=> 5 + at end lisp + +This means that while our set object was not reset, the restore +operation properly restored the old reference structure of our root +hash object. Unfortunately, in this implementation you have to reset +your lisp pointers to get access to the restored objects. + +A future version could traverse the existing object cache, dropping +new references and restoring old ones so that in-memory lisp pointers +were still valid. + + at subsection Snapshot Set Implementation In this section we walk through the implementation of the snapshot set in detail as it provides: @@ -323,7 +552,7 @@ standard object would get an immediately serialized version rather than a reference. This of course makes it impossible to restore multiple references to a single object. The approach taken here is to -instantiate a @emphasize{proxy} object which is a copy of the original +instantiate a @emph{proxy} object which is a copy of the original class and stores references to normal values in its slots. Any references to hashes or standard classes are replaced with a reference object that records the unique id of the object so it can be properly @@ -391,144 +620,6 @@ obj))) @end lisp -A full set of source code for @code{snapshot-sets} can be found in the -Elephant source tree under @code{src/conrib/eslick/snapshot-set.lisp}. - - at subsection Using Snapshot Sets - -A snapshot set is quite easy to use. Load the complete code and play -with this simple walk through. First we need to create a set object, - - at lisp -(setf my-set (make-instance 'snapshot-set)) - at end lisp - -and add it to the root so we don't lose track of it. - - at lisp -(add-to-root 'my-set my-set) - at end lisp - -Then we need some objects to play with. - - at lisp -(defclass my-test-class () - ((value :accessor test-value :initarg :value) - (reference :accessor test-reference :initarg :reference))) - -(setf obj1 (make-instance 'my-test-class :value 1 :reference nil)) -(setf obj2 (make-instance 'my-test-class :value 2 :reference obj1)) -(setf obj3 (make-instance 'my-test-class :value 3 :reference obj2)) - -(register-object obj3 my-set) -(snapshot my-set) - at end lisp - -Now your set should have persistent versions of all three classes that -are reachable from @code{obj3}. - - at lisp -(map-set (lambda (x) (print (test-value x))) my-set) -=> -3 -2 -1 - at end lisp - -Of course such fully connected objects are not always common, so we'll -demonstrate using hash tables to create root indexes into our objects -and sidestep registration calls entirely. We'll create a fresh set to -work with. - - at lisp -(setf my-set (make-instance 'snapshot-set)) -(add-to-root 'my-set my-set) - -(setf obj4 (make-instance 'my-test-class :value 4 :reference obj1)) -(setf obj5 (make-instance 'my-test-class :value 5 :reference nil)) - -(setf hash (make-hash-table)) -(setf (snapshot-root my-set) hash) [286 lines skipped] --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/12 02:47:23 1.16 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/21 17:22:35 1.17 @@ -1048,7 +1048,7 @@ more than two levels of transactional accesses with the top using with-transaction and the bottom using ensure-transaction. -See @ref{Transaction Details} for more details and @ref{Usage Scenarios} +See @ref{Transaction Details} for more details and @ref{Design Patterns} for examples of how systems can be designed and tuned using transactions. @node Advanced Topics @@ -1114,4 +1114,4 @@ lisps. @end itemize -Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. +Further, @pxref{Design Patterns} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/12 02:47:23 1.11 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/21 17:22:35 1.12 @@ -12,8 +12,8 @@ * Class Indices:: In-depth discussion about indexing persistent indices. @c * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. -* Secondary Indices:: Alternative ways to index collections. * Using Cursors:: Low-level access to BTrees. +* BTree Indices:: Alternative ways to index collections. * Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. * Multi-threaded Applications:: What considerations are required for safe multi-threading @@ -23,6 +23,8 @@ * Performance Tuning:: How to get the most from Elephant. * Berkeley DB Data Store:: Commands and concerns specific to the :BDB data store * CL-SQL Data Store:: Commands and concerns specific to the :CLSQL data store +* Postmodern Data Store:: +* Native Lisp Data Store:: @end menu @node The Store Controller @@ -280,7 +282,7 @@ to other objects can copy every reachable object! This is a direct and dire consequence of the ``store-by-value'' restriction. (@pxref{Persistent Classes and Objects} for how to design around the -store-by-value restriction}). +store-by-value restriction). This list describes how aggregates are handled by the serializer. @@ -332,19 +334,26 @@ @section Persistent Classes and Objects Persistent classes are instances of the @code{persistent-metaclass} -metaclass. All instances of this persistent class have a unique ID -and a pointer to the @code{store-controller} specification they are -associated with. Accesses to slot values become direct reads and -writes to the data store and are thus always persisted. When the -instance itself is writtent to the database, for example as a key or -value in a @code{btree}, only the unique ID is stored. - -Thus serialization of persistent objects is exceedingly cheap compared -to standard objects, but slot access can be much more expensive. -Persistent objects are excellent at, for example, storing the link -structure of a graph. A graph of persistent objects can itself be -persisted by storing the head node of a fully-connected graph to the -store controller root. +metaclass. All persistent classes keep track of which slots are + at code{:persistent}, @code{:transient} and/or @code{:indexed} and are +used as specializers in the persistence meta-object protocols +(initialization of slots, slot-access, etc). + +All persistent classes create objects that inherit from the + at code{persistent} class. The persistent class has two slots that +contain a unique object identifier (oid) and a reference to the + at code{store-controller} specification they are associated with. +Persistent slots do not take up any storage space in memory, instead +the slot access protocol is changed into calls to the configured data +store via the store controller. Typically, data stores then perform +reads and writes to disk. + +When a reference to a @code{persistent} instance itself is written to +the database, for example as a key or value in a @code{btree}, only +the unique ID and class of the instance is stored. When read, a +persistent object instance is re-created (see below). This means that +serialization of persistent objects is exceedingly cheap compared to +standard objects, but slot access time can be much more expensive. @subsection Persistent Class Definition @@ -352,10 +361,12 @@ @code{persistent-metaclass} to the class initarg @code{:metaclass}. The only differences between standard and persistent class definitions is the specification of a slot storage policy and an index policy. +The @code{defpclass} macro is provided as a convenience to hide the + at code{:metaclass} slot option. -Storage policies are specified by a boolean argument to the slot -initargs @code{:persistent} or @code{:transient}. Slots are - at code{:persistent} by default +Slot value storage policies are specified by a boolean argument to the +new slot initargs @code{:persistent}, @code{:transient} and + at code{indexed}. Slots are @code{:persistent} by default. @lisp (defclass my-pclass () @@ -365,15 +376,23 @@ (:metaclass persistent-metaclass)) @end lisp +In the definition above, @code{pslot1} and @code{pslot2} are +persistent while @code{tslot1} is transient and stored in memory. + Slot storage class implications are straightforward. Persistent slot writes are durably stored to disk. Transient slots are initialized on -instance creation according to initforms or to initargs. Transient -slot values are never stored to nor loaded from the database. +instance creation according to initforms or initargs. Transient slot +values are never stored to nor loaded from the database. -During a given lisp session transient values will be cached as long as -they are not collected by the Lisps GC. After GC, if you retrieve an -object from the store its transient slots will be reset to the slot -initforms from the class definition. +During a given lisp session transient values will be cached regardless +of how often you retrieve an object from the store as long as there is +a reference to it stored in memory or tere are zero references but the +object has not yet been collected by the lisp garbage collector. +After collection, if you retrieve an object from the store its +transient slots will be reset to the slot initforms from the class +definition. You can only use @code{:initargs} to initialize transient +or persistent slots during the initial call to @code{make-instance} or +by manually creating the instance from an oid (see below). @lisp (setf pobj1 (make-instance 'my-pclass :pslot1 1 :tslot3 3)) @@ -403,27 +422,31 @@ @end lisp The implications of this behavior is that you need to think carefully -about using transient values. Primarily you cannot make assumptions -about the state of transient values in objects loaded from the store -unless you know that they were loaded and cannot be GC'ed. - -Finally, the index policy tells the data store whether to maintain an -inverted index that maps slot values back to their parent objects. -The :index options and behaviors of persistent classes are discussed in -depth in @ref{Class Indices}. +about how to use transient values. Essentially you cannot make +assumptions about the state of transient values in objects loaded from +the store unless you know that they were loaded and cannot be GC'ed. + +The @code{:index} option tells Elephant whether to maintain an +inverted index that maps slot values to their parent objects. The +behavior of indexed classes and class slots are discussed in depth in + at ref{Class Indices}. @subsection Instance Creation -Persistent objects are instances of the persistent classes defined -above. All persistent objects inherit from the class - at code{persistent} and share two properties: a unique ID and a -reference to the specification of the @code{store-controller} in which -they reside. This is ensured by the instance creation protocol -implemented by @code{persistent-metaclass}. +To the user, persistent objects are created as usual with a call to + at code{make-instance} with the desired slot initargs as illustrated in +the last section. However, this call will fail unless there is a +default @code{store-controller} instance in the variable + at code{*store-controller*} or the @code{:sc} keyword argument is +provided with a valid store controller object. This tells the +metaobject protocol implementing persistent slot creation where to +allocate storage for this instance's slots and -Instances are created as normal, with a call to make-instance and -appropriate initargs. +* Default store controller & instance creation +* What happens to persistent objects when store-controller is closed? +The best policy for initializing transient slots is an @code{:after} +method on @code{initialize-instance}. The two properties of @code{persistent} can be specified explicitly during instance creation: @@ -440,12 +463,60 @@ precedence over values in the database, which take precedences over any @code{initforms} defined in the class. -* Default store controller & instance creation -* What happens to persistent objects when store-controller is closed? +A good policy is to initialize transient values using an @code{:after} +method on @code{initialize-instance} to initialize transient values +using system defaults or from persistent slot values. That way +whether the slots are valid when the object is pulled from + +Also currently there is a bug where @code{initforms} are always +evaluated, so beware. (What is the current model here?) + + at subsection Class Redefinition + +This section discusses the appropriate model a user should employ in +thinking about the behavior of persistent object when their clases are +redefined. + +- What happens when you redefine a class online? +- Drop & add slots? Change slot status? +- What if you connect to an old database with a new class specification? + (ref to class indicies behavior) + + at subsection Synchronizing Code and Database + + + at subsection Persistent objects and the MOP +This section provides increased detail on the use of the MOP to create +and support persistent objects. +new slot types +- transient-slot +- persistent-slot -:: User-defined persistent objects +slot instantiation protocols +- effective-slot-definition +- direct-slot-definition-class +- compute-effective-slot-definition +- compute-effective-slot-definition-initargs + +class initialization protocols +- shared-initialize :around (class) +- finalize-inheritance :around (class) +- reinitialize-instance :around (class) + +instance initialization protocols +- initialize-instance :before (instance) +- shared-initialize :around +- update-instance-for-redefinied-class +- update-instance-for-different-class + +slot access protocol +- slot-value-using-class (setf) +- slot-boundp-using-class +- slot-makunbound-using-clas + + at subsection Overriding methods on persistent objects * slot types * caching @@ -458,20 +529,6 @@ the specification to work properly with persistent slots. However the proper behavior has been verified on SBCL, Allegro and Lispworks. -:: Initialization - -Also currently there is a bug where @code{initforms} are always -evaluated, so beware. (What is the current model here?) - -:: Class Redefinition and Evolution - -* What happens when you redefine a class online? -* Drop & add slots? Change slot status? -* What if you connect to an old database with a new class specification? - (ref to class indicies behavior) - -:: Storage and Performance Considerations - @node Class Indices @comment node-name, next, previous, up @section Class Indices @@ -642,15 +699,15 @@ blocks of data is relatively inexpensive after a seek and comparisons on objects that are stored in memory is cheap. - at node Secondary Indices + at node Using Cursors @comment node-name, next, previous, up - at section Secondary Indices + at section Using Cursors Empty. - at node Using Cursors + at node BTree Indicies @comment node-name, next, previous, up - at section Using Cursors + at section BTree Indicies Empty. @@ -665,7 +722,6 @@ throwing a condition when @code{with-transaction} forms are nested dynamically. - ;; Transaction architecture: ;; ;; User and designer considerations: @@ -847,28 +903,20 @@ @code{*inhibit-slot-writes*} in your user method using @code{with-inhibited-slot-copy} a convenience macro. - at node Garbage Collection - at comment node-name, next, previous, up - at section Garbage Collection - -GC is not implemented, but migration (@pxref{Repository Migration and -Upgrade}) will consolidate storage and recover OIDs which emulates GC. -No online solution is currently supported. - @node Performance Tuning @comment node-name, next, previous, up - at section Performance + at section Performance Tuning Performance is usually measured in transactions per second. Database reads are cheap. To get more transactions throughput, consider setting @lisp -* (db-env-set-flags (controller-environment *store-controller*) 1 - :txn-nosync t) +(db-env-set-flags (controller-environment *store-controller*) + 1 :txn-nosync t) @end lisp -or look at other flags in the sleepycat docs. This will greatly +or look at other flags in the Berkeley DB docs. This will greatly increase your throughput at the cost of some durability; I get around a 100x improvement. Durability can be recovered with judicious use of checkpointing and replication, though this is currently not supported @@ -906,13 +954,29 @@ mode is not suitable for use in web servers or other typically multi-threaded applications. + at node Garbage Collection + at comment node-name, next, previous, up + at section Garbage Collection + +GC is not implemented, but migration (@pxref{Repository Migration and +Upgrade}) will consolidate storage and recover OIDs which emulates GC. +No online solution is currently supported. + + @node Berkeley DB Data Store @comment node-name, next, previous, up @section Berkeley DB Data Store - @node CL-SQL Data Store @comment node-name, next, previous, up @section CL-SQL Data Store + + at node Postmodern Data Store + at comment node-name, next, previous, up + at section Postmodern Data Store + + at node Native Lisp Data Store + at comment node-name, next, previous, up + at section Native Lisp Data Store From ieslick at common-lisp.net Sat Apr 21 17:22:39 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:22:39 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-bdb Message-ID: <20070421172239.6C0A6140B7@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv3920/src/db-bdb Modified Files: bdb-controller.lisp package.lisp Log Message: Check for cross-store loading errors in multi-store operation; more documentation; backend language to data store language --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/04/12 02:47:23 1.33 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/04/21 17:22:36 1.34 @@ -43,7 +43,7 @@ et cetera.")) ;; -;; Backend Registry Support +;; Data store Registry Support ;; (defun bdb-test-and-construct (spec) @@ -52,7 +52,7 @@ (error (format nil "uninterpretable spec specifier: ~A" spec)))) (eval-when (:compile-toplevel :load-toplevel) - (register-backend-con-init :bdb 'bdb-test-and-construct)) + (register-data-store-con-init :bdb 'bdb-test-and-construct)) (defun bdb-store-spec-p (spec) (and (eq (first spec) :bdb) @@ -192,7 +192,7 @@ ;; ;; Store the database version ;; -;; For BDB this can be in a file; different backends may require a different approach. +;; For BDB this can be in a file; different data stores may require a different approach. (defmethod database-version ((sc bdb-store-controller)) "Elephant protocol to provide the version tag or nil if unmarked" @@ -282,7 +282,7 @@ (defmethod optimize-layout ((ctrl bdb-store-controller) &key start-key stop-key (freelist-only t) (free-space nil) &allow-other-keys) - "Tell the backend to optimize and reclaim storage between key values" + "Tell the data store to optimize and reclaim storage between key values" (with-buffer-streams (start stop end) (if (null start-key) (progn --- /project/elephant/cvsroot/elephant/src/db-bdb/package.lisp 2007/04/12 02:47:23 1.9 +++ /project/elephant/cvsroot/elephant/src/db-bdb/package.lisp 2007/04/21 17:22:37 1.10 @@ -33,7 +33,7 @@ Elephant, but with some magic for Elephant. In general there is a 1-1 mapping from functions here and functions in Berkeley DB, so refer to their documentation for details.") - (:use common-lisp uffi elephant-memutil elephant elephant-backend elephant-utils) + (:use common-lisp uffi elephant-memutil elephant elephant-data-store elephant-utils) #+cmu (:use alien) #+sbcl From ieslick at common-lisp.net Sat Apr 21 17:22:50 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:22:50 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-clsql Message-ID: <20070421172250.9307F3200B@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv3920/src/db-clsql Modified Files: package.lisp sql-controller.lisp Log Message: Check for cross-store loading errors in multi-store operation; more documentation; backend language to data store language --- /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/12 02:47:24 1.4 +++ /project/elephant/cvsroot/elephant/src/db-clsql/package.lisp 2007/04/21 17:22:39 1.5 @@ -20,7 +20,7 @@ (defpackage db-clsql (:use :common-lisp :uffi :cl-base64 - :elephant :elephant-memutil :elephant-backend + :elephant :elephant-memutil :elephant-data-store :elephant-utils #+sbcl :sb-thread )) --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-controller.lisp 2007/04/12 02:47:24 1.24 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-controller.lisp 2007/04/21 17:22:40 1.25 @@ -88,10 +88,10 @@ (eval-when (:compile-toplevel :load-toplevel) - (register-backend-con-init :clsql 'sql-test-and-construct)) + (register-data-store-con-init :clsql 'sql-test-and-construct)) (defun sql-test-and-construct (spec) - "Entry function for making SQL backend controllers" + "Entry function for making SQL data store controllers" (if (sql-store-spec-p spec) (make-instance 'sql-store-controller :spec (if spec spec @@ -167,7 +167,7 @@ ;; ISE NOTE: Much of the index management functionality is common between ;; bdb and sql - we could lift this along with indices and indices-cache ;; up to the main elephant code base and introduce a new update-index -;; generic function to handle the backend specific method for updating +;; generic function to handle the data store specific method for updating (defmethod map-indices (fn (bt sql-indexed-btree)) (maphash fn (indices-cache bt))) From ieslick at common-lisp.net Sat Apr 21 17:23:02 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:23:02 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070421172302.68F0E431BE@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv3920/src/elephant Modified Files: classes.lisp controller.lisp package.lisp serializer2.lisp Added Files: data-store-api.lisp Removed Files: backend.lisp Log Message: Check for cross-store loading errors in multi-store operation; more documentation; backend language to data store language --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/12 02:47:31 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/21 17:22:50 1.28 @@ -252,6 +252,30 @@ (indexed-slot-makunbound class instance slot-def) (persistent-slot-makunbound (get-con instance) instance (slot-definition-name slot-def)))) +;; =================================== +;; Multi-store error checking +;; =================================== + +(defun valid-persistent-reference-p (object sc) + "Ensures that object can be written as a reference into store sc" + (eq (dbcn-spc-pst object) (controller-spec sc))) + +(define-condition cross-store-reference () + ((object :accessor cross-store-reference-object :initarg :object) + (home-controller :accessor cross-store-reference-home-controller :initarg :home-ctrl) + (foreign-controller :accessor cross-store-reference-foreign-controller :initarg :foreign-ctrl)) + (:documentation "An error condition raised when an object is being written into a data store other + than its home store")) + +(defun raise-cross-store-condition (object sc) + (cerror "Proceed and patch later" + 'cross-store-reference + :format-control "Attempted to write object ~A with home store ~A into store ~A" + :format-arguments (list object (get-con object) sc) + :object object + :home-ctrl (get-con object) + :foreign-ctrl sc)) + ;; ====================================================== ;; Handling metaclass overrides of normal slot operation ;; ====================================================== --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/12 02:47:32 1.46 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/21 17:22:50 1.47 @@ -26,26 +26,26 @@ ;; TRACKING OBJECT STORES ;; -(defvar *elephant-backends* +(defvar *elephant-data-stores* '((:bdb (:ele-bdb)) (:clsql (:ele-clsql)) ) "Tells the main elephant code the tag used in a store spec to - refer to a given backend. The second argument is an asdf - dependency list. Entries have the form of (backend-type + refer to a given data store. The second argument is an asdf + dependency list. Entries have the form of (data store type asdf-depends-list") (defvar *elephant-controller-init* (make-hash-table)) -(defun register-backend-con-init (name controller-init-fn) - "Backends must call this function during the +(defun register-data-store-con-init (name controller-init-fn) + "Data stores must call this function during the loading/compilation process to register their initialization - function for the tag name in *elephant-backends*. The + function for the tag name in *elephant-data-stores*. The initialization function returns a fresh instance of the - backends store-controller subclass" + data stores store-controller subclass" (setf (gethash name *elephant-controller-init*) controller-init-fn)) -(defun lookup-backend-con-init (name) +(defun lookup-data-store-con-init (name) (gethash name *elephant-controller-init*)) (defvar *dbconnection-spec* (make-hash-table :test 'equal)) @@ -86,19 +86,19 @@ (defun build-controller (spec) "Actually construct the controller & load dependencies" (assert (and (consp spec) (symbolp (first spec)))) - (load-backend (first spec)) - (let ((init (lookup-backend-con-init (first spec)))) - (unless init (error "Store controller init function not registered for backend ~A." (car spec))) + (load-data-store (first spec)) + (let ((init (lookup-data-store-con-init (first spec)))) + (unless init (error "Store controller init function not registered for data store ~A." (car spec))) (let ((sc (funcall (symbol-function init) spec))) (ele-with-lock (*dbconnection-lock*) (setf (gethash spec *dbconnection-spec*) sc)) sc))) -(defun load-backend (type) +(defun load-data-store (type) (assert (find-package :asdf)) - (let ((record (assoc type *elephant-backends*))) + (let ((record (assoc type *elephant-data-stores*))) (when (or (null record) (not (consp record))) - (error "Unknown backend type ~A, cannot load" type)) + (error "Unknown data store type ~A, cannot load" type)) (satisfy-asdf-dependencies (second record)))) (defun satisfy-asdf-dependencies (dep-list) @@ -115,7 +115,7 @@ (defun get-user-configuration-parameter (name) "This function pulls a value from the key-value pairs stored in - my-config.sexp so backends can have their own pairs for appropriate + my-config.sexp so data stores can have their own pairs for appropriate customization after loading." (elephant-system::get-config-option name @@ -129,24 +129,24 @@ ((spec :type list :accessor controller-spec :initarg :spec - :documentation "Backend initialization functions are + :documentation "Data store initialization functions are expected to initialize :spec on the call to make-instance") ;; Generic support for the object, indexing and root protocols (root :reader controller-root - :documentation "This is an instance of the backend + :documentation "This is an instance of the data store persistent btree. It should have an OID that is fixed in the code and does not change between sessions. Usually it this is something like 0, 1 or -1") (class-root :reader controller-class-root :documentation "This is another root for class indexing that is - also a backend specific persistent btree instance + also a data store specific persistent btree instance with a unique OID that persists between sessions.") (instance-cache :accessor instance-cache :initform (make-cache-table :test 'eql) :documentation "This is an instance cache and part of the - metaclass protocol. Backends should not + metaclass protocol. Data stores should not override the default behavior.") (instance-cache-lock :accessor instance-cache-lock :initform (ele-make-lock) :documentation "Protection for updates to @@ -157,12 +157,12 @@ :documentation "Governs the default behavior regarding which serializer version the current elephant core is - using. Backends can override by creating + using. Data stores can override by creating a method on initialize-serializer.") (serialize :accessor controller-serialize :initform nil :documentation "Accessed by elephant::serialize to get the entry point to the default serializer or to - a backend-specific serializer") + a data store specific serializer") (deserialize :accessor controller-deserialize :initform nil :documentation "Contains the entry point for the specific serializer to be called by @@ -175,6 +175,9 @@ the superclass and subclasses. See slot documentation for details.")) +(defmethod print-object ((sc store-controller) stream) + (format stream "#<~A ~A>" (type-of sc) (second (controller-spec sc)))) + ;; ;; Per-controller instance caching ;; @@ -208,8 +211,8 @@ ;; (defgeneric database-version (sc) - (:documentation "Backends implement this to store the serializer version. - The protocol requires that backends report their database + (:documentation "Data stores implement this to store the serializer version. + The protocol requires that data stores report their database version. On new database creation, the database is written with the *elephant-code-version* so that is returned by database-version. If a legacy database does not have a version according to the method @@ -280,9 +283,9 @@ ;; (defmethod initialize-serializer ((sc store-controller)) - "Establish serializer version on controller startup. Backends call this before + "Establish serializer version on controller startup. Data stores call this before they need the serializer to be valid and after they enable their database-version - call. If the backend shadows this, it has to keep track of serializer versions + call. If the data store shadows this, it has to keep track of serializer versions associated with the database version that is opened." (cond ((prior-version-p (database-version sc) '(0 6 0)) (setf (controller-serializer-version sc) 1) @@ -367,19 +370,19 @@ ;; ================================================================================ ;; -;; BACKEND STORE CONTROLLER PROTOCOL +;; DATA STORE CONTROLLER PROTOCOL ;; ;; ================================================================================ (defgeneric open-controller (sc &key recover recover-fatal thread &allow-other-keys) (:documentation "Opens the underlying environment and all the necessary -database tables. Different backends may use different keys so +database tables. Different data stores may use different keys so all methods should &allow-other-keys. There are three standard keywords: :recover, :recover-fatal and :thread. Recover means that recovery should be checked for or performed on startup. Recover fatal means a full rebuild from log files is requested. -Thread merely indicates to the backend that it is a threaded +Thread merely indicates to the data store that it is a threaded application and any steps that need to be taken (for example transaction implementation) are taken. :thread is usually true.")) @@ -407,7 +410,7 @@ (defgeneric optimize-layout (sc &key &allow-other-keys) (:documentation "If supported, speed up the index and allocation by freeing up any available storage and return it to the free list. See the - methods of backends to determine what options are valid. Supported + methods of data stores to determine what options are valid. Supported both on stores (all btrees and persistent slots) and specific btrees")) ;; @@ -416,19 +419,19 @@ (defgeneric persistent-slot-reader (sc instance name) (:documentation - "Backend specific slot reader function")) + "Data store specific slot reader function")) (defgeneric persistent-slot-writer (sc new-value instance name) (:documentation - "Backend specific slot writer function")) + "Data store specific slot writer function")) (defgeneric persistent-slot-boundp (sc instance name) (:documentation - "Backend specific slot bound test function")) + "Data store specific slot bound test function")) (defgeneric persistent-slot-makunbound (sc instance name) (:documentation - "Backend specific slot makunbound handler")) + "Data store specific slot makunbound handler")) ;; ================================================================================ @@ -439,7 +442,7 @@ ;; -;; Opening and closing backend stores +;; Opening and closing data stores ;; (defun open-store (spec &rest args) --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/12 02:47:33 1.30 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/21 17:22:50 1.31 @@ -265,6 +265,9 @@ #:struct-constructor + ;; Various error conditions + #:cross-store-reference + #:map-class-query #:get-query-instances ) --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/12 02:47:33 1.38 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/21 17:22:51 1.39 @@ -40,7 +40,9 @@ array-type-from-byte byte-from-array-type database-version - translate-and-intern-symbol)) + translate-and-intern-symbol + valid-persistent-reference-p + raise-cross-store-condition)) (in-package :elephant-serializer2) @@ -198,6 +200,8 @@ (string (serialize-string frob bs)) (persistent + (unless (valid-persistent-reference-p frob sc) + (raise-cross-store-condition frob sc)) (buffer-write-byte +persistent+ bs) (buffer-write-int32 (oid frob) bs) ;; This circumlocution is necessitated by --- /project/elephant/cvsroot/elephant/src/elephant/data-store-api.lisp 2007/04/21 17:23:02 NONE +++ /project/elephant/cvsroot/elephant/src/elephant/data-store-api.lisp 2007/04/21 17:23:02 1.1 ;;; -*- Mode: Lisp; Syntax: ANSI-Common-Lisp; Base: 10 -*- ;;; ;;; backend.lisp -- Namespace support for data store packages ;;; ;;; By Ian Eslick ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; ;;; Copyright (c) 2004 by Andrew Blumberg and Ben Lee ;;; ;;; ;;; Portions Copyright (c) 2005-2007 by Robert Read and Ian Eslick ;;; ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. ;;; (in-package :cl-user) (defmacro defpackage-import-exported (name source-package &rest args) "Define an export list, a source package and this macro will automatically import from that package the exported symbol names." (let* ((exports (find :export args :key #'car)) (imports `(:import-from ,source-package ,@(cdr exports)))) `(defpackage ,name ,@(append args (list imports))))) (defpackage-import-exported :elephant-data-store :elephant (:documentation "Data stores should use this to get access to internal symbols of elephant that importers of elephant shouldn't see. Backends should also import elephant to get use-api generic function symbols, classes and globals") (:use #:elephant) (:export ;; Variables #:*dbconnection-spec* #:connection-is-indeed-open ;; Persistent objects #:oid #:get-con #:next-oid #:persistent-slot-writer #:persistent-slot-reader #:persistent-slot-boundp #:persistent-slot-makunbound ;; Controllers #:*elephant-code-version* #:open-controller #:close-controller #:database-version #:controller-spec #:controller-serializer-version #:controller-serialize #:controller-deserialize #:root #:spec #:class-root ;; Collections #:build-btree #:build-indexed-btree ;; Serializer tools/api's #:serialize #:deserialize #:deserialize-from-base64-string #:serialize-to-base64-string #:initialize-serializer #:serialize-database-version-key #:serialize-database-version-value #:deserialize-database-version-value ;; Cursor accessors #:cursor-btree #:cursor-oid #:cursor-initialized-p ;; Transactions #:*current-transaction* #:make-transaction-record #:transaction-store #:transaction-object #:execute-transaction #:controller-start-transaction #:controller-abort-transaction #:controller-commit-transaction ;; Registration #:register-data-store-con-init #:lookup-data-store-con-init #:get-user-configuration-parameter ;; Misc #:slot-definition-name #:slots-and-values #:struct-slots-and-values )) From ieslick at common-lisp.net Sat Apr 21 17:23:04 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:23:04 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070421172304.9E7A96B0EB@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv3920/tests Modified Files: testserializer.lisp Log Message: Check for cross-store loading errors in multi-store operation; more documentation; backend language to data store language --- /project/elephant/cvsroot/elephant/tests/testserializer.lisp 2007/04/04 15:28:30 1.20 +++ /project/elephant/cvsroot/elephant/tests/testserializer.lisp 2007/04/21 17:23:02 1.21 @@ -496,5 +496,3 @@ (progn (setf (get-value f2 h) f2) (eq (get-value f2 h) f2)))) t t t t t t t t) - - From ieslick at common-lisp.net Sat Apr 21 17:26:37 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 13:26:37 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070421172637.CDB172407A@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv4643 Added Files: testconditions.lisp Log Message: New test file for conditions as we add them --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/21 17:26:37 NONE +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/21 17:26:37 1.1 ;;; testconditions.lisp ;;; ;;; part of ;;; ;;; Elephant: an object-oriented database for Common Lisp ;;; ;;; Copyright (c) 2007 by Ian Eslick ;;; ;;; ;;; Elephant users are granted the rights to distribute and use this software ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. (in-package :ele-tests) (deftest cross-store-reference-condition (if (or (not (boundp '*test-spec-secondary*)) (null *test-spec-secondary*)) (progn (format t "~%Second store spec missing: ignoring") t) (let (*store-controller* (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect nil)) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil))) (unwind-protect (progn (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) (inst2 (make-instance 'pfoo :slot1 200 :sc sc2))) (values (is-not-null (add-to-root 'inst1 inst1 :sc sc1)) (is-not-null (add-to-root 'inst2 inst2 :sc sc2)) (signals-condition (add-to-root 'inst1 inst1 :sc2)) (signals-condition (add-to-root 'inst2 inst2 :sc1))))) (close-store sc1) (close-store sc2)))) t t t t) From ieslick at common-lisp.net Sat Apr 21 18:48:29 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 14:48:29 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070421184829.0054A1E003@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv18678 Added Files: ArchDiagram.png ArchDiagram1.ppt Log Message: Added some architecture diagrams --- /project/elephant/cvsroot/elephant/doc/ArchDiagram.png 2007/04/21 18:48:29 NONE +++ /project/elephant/cvsroot/elephant/doc/ArchDiagram.png 2007/04/21 18:48:29 1.1 ?PNG  ��� IHDR��?�����P���gAMA��????��� cHRM��z&��??��?���??��u0��?`��:?��p??Q<��� pHYs�� �� �??��?IDATx^???????"M@AADA)?????"6P????#?hb?FcL??H4???I?-&????5?h???{?^???=?e???????g>????S?9s??3g?z?SB@! ?@???\??B@! ???D8$B@! ?@??(;?j@! ???? ! ??eG@????! ??B@?C2 ??B@???C???B@??B@!PvD8??B@! D8$B@! ?@??(;?j@! ???? ! ??eG@????! ??B@?C2 ??B@???C???B@??B@!PvD8??B@! D8$B@! ?@??(;?j@! ???? ?* ??????%}???????????[oU??????c=???/?????O,C8=??c??????.]???????!????s?1??N?f?m0`A????7_??????-\??n??V??fG?I?}??G??V?Q?A@??"0?!?<)??Q???????k?b?-8????k????^K???p??s?F??@8x??!rC??Dq.C??C?}?Q??Dn?G*????J??0?[o??S??.?h?N?j??? ??????F8~??_? ? ???????B ???=z?_???p?u??? k?S$??U??e?G?;"?>??]"?w?????Bd$]?v%???_<_?d??j _ ??X???p???Af#???]"?w'?p\;?�' ?????????o??????[?0?????yr??W??@????m??x??'b??E?????P"?O?>?i??7?%??? ?%?; ? �??D ?. ?7[u??! ???O? ?)??>??s?.!O??9`??\??w??? ur???????~???8????K V??@BD8??B #??H???_~???*??????? ???a?8d?? ??A6?Q??l????T?????????????? ??I??s???y??Y C ????-?GF?j?@?! ??h3??? ??0?C??????_???|?????o��?IDATw?:?(;p???K?? O???V??j&z.?Q/??~ ??# ?Qy???hE?[??uA"f??nH??? ?T??2�??? Ym????A?r? ??u?????l???X?h???w???????3??H8yn-?{??Nv??+]??y??`?j"?(6?I??????xl???o????hA????????D?????R ??????8F[o?5?8????0?m!P??p????w??�7Z????qc_?W^y????? ?8? ^$'??_8?!pFs??Q !P??p????sB@! ???G?B! ?@M# ?Q???? ! ??h D8c5 ! ??5??GMO?:'??B?1?h?y?(??B@?4"5==??B@??@@??1?Q?B@!P??p????sB@! ?*??O?=?Y? ! ????{? ?2??;??a! ??B ")@S! ??B "??Rn! ??B ")@S! ??B "??Rn! ??B ")@S! ??B "??Rn! ??B ")@S! ??B "??Rn! ???:???]w?u??Eo????vX???&6VVmgh?1??k?aD'?|?c?U?9??pd??jB@?2"??[o?v??=???>vP????n;??????'y?;????G1?v??Xi? ??Ho??l?"??[? ! ?@*????K/??k??;???{????? ???)?e??=f?U??????o??`????v????mb?m???6???? f??;??PT?z??j??U*G9?b???#G???S?z???< �?&Yi???{o??????????6>?s?'e?3?>??w??P'W??j??n?m????Fhs??C?????K?b?mU20 ?gF?????z?G?��?IDAT?%??f??j?Za??h?b??Q?G?????=4y????????Z????????W??????????=s??? ?V?nk?x????MW?^J=4??`???,? &???6Q??d????k&y????r??o??9???n?w??Q??;??`{??Y|h`?"f0V????JN_u??5Ta????zt?.?my????????g???b_]????6????h)? ??0V?? 3?;)5+-???m?O,????????\^?$?c????=[????\????UK??~ ?w5????^??????U???^>&ogd?i/0???;? ??vP????H,?m????????#?n??+??n??L\?p KsA??R?[?? c??k???kt?qj?"?��?IDAT??Q|W6???{??g?e_g??7(??? ??? [9?ww??-???r????:??-????]=????c?????:lb?R6???z?\#??~??{E???O,XN?1{?????????QlH??6" %"?oq???fb??w?????????? ?????z?]?f+*B8lb???74yh????B)?Qy?k?E????)>?=??[?L???y???????f?n???C??7?F-?B8d+????Dn??T??????|???[t???????P????U?G?3??n?aR? n?f?? ????????8??i????z???>"?????u????w???^????w?Ho?0????????c?p1!??k???l??;~y? ?w??W??}sG??KG{?4??9???No?A=8???@???~+x???????X?(j?x-????m??)RT?+w?V?????$"?z\??7??^??}{???XZ??k?1O?S?6??*6.'??gj???E?AV3a V???c?U??)????n???????;kYT$ f?sK?_Nm>A%$^9????? kq??????T?6L????????j?>?+??kz?????0??=?>]? ???+???^?????[A????D\?#1????I#.???K8??"?????>u?w?v??V_P???r???U??ns????????��?IDAT s}8?j??????(/??`9???F?w?*?tP???"c?~m?????'???u???`?o?!? -???????r?S?)s???m???????]??!?????X?+?!~?3?P6T<m???^L????r?n|?L?9???v?@cF$?????:?????????|?}^??G????]??f?K.???_n?????-??s?Y+?R??7??n??~???Sd?H?$?v???o? h j/?{,?c^???)??+d??9?????i?uZ+??$?m.(??????|??@)???%l? g????_??a?0????2r(??????s?5??|u?m??</?`??u?u?? +????????_??s??????:#? 7?~?+:c???G??(??/QNY?p????? ?">?&E?????????*#z/?a?|? ?????-???4?$????0??f ?V,&i?v at lW???xl????; ???? 9???M??l?zQ????????h????#?O??[ V??r/???????!??? ??[{k??:?A????X???? ?9??????kK?=???vE??X6/??D>??5x?Og??????:U???????Q???p?w7??,a?v???B?P?7 ?K??AV3? ?T?d?#???H//??| ? GW|?7w?pJ?3: n"????Ix? ?a??X ??????:??m[???????s?????zgNX?S#_??? 6??/w??s?,4e?????'o?8??&??VR?`K?A?#ZQ?? ?.?E?M?Z?r??u?1?| ?7o?u?????I?#v??????}?X??&??H????K?y ??/N???OU?I?#??/O?{????????????????????h7?sorm?? n?[~c38?DZ??!h???d???!q?""Ip?l?????o?^???rc???G*????6g7 O???dv!??"??l??Ux????'q8\??K8xHm??h9U!?9??????e?o ??9???+8????'?a????-?3+?qIxi@ H9?%?c?D??????:z|4?8??T)9?jbFP?"I???T??Q;?x6????|???yE(45??%???4*.????'??]2??M7]Y>???????v?l??????x???K,???*????????0g?W?;????pD?p??s ??@.?1??p9R?O???? ~CfxOl?iy*?Td???!q?""Ip?? !?????>?|?O?#V?5y7 w????f???#_?t&?Y?I?#???/?@?????L?k?p??????????*[o??`??9?p ?9???? ?x"??c*4 ?G3?�S??;Z???T????] ??bm??? ??A? ?p$?nICCfv???Km????????~???cu???????$<?p?\?p@??? o?????N8a ??????????}%??K8x? y~???a;???????T?Z8????f?No????'?????|??{?Y ??????$?^?cy????s? ? a?[LY?Y?q?? ??i1?|??z��?IDAT$?p??T???[Y?? U?`? ?j?b???~?D:/????/*/t?^????6."Q??pp???????????{???_?I?????.???????BD??Zm?????????_???Z?????]9X)????+??????,??md????G4??[f?MB? #?]CD R?}????_y??KG`m ??o_??Cn??????;?jbFP?"_?O??~??E8,nA|??=??t???a4ba?o?n??'???J???.?b)??G^???p?y??J???u???p=|??????T??a7 OD&\2??(^&T?????'??HL?`?X?m.2?YU?gS^}?????????????d?P&??????$?"? ?}?=??????t???????? i????J#>?WY??[3???F_9?j??C??E8N??U???"E?,??(q?????# ??E8jd??"?M??+)?????�??V8?qn? (???????????&fe. ?q??Or??y^???5.??????wx?7�GE?? Tn?Y?9?~?? ?\????G?????9VH-??.? ?i??rw???/????W?o?D ?{?;????q?'?P^???N?g???:R)??KN???U0��?IDAT?'?u??????y6?????+????9~o;?Q?I9o?y?????|w,3\???v??????>??F]?~IpK???????0???(K?t?????{?w??I[?A?*??'??Bp??L6?f???????Ib???X?+?!~?3?P? P?O?G???a?>???O{??H?T?"A@????????????XDHx;?????8?;~? ?????~D?vg??? ?S&T?D?+?�?%?t?:??????a?*???J?&M??zw?%m?e(O33l?? z????? ?R??'??????"?p?????l???F???C????+?;s.:r?"?Q??H?C?#Cj????????Be8????;O????p'??W?*????"??????nU?$1???X?+?!~?3?P? P?O?G?}9???Q??z {o`? l???[s?U?f??D?????Z0|??t??I "^??????TI?PMM8?b?ivw???T??~???]??e?pW????? ??2??j???V??? V?f?[*%! ????$W?8?? ??L??h9?N???????V~pk(?_?Y?%`~8H?j ????sK0l? 8A$[????:??GbUR&TS"^`? �? ?;-?????&?????? n??8?-T???j?u0???TN?K?[??H?#???HJyjw+J????H? 6????#P??#?????b??+??,?m?[?8N[??????q????????2????x?2m?????*,?? ?????????v?gV??p?????\?A???{f.is??ype&???>?n?q???{D???fw?nP?E?6?p???Gb??HL8?/5??? m_?M????#z???Yr??Ck?6?&1??Y???? ?R@??D?U?A?R8?9??????}???r??<??????zR?x?Onr?B?w???n?r??t???????o?0&?3?t??*?9??????? {??kg??Y?2Q?'? ????'#??:?w????1?w????{?v?Z???????e???�?@97!T?;Q&_?t??c???X?��?IDAT?Z?[???c~??&?]?vZ?t?HCj@ c5???Fc??X????????H#(sxh?????J?f9?&'TKi?5??]?p%8??G8mIT??\?5??'??nU???&?98V??p?p? I? ??ks??T@"}???D??cc%@??H??j"???-?9w??????????D//LT??\,X0?? ??7?F�??0V??p?p? I? ??~{???�e??;f0V4???L?&?+?za??H|?????6?yC?e??s1zt???? ???&????nb2 7?#?w9_?UyYX2??s?~?????U?S8??????BT???&??u??7n??%[?:?@�?@?E??p?p??I????hR?b???ID8D8w?b??]t?/I?|#[=X??}?????Z???,?( ?Z;?"?\,Zt???>??t?m??`5fLps?$""???m?I;w?8Hv?????e?1.u??)&^?\p~?ju?:?z s1i?N'?????B??d??c?5???????? ??[??G?h???,??%?m?'^??Ge????????u???Q?s?(??6L??p?pdF8???? F>?4?d?'?&????pX]??"r????-7????@?Y`.??{k?????,T?&?9 ?s.?-AL0bh????h?IjM"?!????(?B@???p?p?B@!???????(?B@???p?p?B@!???????(?B@???p?p?B@!???????(?B@???p?p?B@!???????(?B@???p?p?B@!???????(?B@???p?p???????3?:{??q?{???m?&??�#e?????@????wZ?��?IDAT?}??q?????M??M???)?@S ???|M???J?>*_?:=?????5?S?U:L??fuZ?[?#&z???^^?}yZ #n???? ! ????>??w\og??7??a!R???e???F???wf??v?B@??E@??????#^ :I????{d\|H} W??????2????Y?FE@??8??^??%?$?? ? 2?k??N??%??KH?B@????/j-CEo?p????p'n??3??X???'?E?G! ?nb??Z?PQ?A? ?F?G^? >??A? ?l?.!?K! b`??>v??? %~t??????W??%?????K?6bW?2! v?6??c?&? ??/?6?T????W?! ??`/p??k*OE{,???H?C?F! D8b??????B@d??? jM?7??,????#????'??M???~!?QK|?9?8???B@?HEG*??)?&y5?0??%"??_??Q??\k6?l?? @?p??F! d?(??c???k???"???'Z???Q?l'???u?????]??????^?~qY???!?z?>????????+?Q"?Wq! ?@? ??_??Q??c?_?W|K;^Rp???y??(?y,? ????e,[?????????S???????????,o?????Co6B@! G?-a?1`??????.#a?A??.??w???q%o????c? ??w????v??M?%Z???????.?Yym?K??VV???{0?!??$?&?B?D? Y8??p?m?u???(?;?6??^?y,??n}8?�??7?+{???^???R????????3??5???��?IDATw^?~?_???N??T?? @?p??F! d??????????7???aO?Zw?(?8?o????MN?? ??Wk?U?z??Y???????[?j?Qi! ???/??pda??p??K???I?=~?3???uC?p?;??Ab????`?_?6??l????p,???H????D8*??J?C??B??(???]??{??????F?S??p ?R???c-???_???F???Mq%\T???)?.ut?7?�KECB@?Rp?/d????qu????????y???p?8???vs ?? ?p ?CH???? ?&U"?UV!?<8?"????o8??? ?a{????????!3??J#?|.?A?Qn"?(B8VX??F??N~?R?JM2?H??! ??,?k???w,w?2?h??3???pp???X??B????D|8"G*?0?H%"??v?? ! JA?a???#S ???F?? .xz?c? OVY?g[?C8???????i4C?? @?p??F! d??????~?CZOUf???B?rc�??c??? ?x?JA????[ !?%??%-??5??? ?,?Z8 ????????_??_A?q??^?>~?????%?k? P???? kn???#p???=????=???p(??>?m???????^X?????!?p ????FSi: V??#??_????????b??6_C8?|?m>????^????T=WN,?t?Q??G???: ??aD8?`??C????.??B?G`??????b��?IDAT????=#H???G=???? ?M???????(????????????????=W???`??X?2???G??{z//l?eV???(???5^^?^?_?Z??????y3??????{?'V>????3S?C! ??{f?????????????0???? ?????g????????j?=C?B@Tv??????2T?p??c????p?8G?@??G?7r????Q2Z1??[???i_6????2???0v�,)???p?LN?|R??pWV?9t?"??h$????{f??????t?E?-?E?C8l$??p/??????c?5Gb???Q3???~| ?>??Q???c?) ! ?@? ?VG????????????&?H?QeB@! ????;?\! ?@? ?Q7S?? ! ???_D8?w??s! ??u??G?L?:*??B?~????S???R??|��?IDATB@? "u3U??B@??E@??~?N=B@!P7?p??T??B@! ????;?\! ?@? ?Q7S?? ! ???_?C8?z??i?^??m??o????h"X?+???*f-?P?�M$+j???R,?r????/~ ???=?;?y??/??#???O#??�+???Y???????}??n?h1?Z?%??D????+M8?M???;?_ZrB@4?=???C8?)i?F?????j(??+?qaV&[E v?.+????????�???r?" ?8?.*PE?W\eh?{+%?????Tq???e@?u??G?????? ???b?d?3T?p?e??m?A??N! ???Q???c??4 at 5?H???p[q?K??*J8?;)?m$??X??!p???X?% ?!?A?m??.? g?(??/IA !??x??D@ Pd?????0??m.^G?V??A?&3Q+?c??_o???]??M??w?S???g???oy?-?m????V???[?ozb|??k?<`3?????T??g[???? ?poE???????EK???7t?7?po?9??[??>~9????Yj? ????????_C??\M&??j?p??C?m{?=$??l ??d??RG??^?u??`A???X?K???p??"???|9E8?p ??_x???t?????.????;23??=s??(??H]O?l???!????By?7?3?-????? ? G???H??????�ks?q????.W,s?$R3???m???U"?'???Q*2??????qb?w?$^????????Upm??lt+d???F??^0?UOHH???}??????????^k??s4? ????7??:t?x??z??#?-N?????-?Q???U??eZ?M^???k??\?:????\n?a' ??h????W?5????s?_???-?%H????????|???W????Y?k??#F?'???#??$?!?"B8??? ????F???��?IDATZB??p??q???_\y-?0?i4????(????[n??}?!=????????p?v?;L?O?h?zKe??9?MY??ru???"??X??E??????>c??????+???<{6?q ?S?V???????>QOxup?9Q?_w>?� ?????p?v???_~?%???E??p?y]k8r%?#????????\?a+????q  )o?/??9???9??(???q????,I9B??E8?p?�?-^;8n ?E<6b????=??????X|??????}8??????3)????\B?$EB???Rq?"Y?????]?????XP##??+N8????$?A?iq9????`G*??Yo{??^??fA%?{??? ? Gn?kv????9????z?a3c?p?? ? ????v??p? ??u'?@8?FA6???b?)|8? k??? ?^??bm%???  ?:????"?V?????#dZ?G?3??^HdM???oV??3??~??#?????m@<9?5?\?V 9]?T?zm????i??Q?Th??:?hU+?#????po?????;??!?B[ ? ?)?&Ig???????{?m???>c?????H?c???p??u??R?w?3nD??q??Z?��?IDAT?Y6??)[??2?F!v'?UM ?\?E?? ??????C???",???p???CG?a?3P??i?Rp?5?p???????????I?tm?'?[?. G ????D*??2G,t???vy;??p?RfE????m?n???F??"?????R???N at 3D?H??C??.????7?????ei????>??N{?Ko??#?wXqp?L??h??[?Z$f?????p?AHD??? `h?q??sb?A?C,??#???????? 7???H-i?_Z+,N8???7P??\??????j?????u????L?;????')???????T?\??.????e????:?F???QhK!??l??8Hj)L8l???e?^????Z?e?vB??X???;e6v?? y?q????H??X.h????;??L?B??p4 ?@@?}?XXB????+S-?????1?C-A/x?!???A?,?hZ???8??W?vl? ????}?? g.Va??+>?J$????????}???~?nG??k(??<.?$?4??ec??????*????'(???RO.??????y?*?p8 m)?????A????`??? a?$? ?}\??=^ ?K8??n?PP??8^?Ty+H4.?k????7??8??xU??E?????I?N?????��?IDAT????2??V!???????Xk8???d[{?M??SS8??L?B??p?p!??O?C?Cd??????#dZ?G???h?:#???????l??????B?a?e?2?????m??:h?? ???_??c?4@N}M??????$+????c????qc?G>4R!????????U^?�"B ?V\???p???f?i??9?????[C??�?z?A?c??4??????$+?????????q??_o?TB? ?=??????X?% ) ???b?d?3T?p0?????~?jn`?s!?�???k???}?QsIH? ??H??fe?U?p0*t?C????55?~?!4'Dp:?y{"???(i?iNy??KXq??gY3W?p0????????A?{?+ !Po?rY??b???\& Po??V?W\Yi?{??!??SN! ??B??h?I???B@?:"?>C??B@?@@??&QCB@!P??p?? ?B@! �??D A! ?@?# ?Q?3?? ! ??h�D8`5! ?????G????'??B??h?I???B@?:"?>C??B@?@??????O?r?z5/??B??`??; Re?Qwx??B@! ?@ D8R??"B@! ?@2D8????B@! ?@ D8R??"B@! ?@2D8????B@! ?@ D8R??"B@! ?@2D8????B@! ?@ D8R??"B@! ?@2D8????B@! ?@ D8R??"B@! ?@2D8??? ??????-?}???????????????E?????????o???????z??????f??????O?~???j??????{???????{????????_gF^z??"?}????D>???B?B2???[E????????ey??????_|1(???O????,R?K??@?@???��?IDATV?@qD8$!Q?y??B_??????s?}??RP{??????RC?,??V[m??M?p?;w?|?G|??g??g??-????????ae=|?p?????Yu??zRv?r?e?}??^?z??????%????J?2?????p??D??*r?FS ???/???>zj??77t8?6?u??G?+???^????qOu?c? ??????n?.e????g?m~?kouK?.????????/??????l??}???b?{ ????e?????,}?r?-y???y???#?c?Z?a??v??+????\???????dl?D`?v?)[?Qh??B?D?#??0L?/?6G?:t?y??'/.???pA@??????B8???V?f" ?????zk?J?"D$?p??-2 4M[????q?;??I?M?A'o???H?oc??O6.???:D??c??lEn???u??*?~?\b?9H;??S???K8 Lv?J??��?IDATlmq??8? 6?H7??Y?#??T[?:+??B??m#R? ? g?"???3F8r???p?(>A?(Q'??"?u?^?????p?v?m?zz????????1???s?E?}??_w'a??H=k??&??????rX8A????z??!????3??w??*?G"??(.??=?qCP=?n???+?????l??Bubh??-= |?s?>? d [?P?v??N????:?????"???nJA8b??NPq???#??XC?G*?VD?R????8v??p?O?]$?u42X;}?????{?p?w??????pd?????K8???_~???sc???H??s?+?G?I??Rq?[??|?????v?mG?a]M??4??# ?Q~?????p????1�c?Ts?$?^?62?|E????~z0V?)t?;???????��?IDAT'??_????a????!?*?Yg??? m?"?????U?e???WRj??cg? ???8?!??????????c?r?r??;?@???? �A???4??%???oY???k????:?\p??%w?????!? c???x????NB?l-?H?)0??v?????V??|??t1????????y??Q??????2?????oB"p???p/??*????L/????T??]V?.??????D]?`?$?E,Q?z?8SpK??&Mkz? ??.??????I???cq??;?H4A?40??? ????p??????V?]@<\nPo??6&??7?pP?? .<.!.?? I???x"??W ??[In?Q?aH&{?&\??J?}.^$x??$|?R-E?v????RG??pp:??942F??????F?e{?#?s?l?I (s& ????q?7C,????l??q?? )c???FXH? A???,?(??X???`8v????S?????????Bc??L"??h?8? G?'p8.???+?{y'????????De??v??y,????Xy??????N ?th???w^??[??X??L? "??=???\?"??{?? ?e???c?[S?????c|?qc??S?v ??co-?c%. ? ??????y?*??E@?CR!??B@???C???B@??B@!PvD8??B@! D8$B@! ?@??(;?j@! ???? ! ??eG@????! ??B@?C2 ??B@???C???B@??B@!PvD8??B@! ?L8N????W??B@?yoa��?IDAT!?�?g?1?*???KB@! R ??4B@! ?! ?? /?B@! R ??4B@! ?! ?? /?B@! R ??4B@! ?! ?? /?B@! R ??4B@! ?! ?? /?B@! R ??4B@! ?! ?? /?B@! R P}?q??????`??#;t??ymb?+Cm?�???;?H!E?0; ?9rp????hr$| @f??)?V8?7N?c?r)_???;r????:I??NpZ? '? "im?$??v????? ?f? ??y??z??O?i�??c?????[oM*????q?F???? ??????Kg????#?"?? ?H??Q}????Sz#??pW?Y+_??5?_? ??h???3g?o??m?{?U??�???9 ??F??F?w?$?'G?md?6?B?^??Sk?*J8??????? Xj? ??8??bv????PT ?????F??LGe?k?~?????Z??????5?A*J8???????z???"yDJ??b?65$?7?.V8?7jDS9*BGp?S??|�/ ?.??PQ?1p?Q??kd?4K7??DL0?5j??N?.(*????��?IDAT�?9? +?????????!C?1?bN f?(??c??]???EK??E??)w????j?7?.V8?7j?p?)Bd?]???x9v??`???ry??[5^?# ?vA!PaZ?.&Io???tP&??.?v??A??SE;???<n???Tx?QsB?E?b ??F?|??m?? ??????[XU?{?j??S"??[:?????b*???????????j??L?9?"??*)?????p???*??yu(???AG?p??P\?????G???K???Vq???*;n?7??7E8D8D8D8?)??(Y8*~R??4?}?"?? ????DG*?G. '?]??p??! ?i[?UA?Eo?E??H?Gm???,?????-?p?p?p?mT ?)?!?Q?A}5??p????????v[?BM5??7E8D8?? T??. ???!q """"UC?Eo?p?pT` ??&\Nb:P?"""U?nk??P????M??b????)?H?????G?p??""???? ??????@??c_???w?K? ??7??|?u?? ????JYC??It?????b??^????m??^?j??o?9?>???Uy?n?F#????\?f?GU?!???e?')K=p?Zt?????L*s-????^{?4u??7?O?????j????)?????V?[-??? S??????)?H?B?????+? f?.??.??9tr???y??CMYC???[????m??g?k????N???'??m????\p????wF-t;i-t??.z?B??*??}gt??#?b)??=??y???k?????Z4?????|??M2p??n?7O?v?????k??????] H]aY ?,??t??D8??TD82?T?�??1|??,???>?}g-_N?l?T????(???p?pT????????=??wD8Ju?pyQ??# ???#!3?X8^9??????,e5T???e??@ ?UT?C?C?X)??��?IDAT#??)?????Eo?p?p?n??>RIM*S?e???5?PoC??????1?u?????|?u?m5??X?w7?+???&?{l???>v????;w?o?? s??/?????l{?q#??s?????*?K?l_|1??????????c???3?d?{?A  ??ovL?7J?B???O???7?7o?}?DW???I?F|p2]???=_|q??E?q?????S6?????????N?o�?d???v??jXU.z?,??b?????}??=?Ui?x9????q??~7y????????`_M?L%???y??nQz??hm?,?#-E?_+?????n?~???S?l?q???????_???w???Y?m?z$O???.?|?s?a?#e{?9x~?????3????? b? ?wxt.}#? ?V? 7l??G?p?m?xG??G? ???? d8?? ?Z???$T??r?cl?? ??2)?y???????L?Q???? ???]?w??C ???M?^?;????C8?_????=?h`?q??#?t ?Az?i????1X??????????5k??-?d???&O?6???l?j??4??????m/???i??or??M?y??????F?h?^??j???N?|????y????Un????m?+7????3?>?????!?????pbWV?3??? ????`?nz?_??X?_??9?8???�3aBo@??r?8.????-????[????????:?t????A?W_=?M??????=?????????9???7?d5N[x>j??????-??C+A??p?U????o???li????!??E????.Q?!???9C??D??]?j??V???m???oeWf? E??r??[??-??CfxI???g??X ?w1???\?E????????u?D??????=???????????????N??????\??????? ?=o?#??F?fXz??D# ?w??`V??f?-??? !.?[??Q??h?? [????S??E#??????#F?p?~?bI?{????*?!~?g????????FT ????Xe??x E` ?N|/F?????l?2?a��?IDAT????',l}^??$??:=s?a???@8F?jby t!?K???s?????^L? #W_?_??/x?????j?O"{!????? ?n??B?O????"}+B8V\?}???k???????K&&??#????3??Q????[?vx??=4??#?????w?(O??w)$#\?e?????g?_??)c?????????n#?cf?????h? x?F/V?.]?????? ???'+%????%\?0 ??? ??!?I?nQ?r???n???*?? /?a?a??.\P?Jx?????; y???I??_?d+<=?????{+A?E??;??K8?&K???~#?}s???5Q1^?? ?zb??S^??K,??B??&?/?BJ? Rl???[*??x+?^n?V??[1m???%<<"?%Fj,?????4?W??M?@?p?W???_?E?l?-??4?I??u!???I??|G???={ro%B8??a?7????!?j?pk???0?��?IDATXU???EYm?????cK??E???Z?y?p?;"O?+ ?s?????V8{???k?_?HC?????t?????r? N??P~?-??Io?5=???k?????nO?V5`@Wr???C? ??z&?A??C????_?l??r?)??4?????? ?D???p?#=$\OFR?pL????`?(B8????'???[m??????w??z+?7u???X????bJ?Pg??,?????o?j?u?E�?????d>huP?yDj??c?cc?3??_6????n??I??p????????-?d?????!C???#??=[G?A?????}???a$?eb?oM?h5??J?>p????1? ??j?K????v????6Q???$|?#?nu??g????D?p?????Ir?????-???;?8??9s??x??HQ|??????!?U?????%C{??K" ?V?????Z? ?KLU?zH?S?f6????v??bW??!8XD????????p???I1QI?{^???v8????K?9???m???? ??wB???B???k?8?? ?9q?4/Q;???{(???9?>x??U?8?????G\.??ww????"????7??o??2????ods????????_ ?????o??Oc?4)???"???????\Ni?n?[1WX?Q=L8???T?????p R?????"D$/9?J?2-??~???1"?U??o??E?,o??????_???cw?"??????{??o??z??-??m[,???? +D?17l????????}p?@?????k?????C??26????K?E;??j??R??Xn???Zn?[ ??-?>��?IDAT=?+#???[qh??'?{???xX???"Sm?A???r7?` ~??L???}?8X???Fp,?H8?u[!p#?A?????????j??j?????NbTpA??A?a??g?8W???h^?Q?E??;??-?%(B?N?>|???? Q?w?C??jN??�q???#?9?5\?Q?hq? ????a????D ?~??m?'???3???????Ej?��?IDATq?;@e????B8??MW#??\??r ??>?<\O"FB??????T?N??'?)?3??q?#~8 ;?,????O?9R????????????Y?????&?-?1??B???9@?????{??jXUF8~??N?{??j!?V?????,?{?%\ gUX??7{?l???k??a??u??;? I????? cO"T?;\bN?Y.?Q?E?R:?1o?}?????? p?{??????K? ??yh_? G?ZH??A??s????p=8g?u4?i??Zawb???2??j$?-???????B[?`@8xN??=???M??3?,??????p?[????j+????????^_M?,!? ?)?!?Q_l??uY8"""?n??G5?@&?k??M{K?<#hB?#?N?M?:o*:BN?8>P???M???????A??pT?y?,??-??9t-?n_5??Wt???@Up??""???? ??Sq:?????p???j??M??b???????+?????M ?w A??????U?6e?ZlU???h? ?"u1???7j???A??)??b+???Q>?u?*????z???sT?EK??E2?v??t??z????@�yC?b?Sz???????2???S???^G???]^@?]85????c????y????w??c???4p??.?7j????�???P?(?��?IDAT/CE ??y ??pa?,? ???8????s/?p|??U?g ??!u??)?Q#:?Q?,?;???????u?? ?? ?3T?p?~????????$?�p`w?}.???r7????]w???????n7a??????????6????_???M???????4s???/?????v???????4(W_}??)Stp?3??f6O??s?Fm???6??i$???G?9|?????n;?h????U?X?!??[ou??]Z??'?????u??}?????B}k6D8???g?}6g????G?????kQ??B`?m????Xcj??4?`?>???dy?l��?IDAT?c??y???3I?# ?Q??{??k???#???`?X?????-???A]????So??W???O?|???&d6?? ????6??j??????rf??GVH??p.???7??D{j?6?q??????4~/?6?1Ji??7??9Q)e? ?Q z ?r?r??'4????NPLY????:J?????m??Y?VR??"? Pe?q???{JB@! ?@?=K???R????*c?n?w?uW?^??????vC?W�?????j?>?l?j?&d?(q?????_?W_}??zT\?" ? Q??|????#???F?q?????~???j?????}?k?s??=0UID8?#??????c??:=P?A???n?0aBE?j?F?62?r?????a??J?" ?Q5??o|?m??7????? 5\f???_~??2??t??md;??7x???lQUmD8?)\]9????c?jNC???c~?]vY?i???6?1?\Z???7?5#��?IDATg??Y9*W?B�D8?/W^y%????{o??????n?i???.C?MZ??F?&?????????R>???f???[o???+7?xcM?F??????w'?l??6ieb??x]Z)7??\?G?????>??E?9??Z????S?L???????Ik??????Jepn?VD8jh??~??n8{???>????????? 7???.??\MSW ?Q?????J???o????;w?L??????5?Wu?(?7?,YB???; ??X?"???????#??2d?�??ONn??FLD???4s???B????_??W???}???v???@?????8F>????K?G�??v??G ?? ??N???S?)h?FE8jwB?d?????]???Q??U? ?????,?k?NN???9G??,??V?*?@Dn??B%?o??D8jtZ9???4?f??????n???\s?????w?(???,WvJ???%SrXu??|N3WV?Pa??O ??P??f?????f$H?=?4?]????? ?d?????K[n?a?K??$?0m?��?IDAT??3?-\ u????+???6?l??mt????[j?UW=ztu?*? ???/?8p???3??e>5V!??????e??C9?s????????7???B?y????H?N?:????;Y?0??????"???C????}o???`Q;??S!???n?-?GJ????0?BK{?=?(]?~???????6??_???pD ??????$?K??=??ooKq???nuG8z??m????X ??m[^R#??G??]?????u3$?r??????????+??pJ??@x6?d??????_c?5??Q!EE?`??~??mN????r?]v?e?? =z?0>z?a?E?C ?V?{??l???I???[o={??*??s?9?R"???[y??m}?;T?z??a,?|????T???}?&?U??Zkm??V{???f?m???RP?n?!\??vR?Q]Au0?????]???pD G?E??9?????v?]?n?h?????Fwc0??=?V??e?d?&%?Ln~??:???=??b?0??B?? a?6c??q\?A??).???4J?#?Q#??D8P?Hd???n:4?8??'Q????Q??0f???X???`?x\??K8b ?h?c????7?f3c? ?A?????????o????X??t*?R%????c9?;???;??CDr8f ?M`?<??0Kn?X=?H ??3V$N+f?????A!???X???*?K?.???eb???G%?0?????nV?G,????8?4.??0]2??\?rX#????I????oE?Q?\(1O:mS?M??Z??!???xu`?`? ???K?-Z\ y???+S??{??u??x?????dw?????R?+$m"?p???$??<7?IlJ???rl?)2?p1??o?z???O??(?Z?O?z???rG???v{?5? +z???"?R??q?1???????g\??? I???X%ph?I?%.J??q2? ?j?!6,????? Y8?e?s?k>?,s,j?J?? *^9?V?g9SJ??D??B????M???5?1?V?,????%??? 1?m??)?1?kj???2?9???K:????#??? wFlNv??^}????B8??d(???a?8v??$?��?IDAT1??S?J?qR?*???'|L??!?("-BPl?0?]:?b?-?na)???W???{??c?9!???(W*?c?%?????@???` )?C?S5N8????z?????????N?)?|??#??e????????j??8?p????~L?z?d{??????'"??t, at c??????58??p?MsI??c?&��?IDAT?F? ?????L?ABlxbV^????5Pz???H<.r??*?p8 So???(?[2?*????#??-"3GNA?L? ?? ?A??e75?6??y2(|??I ??>?aC?]?fm?3d????/??#y?Q?\?_z?t??f?????G?p?!??x?7??G??|??W?S?]?vT????8??????c????? ?F?A?E?????�?Ch *?H?1R?k???3?"???E?\I??W?T@??m????R?�f�?? )??X?v?j???&????$??9? j????G??O????uUaE?`cO?r????4??????A???s?bG+??V??=?m???i?????? ?-??????J? ?~???:#G????$3??$???/??@???5?.?u? ac?>? Xb?A?? ?a????/^ly ] ?@?if?x???FT??y#???J ?z?i??N??J????+????b?&"d 6l?"?3???(???FD??0\?6,?)V??=D???a@%c??��?IDAT!?????X??Hx0n??d???`?lcE?=3n6?-(?/?v?g?kT???9???F???3????E`??eP'v)???[9H,p&?KR?>?K6&5?($??j??9?i?:??k??")?3ZC%A?9=?{? ?Z.??k?? ??? L??0d2?J???????????|??<7'Le?t?????q?I????K?]?'????r?k??"lu??p????? ???????"???C$?+Hgn8?B?X?}?)???1?X?????dl!AM$i??$R??????]???(???Y??+?\V??>? ?????p?????b?? }?z????j??E2I-|_???L?.sZ?;}?t^??2?\?????[)???Mb?(t??Z]O?I??C?.pdL(?? F??'??5???\$SrX_B?2?5?g?u7D8?k]d??????L??/? ???^~E8 ?a^????|?&?l??\$SrX_??2?u???H?E8?k]d?[.?p????Y??\W: E???3?8?2?8�r??????<(c???????i???u? ??Z???'?2n&Uu?T?}l?ky?S?E? ?H???????.sZ?;}?t^?#?T*oH???+8EM???dJ?K8\??F?????G}??&?m: E?D"R???H???J???Y?9????F:/??R at U???SR?????"??????9??=???!?Q_???z?NH?7??Ti?.?)9????l?eN?z?????p?P+7?T?}??E??H??????eNkd???n?p???h???SR?M$"U??dJ?49)?u?????k??")T???@: E_?yQ?.?)9?/9q??????"??.????T?}?H???"???*MN?f]???w???GJU?r#?NH??{^T??dJ?KN\??F????F?.]?\w?u?? ??????bii??Vf*3pG??Vf:2i?qNS?#? ?D?c??a??{n&?J???y??7t????/?��?IDATU?\??i??%SrXGr?8?)???4/????????h4sW?)?+?r?,7??T`???)9??\d?????PG*??????o_g?u??Q?SV?)?+?r?,?u^T??dJ?HT?4?:R??%?|? 7????n????\GK_?????a????KT?\??i???J?????%???VJ*????1?x????????k14a????3G%?O?r OY??N2%?e??+O7?%??&/?\??]v?r?9J\??gF??'?pB&kR???5m??H???6???9?DA5g%?H8????^??? ? f?t?Fxk?kg~??'?K????f??9mN?P????p??^]?'s'??2xT^)?9?????\???Qh%h?+?? ?b??)9??Td>????MXC??ln?qc?"b?W?$`??????r?,WrZ??2I?????Q?9u?B??t?T4?B@! ?@Uhv GU at W?B@! ? ?f?q?W! ?@???jR! ?@?! ??l3?? ! ???"U�]M ! ??h6D8?m?5^! ??U@@?? ??I! ?????G????+??B? ?pTt5)??B???h??x??B@T?*??&??B@4"?6??B@?* ?Q???B@?fC@???f\?B@!PD8?�??B@!?l?p4??k?B@!?1_~?e?56bu"?8??B?"x ???i??O?????.7n\?.]6?|??????????~[o??c???????s?=) ??H??M#?oj?E'N???????r??w?UF?M+????W^y%<???_u?U??c??????=??cd???\??5R?m?^????c?u?]w???z??W3?????g????o???BA??;w? 7?m"E??u???_|???????R??? 8????H:p????��?IDAT??e????JW\q?????C;?|?I???J8>??3??k????'?????u?WF?;????o??5j????s ??3?oI???/????W_?G??=??G????????Oj??6 ?l?????????*? 8-???E??t ??k?=?????9]??q??????K???!i?f&~?!?????7?????????_???\0,+??????p?O?GU,?w??1cx7????|??Y8.?????=??C<}??g???;????c?=????Zk?????j+?=??wy;? l???R?s,??>??e?p_l?S??wn??q??? ??Y?-?? O???N??A??4???? +h??????$??c?J?pp4?????c?????/4:{??,???n?p?=t??!L8????????hR?-?_?#?{9?(??x????????g??2??G??L?0fp@??O??????????O(y????D? 9??=???? E8j?pp ?{?????\?-???Oz???I?=????dY??l??s??,?TP??6?pp??k ?Hq???[?#|??;=?`W3?Kr!??}?J?k?????????m? ??K/???i$? #b????!???9s?@?"????pa??_!8? sE?Td? ?I?N?t??m???L8??1]N?@d?CX2?p?3 7??????8s??B?#B 9?? ?????+.zxX??o????9L88>+D8x>d???8?qd5?I???-????p???`?.S"?????A?#??k?k?a????v?^?0@??$R?pn o??-?? ???m?????p ??< ?a???s??????X?4v?8???????R???l?6^???ck??#?caT'?F????�???????j???R.??%?K$???????i?RC?yj?pd86U%?@?#PS???\@0?b?Y??\ko?l??gN8?m?'?=UL??O?: ?@??)m[|?\w? 7??6b?1??L?Vn??F?m~????d??|?l?? W\qE�?%?\d??f|?l??1?^{m?S?YM??B sj?px?????/%?????n@?.]?4s???P?#[??????W???(?????/??V_}?q???|????pz??<\?d ?b???? ?XSF8??????????9?????;?d?????B@T?4Z??}??`D8?,????D8?*????? G?!C????>?'F8???????3F?p???? 2` bA?H?r????"s???d?"? B@T??&??R??{????.??^5?\??n?,??{n sh??6??? ?>|86?:@Yp?????E82_?"? G?i?HG?EV??C?f>S?P?l??i???"?c??i? ?F9L ??~??^z)?y???K8p>E??yz??wR ?!]t?G????D8;????? ?????????~????L?B! ?E@??. q???Z??h????&?dc??????B8???y#?-hR?pd???8GI?[*r????m!?q??'??????v?T??# ?Q????9?????/r?? &\????7?��?? ���������������:����������=������????����;���????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????9��������������������� ��� ��� ��� ��� ������������������������������������������������????????8��� ���!���"���#���$���%���&���'���(���)���*���+���,���-���.���/���0���1���2���3���4���5���6���7���????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????�R����������������������������������������������������������������????????���??d?O???�?�?)?�������������b??#??���?������P�o�w�e�r�P�o�i�n�t� �D�o�c�u�m�e�n�t���������������������������(�������????��������������������������������������� 5�������S�u�m�m�a�r�y�I�n�f�o�r�m�a�t�i�o�n���������������������������(����????????���������������������������������������t1�������D�o�c�u�m�e�n�t�S�u�m�m�a�r�y�I�n�f�o�r�m�a�t�i�o�n�����������8�????????????�����������������������������������������������?; ���?(���?��?��?��?����� ������������������?b��/�? ���0�?��������??�����?D���A�r�i�a�l���?0?0��?U1�pB1�pB????????���??@?????????����1???������?�?D���-?3? �0??0?0?0?0��?U1�pB1�pB????????���??@?????????����1???����?�?��? ���?�`��������? ������.��������? ������� ��@�?n����???���" ��d�������d���������@�������???��������??���������  �������@@�������``�������??����� ?�����??�����? ��� �����-���������������5���S� ?���?��?���?��?��?��P�?���???��??�??f��??�?ff�@�?���??f�������?���?�����?�����������������?�����?g�����������?u?�??;2N??�??;��?g�����?�������?4���d���d���d���d����?H���������????����??�?H???????����p�?�������p��p�?������@ ���<�����?4���!���d���!���d���@???���?%??����g����??;������������<�����?4���d���d���d���d���@???���?%??����g����??;������������?����� ���������������?(���?h�����?���_�_�_�P�P�T�2�0�0�1���?D����?<�����?4���������������X��������������??����������??�??����?���_�_�_�P�P�T�M�a�c�1�1���??��@�f������ ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ������������ ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �����?�? �����?�����%��?X�����?���������������������?��������? ������������?��������? �������������?�����?? ���?������ �������������?0`�? ���???�����???�����???�33?��??�??��`�? ���???�����???�����??S�??f�?3��?f��`�? ���???�����???�����???�???�33?�?g?�`�? ���???�����???�����???�???��f?��?��`�? ���???�����www�����???�3??�?PP�??��`�? ����??�???��ZX�???��db�mo?��??��?��`�? ���?���???�\��???�q>9�?y`�???�??�`�? �����?�???��3f�???�3f?��?��f??�??�`�? �������???�3f?�???��3?�F?K�f??�??��`�? ���hk]�???�www�???�???�???�??f�???�`�? ���ff?�???�>>\�???�`Y{�ff?�???�???�`�? ���R>&�???�- �???�?{p�?_/�??��???���?>����???���" ��d������d���������@�������???��������??,�������?|����???��" ��d�������d����?���@�������???��������?? ������?�� ? ����?��" ?@����?�� ?`����?��?�?���� �?n����???���" ��d�������d��������@�������???��������?? ���������  �������@@�������``�������??����P�?R������ ������������ ����� ����� �����@����� �����`����� �����?����`�? ��������������p�?>��������������������������������������������������?�?>��������������������������������������������������� ?���??���?���������?~���?(���� ?�������������������� ?����������??���� ?������ ��?� ?0������?�?{e?����?��?���?��?��?� ���?���???P�?�����? ��������\�� ?T�����?���������? ���Click to edit Master title style��?���!�������? ���!���������?��� ?������ ��s� ?*������?�?~e?��?���?��?��?� ���?���???��?�����? �������\�� ??�����?��������?R���Click to edit Master text styles Second level Third level Fourth level Fifth level��?���!����� ���� ���� ���� ������? ���S���������??���� ?������ ��s� ?*������?�??e?��?���?��?��?� ���?���`?`?�?�����? ������\�� ?P�����?��������?���*���?���������������������?���������? ������������??���� ?������ ��s� ?*������?�??e?��?���?��?��?� ���?���`???�?�����? ������ \�� ?R�����?��������?���*���?���������������������?���������? ������������??���� ?������ ��s� ?*������?�p?e?��?���?��?��?� ���?���` ??�?�����? ������\�� ?R�����?��������?���*���?���������������������?���������? ������������?H���� ?������ ��?� ?0���?���?��????�???h�?��?��� ���?���? ���???�����???�����???�33?��??�??���?8����?0�����?���_�_�_�P�P�T�1�0���?�����?.���??s��?U �?$���B�l�a�n�k� �P�r�e�s�e�n�t�a�t�i�o�n��?F���?����������������?�����?P� ?���??�� �?���'���4���?F���?(���� ?���ppa.a.eloiduroC.� ?����������??���"� ?������ ��s� ?*���?�? f?����?����?�??�?��?��?����?���?0` ?� ?4�����?��������?���PSET��?��������������������??���"� ?��� ��� ��s� ?*���?�? f?����?����?�??�?��?��?����?���?? ? ?� ?>�����?��������?���Class Indexing��?��������������������??���"� ?��� ��� ��s� ?*���?�?f?����?����?�??�?��?��?����?���? ??� ?@�����?��������?���Persistent Slots��?��������������������??���"� ?��� ��� ��s� ?*���?�0f?����?����?�??�?��?��?����?���???�� ?:�����?��������?���������������������? �����������??���"� ?������ ��s� ?*���?�pf?����?����?�??�?��?��?����?���P??�� ?:�����?��������?���������������������? �����������?L���� ?������ ��c� ?$���?����?����?�??�?��?��?�����?���.???�??���? ?������ ��s� ?*���?�? f?���?��?���?���?��?�����?���[?w{� ?G�����?��������?���BTree��?��������� ��2����������? �����������??���"� ?������ ��s� ?*���?� %f?����?����?�??�?��?��?����?���` p ?� ?L�����?��������? ���Serializer��?��� ��������� ���������? ��� ��������??���"� ?������ ��s� ?*���?��*f?����?����?�??�?��?��?����?���`? ??� ?O�����?��������? ���Memutils/UFFI��?���������������������? �����������??���"� ?������ ��s� ?*���?�?.f?����?����?�??�?��?��?����?���P  �� ?3�����?��������?���MOP��?��������������������??���? ?������ ��s� ?*���?�03f?���?��?���?���?��?�����?���`�� ??� ?8�����?��������?���User API��?��� ������ ��2� ��������?R���B ?������ ��s� ?*���?����?����D������?���?��?����?���?P??�??���? ?������ ��s� ?*���?�?8f?���?��?���?���?��?�����?���?H? � ?R�����?��������?"���Data Store Interface and Utilities��?���#������ ��2�#��������??���"� ?������ ��s� ?*���?�=f?����?����?�??�?��?��?����?���? ? � ?H�����?��������?���db-bdb��?���������������������? �����������??���"� ?������ ��s� ?*���?�?Af?����?����???f�?��?��?����?���? p� ? � ?=�����?��������? ���db-postmodern��?��������������������??���"� ?������ ��s� ?*���?�0Ff?����?����?�??�?��?��?����?���? ` ?? � ?J�����?��������?���db-clsql��?��� ��������� ���������? ��� ��������??���"� ?������ ��s� ?*���?�Kf?����?����??ff�?��?��?����?���? ??? � ?7�����?��������?���db-lisp��?��������������������??���"� ?��� ��� ��s� ?*���?�pOf?����?����????�?��?��?����?���? p??� ?L�����?��������? ���BerkeleyDB��?��� ��������� ���������? ��� ��������??���"� ?���!��� ��s� ?*���?��Tf?����?����????�?��?��?����?���? ?? ?� ?L�����?��������? ���PostgreSQL��?��� ��������� ���������? ��� ��������??���"� ?���"��� ��s� ?*���?�?Xf?����?����????�?��?��?����?���? @??� ?H�����?��������?���SQLite��?���������������������? �����������?X���B ?���#��� ��?� ?0���?����?����D������?���?��?���?����?���? 0?? �?X���B ?���$��� ��?� ?0���?����?����D������?���?��?���?����?���? ?0 ? �?X���B ?���%��@ ��?� ?0���?����?����D������?���?��?���?����?���? 0 P ? �?X���B ?���&��� ��?� ?0���?����?����D������?���?��?���?����?���? ??? �?R���B ?���'��� ��s� ?*���?����?����D������?���?��?����?���?P??�?R���B ?���(��� ��s� ?*���?����?����D������?���?��?����?���0 P?0 �?R���B ?���)��� ��s� ?*���?����?����D������?���?��?����?���0 P?0 �??���"� ?���*��� ��s� ?*���?�?df?����?����???f�?��?��?����?���?? @?� ?3�����?��������?���DCM��?��������������������?L���� ?���+��� ��c� ?$���?����?����?�??�?��?��?����?���g???�??��? ?���,��� ��?� ?6���?�?mf?����?����?���?��?���?���?��?�����?���0?Z�??���??���?8�����?���_�_�_�P�P�T�1�0���?�����?�����������������?F�����?���_�_�_�P�P�T�9���?(�����? ���������������������������������?F����?���_�_�_�P�P�T�M�a�c�1�1���? ��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ���������� ?B�����?��������?���User API��?��� ������������������������?L���� ?���-��� ��c� ?$���?����?����?�??�?��?��?����?���C???�??��? ?���.��� ��?� ?6���?��wf?����?����?���?��?���?���?��?�����?��� ??�??���??���?8�����?���_�_�_�P�P�T�1�0���?�����?�����������������?F�����?���_�_�_�P�P�T�9���?(�����? ���������������������������������?F����?���_�_�_�P�P�T�M�a�c�1�1���? ��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ���������� ?B�����?��������?���Internal��?��� ������������������������?L���� ?���/��� ��c� ?$���?����?����???f�?��?��?����?���g0 @ ?�??��? ?���0��� ��?� ?6���?��}f?����?����?���?��?���?���?��?�����?���0? �??���??���?8�����?���_�_�_�P�P�T�1�0���?�����?�����������������?F�����?���_�_�_�P�P�T�9���?(�����? ���������������������������������?F����?���_�_�_�P�P�T�M�a�c�1�1���? ��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ���������� ?L�����?��������?���Ready to integrate��?���������������������������?L���� ?���1��� ��c� ?$���?����?����??ff�?��?��?����?���C0 @ ?�??��? ?���2��� ��?� ?6���?�??f?����?����?���?��?���?���?��?�����?��� ? ?�??���??���?8�����?���_�_�_�P�P�T�1�0���?�����?�����������������?F�����?���_�_�_�P�P�T�9���?(�����? ���������������������������������?F����?���_�_�_�P�P�T�M�a�c�1�1���? ��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ���������� ?=�����?��������?���TBD��?���������������������������?L���� ?���3��� ��c� ?$���?����?����????�?��?��?����?���`???�??��? ?���4��� ��?� ?6���?� ?f?����?����?���?��?���?���?��?�����?���)??�??���??���?8�����?���_�_�_�P�P�T�1�0���?�����?�����������������?F�����?���_�_�_�P�P�T�9���?(�����? ���������������������������������?F����?���_�_�_�P�P�T�M�a�c�1�1���? ��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ���������� ?C�����?��������? ���3rd party��?��� ��������� ���������������?H���� ?������ ��?� ?0���?���?��????�???h�?��?��� ���?���? ���???�����???�����???�33?��??�??���?8����?0�����?���_�_�_�P�P�T�1�0���?�����?.���??s�???V�c������x??p^????R??3???? lHb?P  @???AL ?G?????@;b `B???&V?????�??�??��??�?�????�??�??��??�?�????�??�??��??�?�??�D????�???� ??��?�??�?�????�???� ??��?�??�?�????�???� ??��?�??�?�??�J????�??�??�??��??�??�????�??�??�??��??�??�????�??�??�??��??�??�??�5?????�?�??��??��?????�?�??��??��?????�?�??��??��??� ????????????� ????????????�"????�??�?��???�??�?��???�??�?��???� ????????????�.????????�??????"?�#???????�???�??????g ?�j????7?� ~????g ?� ~????k?� ????�d?�G???k?�{????@?�  ?????k?�  ?????k?� ????q�??????Q???i ?�r????;?� ?????i ?� ?????h ?�????j�h~??~iL??�??????�D??k?�g???�g???/??????0k????��????/?????H??rS??? ??0k???/???? ??0k???/??????0k???????�????n~??} F??n,}?? ~ni???n~?? ~ni???Xe?feXh???~??~�???�?????�?�??�?� ???� ??!???????��??!????�????????!??????!??????#????#??n??p�??o??n ??n??n ??X?fX ??i??i??�?????�?��??�?�???�?????????��??????�??��??��???????????????�?????????????????~???�??��??��??~??~??~??~???�??~??e?fe??~??~??�??????�???�???�??�??�???�??��??�???�??��??��??�??�??��???�??�??�??�??�??��??��??�???�??��??��??�??�??��???�??�??�?f�??�??��??�??????�???�???�??�??�???�??��???��??��??��??�?��??��??��??�??�??��?? �?��?��?��??�??�??�??�??��??��???��??��??��??�?��??��??��??�??�??��?? �?��?��?��??�??�?f�??�??��??�??????�???�???�??�???�??�??�???�??��??��???�?��??��??��???��??�??��??�??�???�??�??�???�??�??�??�???�??��??��??��??��???�?��??��??��???��??�??��??�??�???�??�??�?f?�ff�?f�??�???�??��??��??�??????�???�???�??�??�??��??��?��??�???�??��??��??��??�??�?�??�??��??�??�??�??�??�??��??��?��??�??�??�?�??�??�??��??��??��??��??�??�?�??�??��??�??�??�??�?f�ff��?f��f��?f�??�??�?�??�??�??��??�??????�???�???�??�??�??��???�?�??�???�??��??��??��???�???�??��??�??�??�??�??�??��???�?�??�??�??��?�?�??�??��??��??��??��???�???�??��??�??�??�??�?f�ff��?f?�f�?f�??�??��?�?�??�??��??�??????�???�???�??�???�???�?�?�??�???�??��??��??��??��??��??�??�??�??�??�???�???�?�?�??�??�???�???�??��??��??��??��??��??��??�??�??�??�??�?f?�ff?�f�f�?f�??�???�???�??��??�??????�???�???�??�??�???�??��???�?�??��??��??��??�??�??��??��??�??�??�??�??��??��???�?�??��??��??��??�??�??��??��??�??�?f�??�??��??�??????�???�???�??�??�???�??��??�???�??��??��??�??�???��??�??�??�??�??�??��??��??�???�??��??��??�??�???��??�??�??�?f�??�??��??�??????�???�???�??�??�???�??��???�?��?��??��??��??�??�??�?��??� ?�?��?��?�??�??�??�??�??��??��???�?��?��??��??��??�??�??�?��??� ?�?��?��?�??�??�?f�??�??��??�??????�???�???�??�??�???�??��??��??�??��??��??�??�??�??��??�?��?�??�??�??�??�??��??��??��??�??��??��??�??�??�??��??�?��?�??�??�?f�??�??��??�??????��???�???�??�???????�??�??????�??????�????�?????�??????�??��??�}??}??�}??}??�}???�??}??�d?fd??�??��??�??????�5??�?�:???�:??Q??????D???�??Q?????8@??�8??? ??D??Q???? ??D??Q??????D??�??��??+o~??}:??�}?? ~o ???+o~?? ~o ???$Ye?feY>??�??��??�??????k?��????5?� �D????k?� �D????r?��H????�???r?��??????� �H????r?� �H????r?��H???�??��???n?��????:?� �F????n?� �F????m?��E???�??��??�???????�????��??��????��??��??�???????�????��??????��??�???????�????��??????��??�;????�??�5?????���??�??�:?????�??��??�??�@?????�??��??�B????5?��??5?�??:}??}�???8????7�???8????7�??:}??}�??�8????�?�???���??}??}�??????�??????�??}??}�??�.?????�???���??��??�??�??��??��??�??�??��??�@?????�???���??��??��???�??�??�??��??��??��???�??�??�??��??�L?????�???���??��?? ��?��??��??�??�??�??��??��?? ��?��??��??�??�??�??��??�D?????�???���??��???�?�???�??�??�??��??��???�?�???�??�??�??��??�@?????�???���??��??�?�??�??�??�??��??��??�?�??�??�??�??��??�.?????�???���??��??�??�??��??��??�??�??��??�.?????�???���??��??�??�??��??��??�??�??��??�.?????�???���??��??�??�??��??��??�??�??��??�7?????�???��??�}??}??�??????�??????�}??}??�D????�?�5??�?��5??�}??}:??�8????6@??�8????8@??�}??}:??�=?????5?�????5?��????:?�??????��??????�????:?��???� ????????????� ????????????�:????�??��L??�f?�(???�??��O??�j?�-???�??��S??�o?�2???�6?????�???�??�k??|�??�|??~U�??�????�??�?????�??�,?????�???�??��??��??��??��??��??��??��??��??�x?????�???�??��???�??��??��??��??��??�??�?�??��???�?��??�??��??��???�??��??��??��??��??�??�?�??��???�?��??�??��??�t?????�???�??��??�???��??�??��??��??��??��??�??��??�??��??��??��??�???��??�??��??��??��??��??�??��??�??��??��??�??????�???�??��???��??�?�??�??��??��???��??�?�?�??��??�???�?��??�??��??��???��??�?�??�??��??��???��??�?�?�??��??�???�?��??�??��??�z?????�???�??��??��??��??�??��??��??�?�??�?�??��??�??�??��??��??��??��??��??�??��??��??�?�??�?�??��??�??�??��??��??�4????�?�???��??}??��??��??}?????�??�?????�;????�(?�???�5??-T??v�??�v??}:??2??????�????8@??�4????�??��q??�`?��???�??��u??�b?��???�??��y??�f?��???� ????????????� ????????????�"????�??�?��???�??�?��???�??�?��???� ????????????� ????????????�|?????�??��???�?�??�??�??��?? �?�??��?�?�???�??��???�?�??�??�??��?? �?�??��?�?�???�??��???�?�??�??�??��?? �?�??��?�?�??�d????�???�??�???�???��??�?�???�??�???�??�???�???��??�?�???�??�???�??�???�???��??�?�???�??�?????�??�??�?��???��??�??�??��??�???�???�?��?�?�???�?��??�??�??�?��???��??�??�??��??�???�???�?��?�?�???�?��??�??�??�?��???��??�??�??��??�???�???�?��?�?�???�?��??�|?????��??�??��??�??�??��??��??��??�???�???��??�??��??�??�??��??��??��??�???�???��??�??��??�??�??��??��??��??�???�??� ????????????� ????????????� ????????????� ????????????�"????�??�?��???�??�?��???�??�?��???� ????????????� ????????????�??????D�?�k???�??� r????D�?� k????H�?�r????F�?�n???�??� r????F�?� n????E�?�m????H�?�r???�??� m????H�?� r????E�?�m???�?????:?�A???8?????SK??:?�A??C??????SK??> p~??~p,F???8??? ??SK??> p~?? ~p,F??= Ye?feY%D??C??????SK??9d?f eY%D??C???? ??SK??= Ye?feY%D??�o????�?�??????�???�??????�??}??}�??????�??}??}�??d?fd�??????�??d?fd�??????�??d?fd�??�X?????�??��??�???�??�??��??��??��??��??�??�??�??�?f��??��??��??��?f�??�??�??�?f��??�X?????�??��??�???�??�??��??��??��??��??�??�??�??�?f��??��??��??��?f�??�??�??�?f��??�X?????�??��??�???�??�??��??��??��??��??�??�??�??�?f��??��??��??��?f�??�??�??�?f��??�??????�??��??��??��??��??�???�??�??��??��??��??��??��??��??��??��??��??��??��??��??�??�??��??��??��??�??�?f��?f��?f��??��??��??��??��??��??��?f��?f��?f��?f�??�??��??��??��??�??�?f��?f��?f��??�??????�??��???��??�??�???�??�???��??�??��??��???��??�??��??��???��??�??�??�???��??�??�??�?f?��f?�?f��??��???��??�??��??��?f?��f?�?f�??�???��??�??�??�?f?��f?�?f��??>?????�??��??�?��??�?��?�??�?�?�?��??�?�??�???�??�??�?��??��??�?�??��??��??�?��??�?��?��?�??��??��??�?��??�?��?�??�?�?�?��??�?�??�??�??�?��??�??�??�?��??�??�?f�f��f?��f?�f�?f��??��??�?��??�?��?��?�??��??��?f�f��f?�f��f�f?�f�f�f��f?�f�?f�??�??�?��??�??�??�?��??�??�?f�f��f?��f?�f�?f��??�??????�??��???�?�??�?�??�?�??�???�??�???��??�?�??��??��???��??�??��??��???�?�??�?�??�?�??�??�???��??�?��??�??�?f?��f?�f�?f��??��???��??�??��??��?f?�f�f?�f�f?�f�?f�??�???��??�?��??�??�?f?��f?�f�?f��??�??????�??��??��??�???�??�??��??��??��??��??��??��??�??�??��??�??�?f��?f��??��??��??��?f��?f�??�??��??�??�?f��?f��??�X?????�??��??�???�??�??��??��??��??��??�??�??�??�?f��??��??��??��?f�??�??�??�?f��??�X?????�??��??�???�??�??��??��??��??��??�??�??�??�?f��??��??��??��?f�??�??�??�?f��??�r????�?��?????????�????????~??~????????~??~??e?fe????????e?fe????????e?fe??�r????� ?��??�????!?? ?�??????!?? n??n??�????!?? n??n?? X?fX??????!??�Y?fX??????!?? X?fX??�?????h?�~??H?????0???h?�~??l/??????0???jn~??~n???H??? ??0???jn~?? ~n???iXe?feX???l/??????0???Ed?f eX???l/???? ??0???iXe?feX???�???????� h????C?� l?????�  h?????!?�l??????� j????C?� l??????�  j??????� i?????!?�l????=?�  i?????!?� l??????� i???�I????????���������������������������������������������������������������������������????????????��������������������������������������������������������������������������������������������������������������������????????????��������������������������������������������������������������������������������������������������������������������????????????�������������������������������������������������R����������������������������������������������������������������????????���??d?O???�?�?)?������������?s?#??���?������P�o�w�e�r�P�o�i�n�t� �D�o�c�u�m�e�n�t���������������������������(�������????��������������������������������������� 5�������S�u�m�m�a�r�y�I�n�f�o�r�m�a�t�i�o�n���������������������������(����????????���������������������������������������t1�������D�o�c�u�m�e�n�t�S�u�m�m�a�r�y�I�n�f�o�r�m�a�t�i�o�n�����������8�????????????����������������������������������������������????????������������������ ��� ��� ��� ��� ������������������������������������������������????????>��� ���!���"���#���$���%���&���'���(���)���*���+���,���-���.���/���0���1���2���3���4���5���6���7���????????????<���????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????C�u�r�r�e�n�t� �U�s�e�r�������������������������������������������????????????������������������������������������ ���>���������������������������������������������������������������������������????????????��������������������������������������������������������������������������������������������������������������������????????????��������������������������������������������������������������������������������������������������������������������????????????������������������������������������������������������������������������????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????����������������������������������������������������������������?"������_????4�� �?�??Ian Eslick���I�a�n� �E�s�l�i�c�k��������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������� From ieslick at common-lisp.net Sun Apr 22 03:02:35 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 23:02:35 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070422030235.E52977B131@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv10154 Modified Files: testconditions.lisp Log Message: Fixed bug when two stores not set in new test --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/21 17:26:37 1.1 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/22 03:02:35 1.2 @@ -18,19 +18,18 @@ (null *test-spec-secondary*)) (progn (format t "~%Second store spec missing: ignoring") - t) + (values t t t t)) (let (*store-controller* (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect nil)) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil))) (unwind-protect - (progn - (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) - (inst2 (make-instance 'pfoo :slot1 200 :sc sc2))) - (values - (is-not-null (add-to-root 'inst1 inst1 :sc sc1)) - (is-not-null (add-to-root 'inst2 inst2 :sc sc2)) - (signals-condition (add-to-root 'inst1 inst1 :sc2)) - (signals-condition (add-to-root 'inst2 inst2 :sc1))))) + (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) + (inst2 (make-instance 'pfoo :slot1 200 :sc sc2))) + (values + (is-not-null (add-to-root 'inst1 inst1 :sc sc1)) + (is-not-null (add-to-root 'inst2 inst2 :sc sc2)) + (signals-condition (add-to-root 'inst1 inst1 :sc2)) + (signals-condition (add-to-root 'inst2 inst2 :sc1)))) (close-store sc1) (close-store sc2)))) t t t t) \ No newline at end of file From ieslick at common-lisp.net Sun Apr 22 03:35:09 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 23:35:09 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070422033509.E48ED431BF@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv15141/doc Modified Files: user-guide.texinfo Log Message: Provide simple restarts for some potentially annoying error conditions. Especially storing an instance of one store in another database --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/21 17:22:35 1.12 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/22 03:35:09 1.13 @@ -433,14 +433,16 @@ @subsection Instance Creation -To the user, persistent objects are created as usual with a call to +To the user, persistent objects are created normally with a call to @code{make-instance} with the desired slot initargs as illustrated in the last section. However, this call will fail unless there is a default @code{store-controller} instance in the variable @code{*store-controller*} or the @code{:sc} keyword argument is -provided with a valid store controller object. This tells the -metaobject protocol implementing persistent slot creation where to -allocate storage for this instance's slots and +provided with a valid store controller object. + +An instance of the class is created and any initforms and initarg +values are written to transient or persistent slots as one would +expect. * Default store controller & instance creation * What happens to persistent objects when store-controller is closed? From ieslick at common-lisp.net Sun Apr 22 03:35:10 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 21 Apr 2007 23:35:10 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070422033510.7014E4507D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv15141/src/elephant Modified Files: classes.lisp controller.lisp package.lisp serializer2.lisp Log Message: Provide simple restarts for some potentially annoying error conditions. Especially storing an instance of one store in another database --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/21 17:22:50 1.28 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/22 03:35:09 1.29 @@ -260,16 +260,16 @@ "Ensures that object can be written as a reference into store sc" (eq (dbcn-spc-pst object) (controller-spec sc))) -(define-condition cross-store-reference () - ((object :accessor cross-store-reference-object :initarg :object) - (home-controller :accessor cross-store-reference-home-controller :initarg :home-ctrl) - (foreign-controller :accessor cross-store-reference-foreign-controller :initarg :foreign-ctrl)) +(define-condition cross-reference-error () + ((object :accessor cross-reference-error-object :initarg :object) + (home-controller :accessor cross-reference-error-home-controller :initarg :home-ctrl) + (foreign-controller :accessor cross-reference-error-foreign-controller :initarg :foreign-ctrl)) (:documentation "An error condition raised when an object is being written into a data store other than its home store")) -(defun raise-cross-store-condition (object sc) - (cerror "Proceed and patch later" - 'cross-store-reference +(defun signal-cross-reference-error (object sc) + (cerror "Proceed to write incorrect reference" + 'cross-reference-error :format-control "Attempted to write object ~A with home store ~A into store ~A" :format-arguments (list object (get-con object) sc) :object object --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/21 17:22:50 1.47 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/22 03:35:09 1.48 @@ -32,8 +32,8 @@ ) "Tells the main elephant code the tag used in a store spec to refer to a given data store. The second argument is an asdf - dependency list. Entries have the form of (data store type - asdf-depends-list") + dependency list. Entries have the form of + (data-store-type-tag asdf-depends-list") (defvar *elephant-controller-init* (make-hash-table)) @@ -57,17 +57,29 @@ we re-open the controller from the spec if it's not cached? That might be dangerous so for now we error")) +(define-condition controller-lost-error () + ((object :initarg :object :accessor store-controller-closed-error-object) + (spec :initarg :spec :accessor store-controller-closed-error-spec))) + +(defun signal-controller-lost-error (object) + (cerror "Open a new instance and continue?" + 'controller-lost-error + :format-string "Store controller for specification ~A for object ~A cannot be found." + :format-arguments (list object (dbcn-spc-pst object)) + :object object + :spec (dbcn-spc-pst object))) + (defmethod get-con ((instance persistent) &optional (sc *store-controller*)) (declare (ignore sc)) (let ((con (gethash (dbcn-spc-pst instance) *dbconnection-spec*))) (cond ((not con) - ;; ISE NOTE: Create a new one here & warn instead? - ;; (get-controller spec) - (error "Object's store controller was lost")) + (progn (signal-controller-lost-error instance) + (open-controller + (get-controller (dbcn-spc-pst instance))))) ;; If it's valid and open ((and con (connection-is-indeed-open con)) con) - ;; If the object exists but is closed, reopen + ;; If the controller object exists but is closed, reopen (t (open-controller con))) con)) --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/21 17:22:50 1.31 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/22 03:35:09 1.32 @@ -266,7 +266,8 @@ #:struct-constructor ;; Various error conditions - #:cross-store-reference + #:cross-reference-error + #:controller-lost-error #:map-class-query #:get-query-instances --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/21 17:22:51 1.39 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/22 03:35:09 1.40 @@ -42,7 +42,7 @@ database-version translate-and-intern-symbol valid-persistent-reference-p - raise-cross-store-condition)) + signal-cross-reference-error)) (in-package :elephant-serializer2) From ieslick at common-lisp.net Mon Apr 23 02:26:54 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 22 Apr 2007 22:26:54 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070423022654.5E58950019@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv18051/src/elephant Modified Files: classindex.lisp collections.lisp controller.lisp metaclasses.lisp Log Message: :from-end option for map-index and simple test; better error handling and argument checking --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/12 02:47:32 1.36 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/23 02:26:53 1.37 @@ -394,11 +394,11 @@ (declare (dynamic-extent map-fn)) (map-btree #'map-fn class-idx)))) -(defun map-class-index (fn class index &rest args &key start end value) - "This function maps over a subset of class instances in the - order defined by the index. Specify the class and index by - quoted name. The index may be a slot index or a derived - index. +(defun map-class-index (fn class index &rest args &key start end value from-end) + "map-class-index maps a function of two variables, taking key + and instance, over a subset of class instances in the order + defined by the index. Specify the class and index by quoted + name. The index may be a slot index or a derived index. To map only a subset of key-value pairs, specify the range using the :start and :end keywords; all elements greater than @@ -410,17 +410,20 @@ element or last element, respectively. To map a single value, iff it exists, use the :value keyword. - This is the only way to travers all nil values." + This is the only way to travers all nil values. + + To map from :end to :start in descending order, set :from-end + to true. If :value is used, :from-end is ignored" (declare (dynamic-extent args) (ignorable args)) (let* ((index (if (symbolp index) (find-inverted-index class index) index))) (flet ((wrapper (key value pkey) - (declare (ignore key pkey)) - (funcall fn value))) + (declare (ignore pkey)) + (funcall fn key value))) (declare (dynamic-extent wrapper)) - (map-index #'wrapper index :start start :end end :value value)))) + (map-index #'wrapper index :start start :end end :value value :from-end from-end)))) ;; ================= --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/19 05:24:37 1.22 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/23 02:26:53 1.23 @@ -382,7 +382,7 @@ (funcall fn k v) (return nil))))))))) -(defgeneric map-index (fn index &rest args &key start end value) +(defgeneric map-index (fn index &rest args &key start end value from-end) (:documentation "Map-index is like map-btree but for secondary indices, it takes a function of three arguments: key, value and primary key. As with map-btree the keyword arguments start and end @@ -393,47 +393,75 @@ use the value keyword which will override any values of start and end.")) -(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p)) +(defun pprev-hack (cur) + "Get the first duplicate instance of the prior value off the current cursor" + (let ((e? (cursor-pprev-nodup cur))) + (when e? + (let ((e? (cursor-pprev-nodup cur))) + (if e? + (cursor-pnext cur) + (cursor-pfirst cur)))))) + +(defun cursor-last-range-hack (cur) + "Get the first duplicate instance of the last value of the cursor's index" + (let ((e? (cursor-plast cur))) + (when e? + (let ((e? (cursor-pprev-nodup cur))) + (if e? + (cursor-pnext cur) + (cursor-pfirst cur)))))) + + +(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p) from-end) (declare (dynamic-extent args) (ignorable args)) + (unless (lisp-compare<= start end) + (error "map-index called with start = ~A and end = ~A. Start must be less than or equal to end according to elephant::lisp-compare<=." + start end)) (let ((sc (get-con index)) (end (or value end))) (ensure-transaction (:store-controller sc) (with-btree-cursor (cur index) - (labels ((next-range () - (multiple-value-bind (exists? skey val pkey) (cursor-pnext-nodup cur) - (if (and exists? - (or (null end) - (lisp-compare<= skey end))) - (progn - (funcall fn skey val pkey) - (next-in-range skey)) - (return-from map-index nil)))) - (next-in-range (key) - (multiple-value-bind (exists? skey val pkey) (cursor-pnext-dup cur) - (if exists? + (labels ((continue-p (key) ;; Do we got to the next value? + (or (if from-end (null start) (null end)) + (if from-end + (or (not (lisp-compare<= key start)) + (lisp-compare-equal key start)) + (lisp-compare<= key end)))) + (value-increment () ;; Step to the next key value + (if from-end + (pprev-hack cur) + (cursor-pnext-nodup cur))) + (next-value () ;; Handle the next key value + (multiple-value-bind (exists? skey val pkey) + (value-increment) + (if (and exists? (continue-p skey)) (progn (funcall fn skey val pkey) - (next-in-range key)) - (progn - (cursor-pset-range cur key) - (next-range)))))) - (declare (dynamic-extent next-range next-in-range)) + (map-duplicates skey)) + (return-from map-index nil)))) + (map-duplicates (key) ;; Map all duplicates for key value + (loop as (exists? skey val pkey) = (multiple-value-list (cursor-pnext-dup cur)) + while exists? do (funcall fn skey val pkey)) + (cursor-pset-range cur key) + (next-value))) + (declare (dynamic-extent next-value next-value-increment continue-p map-duplicates)) (multiple-value-bind (exists? skey val pkey) (cond (value-set-p (cursor-pset cur value)) - ((null start) + ((and (not from-end) (null start)) (cursor-pfirst cur)) - (t (cursor-pset-range cur start))) - (if (and exists? - (or (null end) - (lisp-compare<= skey end))) + ((and from-end (null end)) + (cursor-last-range-hack cur)) + (t (if from-end + (cursor-pset-range cur end) + (cursor-pset-range cur start)))) + (if (and exists? (continue-p skey)) (progn (funcall fn skey val pkey) - (next-in-range skey)) + (map-duplicates skey)) nil))))))) - ;; =============================== ;; Some generic utility functions ;; =============================== --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/22 03:35:09 1.48 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/23 02:26:53 1.49 @@ -64,7 +64,7 @@ (defun signal-controller-lost-error (object) (cerror "Open a new instance and continue?" 'controller-lost-error - :format-string "Store controller for specification ~A for object ~A cannot be found." + :format-control "Store controller for specification ~A for object ~A cannot be found." :format-arguments (list object (dbcn-spc-pst object)) :object object :spec (dbcn-spc-pst object))) --- /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/12 02:47:32 1.16 +++ /project/elephant/cvsroot/elephant/src/elephant/metaclasses.lisp 2007/04/23 02:26:53 1.17 @@ -32,6 +32,10 @@ (:documentation "Abstract superclass for all persistent classes (common to both user-defined classes and Elephant-defined objects such as collections.)")) +(defmethod print-object ((obj persistent) stream) + "This is useful for debugging and being clear about what is persistent and what is not" + (format stream "#<~A oid:~A>" (type-of obj) (oid obj))) + (defclass persistent-metaclass (standard-class) ((%persistent-slots :accessor %persistent-slots) (%indexed-slots :accessor %indexed-slots) From ieslick at common-lisp.net Mon Apr 23 02:26:54 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 22 Apr 2007 22:26:54 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070423022654.C6A888307D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv18051/tests Modified Files: testcollections.lisp Log Message: :from-end option for map-index and simple test; better error handling and argument checking --- /project/elephant/cvsroot/elephant/tests/testcollections.lisp 2007/04/06 02:51:50 1.20 +++ /project/elephant/cvsroot/elephant/tests/testcollections.lisp 2007/04/23 02:26:54 1.21 @@ -325,6 +325,19 @@ 10945 ;; sum 990 to 1000 inclusive )) +(deftest map-index-from-end + (let ((sum 0)) + (flet ((collector (key value pkey) + (incf sum (slot1 value)))) + (map-index #'collector index1 :start nil :end 10 :from-end t) + (map-index #'collector index1 :start 990 :end nil :from-end t) + (map-index #'collector index1 :start 400 :end 410 :from-end t)) + sum) + #.(+ 49 ;; sum 4-10 inclusive (1-3 removed by here) + 4455 ;; sum 690-700 inclusive + 10945 ;; sum 990 to 1000 inclusive + )) + (deftest rem-kv (with-transaction (:store-controller *store-controller*) (let ((ibt (make-indexed-btree *store-controller*))) From ieslick at common-lisp.net Mon Apr 23 02:41:13 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 22 Apr 2007 22:41:13 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070423024113.9115B3A01C@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv21337/src/elephant Modified Files: collections.lisp serializer2.lisp Log Message: Fix condition handler; map-index error checking --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/23 02:26:53 1.23 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/23 02:41:11 1.24 @@ -415,7 +415,7 @@ (defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p) from-end) (declare (dynamic-extent args) (ignorable args)) - (unless (lisp-compare<= start end) + (unless (or (null start) (null end) (lisp-compare<= start end)) (error "map-index called with start = ~A and end = ~A. Start must be less than or equal to end according to elephant::lisp-compare<=." start end)) (let ((sc (get-con index)) --- /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/22 03:35:09 1.40 +++ /project/elephant/cvsroot/elephant/src/elephant/serializer2.lisp 2007/04/23 02:41:11 1.41 @@ -201,7 +201,7 @@ (serialize-string frob bs)) (persistent (unless (valid-persistent-reference-p frob sc) - (raise-cross-store-condition frob sc)) + (signal-cross-reference-error frob sc)) (buffer-write-byte +persistent+ bs) (buffer-write-int32 (oid frob) bs) ;; This circumlocution is necessitated by From rread at common-lisp.net Mon Apr 23 03:40:39 2007 From: rread at common-lisp.net (rread) Date: Sun, 22 Apr 2007 23:40:39 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070423034039.4E56E33080@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv1153 Modified Files: ArchDiagram1.ppt Log Message: My take on a talk Binary files /project/elephant/cvsroot/elephant/doc/ArchDiagram1.ppt 2007/04/21 18:48:29 1.1 and /project/elephant/cvsroot/elephant/doc/ArchDiagram1.ppt 2007/04/23 03:40:39 1.2 differ From ieslick at common-lisp.net Mon Apr 23 04:58:17 2007 From: ieslick at common-lisp.net (ieslick) Date: Mon, 23 Apr 2007 00:58:17 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070423045817.17B064051@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv21560 Added Files: IntroTalk1.ppt Log Message: Expand on talk sketch --- /project/elephant/cvsroot/elephant/doc/IntroTalk1.ppt 2007/04/23 04:58:15 NONE +++ /project/elephant/cvsroot/elephant/doc/IntroTalk1.ppt 2007/04/23 04:58:15 1.1 ??????����������������>��?? �������������������������v������????��������x������?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????��������������������� ��� ��� ��� ��� ��������������������������������������������������������� ���!���"���#���$���%���&���'���(���)���*���+���,���-���.���/���0���1���2���3���4���5���6���7���8���9���:���;���<���=���>���?���@���A���B���C���D���E���F���G���H���I���J���K���L���M���N���O���P���Q���R���S���T���U���V���W���X���Y���Z���[���\���]���^���_���`���a���b���c���d���e���f���g���h���i���j���k���l���m���n���o���p���q���r���s���t���u���?���?????���????z���{���|���}���~������?����R����������������������������������������������������������������????????���??d?O???�?�?)?������������?u?\B??w���@������P�o�w�e�r�P�o�i�n�t� �D�o�c�u�m�e�n�t���������������������������(�������????���������������������������������������?,������S�u�m�m�a�r�y�I�n�f�o�r�m�a�t�i�o�n���������������������������(����????????������������������������������������y���?7�������D�o�c�u�m�e�n�t�S�u�m�m�a�r�y�I�n�f�o�r�m�a�t�i�o�n�����������8�????????????����������������������������������������?�������??S���?(���?��?��?��?���������������������� ?����� �������?j�����?��������?V���h�t�t�p�:�/�/�w�w�w�.�c�o�m�m�o�n�-�l�i�s�p�.�n�e�t�/�p�r�o�j�e�c�t�/�e�l�e�p�h�a�n�t��?\�����?��������?H���h�t�t�p�:�/�/�t�r�a�c�.�c�o�m�m�o�n�-�l�i�s�p�.�n�e�t�/�e�l�e�p�h�a�n�t��??��/�? ���0�?��������?0����?D���A�r�i�a�l���N�e�w� �R�o�m�a�?????0?���??@??????0?����1???������P�?D���-?3? �0??0?0?0?0�� �R�o�m�a�?????0?���??@??????0?����1???����?�P �?D���W�i�n�g�d�i�n�g�s���R�o�m�a�?????0?���??@??????0?����1???�����P0�?D���T�i�m�e�s� �N�e�w� �R�o�m�a�n���??0?���??@??????0?����1???������P��?���?�d���????������? ������.��������? ������� ��@�?n����???��" ��d����?��|��������� �������???������????����?�����  �������@@�������``�������??����� t����?l����?��?��!���?���������������������������������)��� ������������� ������������������������������������ ������������������������������������������������������������������������������������������������������������� ������c� ?$���?�??�?����?��?��?���� �?���???�ff?�@�?�����������?���?�����?����������������?�����??1���?����� ���������������?g�����?�������?4���g���d���g���d����?H���������P???����??�?????????���p�?�������p��p�?������@ ���<�����?4���?���?���?���?���????���?%??����g����??;????????���?g�����?�������?4���;���d���;���d����?H���������P???LK????(???????���p�?�������@ ��p�?������p�����������?u?�??;2N??�??;��<�����?4���d���d���d���d���????���?%??����?u?�??;�����������<�����?4���d���d���d���d���????���?%??����?u?�??;������������??/���??����?���_�_�_�P�P�T�1�0���??���??���?�����������?H�����������������������������������������������������������?����������?`������������������������������������������������������������� ������ ������ ����?��� �������?8�����������������������������������������������?��� �������?(�����������������������������������?����������?@�����������������������������������������������������?����������?@�����������������������������������������������������?��� �������?X������������������������������������������������������������� ������ ����??����?���_�_�_�P�P�T�9���?p���?h���?�����������??�����������������������������������������������������������������������������������������������������������������������������������?����������??����������������������������������������������������������������������������������������������������������������������������������������� �������������� �������������� ��������?��� �������?p�������������������������������������������������������������������������������������������������������?��� �������?P���������������������������������������������������������������������������?����������??���������������������������������������������������������������������������������������������������������������������?����������??���������������������������������������������������������������������������������������������������������������������?��� �������??����������������������������������������������������������������������������������������������������������������������������������������� �������������� ��������?h�����?���_�_�_�P�P�T�2�0�0�1���?D����?<�����?4���������������X��������������?����������??�?0(����?���_�_�_�P�P�T�M�a�c�1�1���? (��@�f������ ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ������������ ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ������x!���?�����������4��������������������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ����������������������?����������?������������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������������������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ������?��� �������l��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y�������� �������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y�������� �������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y�������� �������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y�������� �������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y�������� �������������������� ����������?��� �������4��������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y�������� �������������������� ������������ ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ����������?����������0����������������������������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ����������?����������P������������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ������������ ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ���������������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� ������?��� �������? ����� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �������� ������� ���������� ����������������?�������������������������������������������������������������hnamd���`�������������������� �A�r�i�a�l�������������������&�M�o�n�o�t�y�p�e� �T�y�p�o�g�r�a�p�h�y���������������������������� �����?�? �����?��� �%�O�? �����?��� �=��?����?����������������������?���������?W���Elephant: A Persistent Metaobject Protocol and Object Oriented Database for Common Lisp��?(���X��������� �������M����������������?,���������� ������� ���������������)��������?��������?T��Persistent object protocol persistent-metaclass specializes slot accesses Enables persistent links between objects Enables automatic indexing of class instances by slot value Efficient mapping operators over class, slot values, etc. Simple persistent collection APIs Persistent sets (add, remove, find) Btrees (insert, delete, pred, succ)��??������������?��������"���������H������������������������������� �� �(�������x�������!������������H���� �� ���??�������������������������B���������������5������������������������������?���������������������������������������������������?���������������������?���������?����������� ���?��������??��Transactional architecture Provides support for concurrent transactions Provides full ACID compliance Modular Data Store design The interface to disk storage is abstracted Currently supports Berkeley DB (direct/fast) and CL-SQL (general/slower) for flexible speed, license issues Full support for migration between database instances Migration also provides compaction and garbage collection Runtime database configuration Schema evolution Add/remove slot indexes ��??�����������r�K�������r���������r�������r���������r�*�������r��������-����������������� � ������,�������l�������6�������9���� �� �����$��$����(�(�*����,��,���?>������������������?���������������"���������������?���������?��������������������?��������?����������� ���?��������?����������� ����?��������������������?��������?����������� ���?��������?����������� ����?��� �����������������?���������?���Persistent References�?��������? �������������?��������������������?���������? ���Transactions�?��������? �������������?������������ ��������?���������?���Persistent Sets�?��������? �������������?������������ ��������?���������?���BTrees��? �����������?��������? �������������?������������ ��������?���������?���Fast Binary Object Serializer��?���������� ��������?��������??���All numeric types Strings, symbols and pathnames Full Unicode support Persistent objects Standard objects and structs Arrays, hash-tables and lists��?h���1��������r��������r�N��������r�������� ������������������ ��� �����������������������?���n�����������������������?������������� ��������?���������?���Flexible Indexing Scheme�?��������? ��Don't optimize prematurely...index a slot when you need to Functional indexes provide lots of power; you can index objects by the values returned from arbitrary lisp functions Indexes can be ordered by Any numeric quantity Lexical ordering of strings, symbols and paths��?R���?��������r�D�������r�?���������������������.���� �� ����������?>���z���������������N������������������������������/���������?���������������������?���������?���Broad Platform Support�?��������??���Multiple Lisp environments: SBCL, ACL, LispWorks, OpenMCL Multiple Platforms: Linux, Mac, Windows 32 and 64 bit Multiple data store implementations Berkeley DB and CL-SQL (PostgreSQL and SQLite) Native PostgreSQL and Lisp data store planned��??�����������r��������r���������r�"�������r�$��������r�]�������r����������������������"���� �� �$�������/�������-����������������??������������������ ������� ������������������������������������!���������������<������� ��������������������� ������� ����������������?��� ������������������?���������?���Multiple Data Stores�?��������?��Berkeley DB: fast, not free for a website unless website is open-source Postgres via CLSQL: slower, liberal license, very solid and well-supported SQLite3 via CLSQL: simple, slowest, good for prototyping Easy to migrate your application between them and more data stores are on the way��?R����������� �������=������������9���� �� ������z���������?,���H������� �������z���������������R���������?��� ������������������?���������? ���The Future�?��������?��A� �r�o�b�u�s�t�,� �l�o�n�g�-�t�e�r�m� �s�u�p�p�o�r�t�e�d� �1�.�0� �r�e�l�e�a�s�e� �p�l�a�n�n�e�d� �b�y� �e�n�d� �o�f� �s�u�m�m�e�r� � 0�7� �R�e�f�e�r�e�n�t�i�a�l� �i�n�t�e�g�r�i�t�y� �a�n�d� �f�u�l�l� �s�u�p�p�o�r�t� �f�o�r� �s�c�h�e�m�a� �e�v�o�l�u�t�i�o�n� �(�c�h�a�n�g�i�n�g� �p�e�r�s�i�s�t�e�n�t� �c�l�a�s�s� �s�l�o�t�s�)� � �B�e�t�t�e�r� �s�u�p�p�o�r�t� �f�o�r� �c�l�a�s�s� �h�i�e�r�a�r�c�h�i�e�s� �w�h�e�n� �i�n�d�e�x�i�n�g� �B�e�t�t�e�r� �p�o�s�t�g�r�e�s� �u�s�a�g�e� �w�i�t�h� �p�o�s�t�m�o�d�e�r�n� �O�b�j�e�c�t� �q�u�e�r�y� �l�a�n�g�u�a�g�e� �(�a�n�d� �c�o�m�p�i�l�e�r�?�)� �P�o�s�t� �1�.�0� �f�e�a�t�u�r�e�s� �u�n�d�e�r� �c�o�n�s�i�d�e�r�a�t�i�o�n� �N�a�t�i�v�e� �L�I�S�P� �d�a�t�a� �s�t�o�r�e� � �O�b�j�e�c�t� �p�r�e�v�a�l�e�n�c�e� �o�r� �O�b�j�e�c�t� �m�a�n�a�g�e�m�e�n�t� �(�D�C�M�)���??���G��������r�?�������r�&��������r�E�������r�G�������_�������3�������K���� �� ��������&��������������-���������?b���!���������������?���������������:������� ������� ������� �������b���������������-���������?������������� ��������?���������?���Elephant�?��������??��P�o�i�n�t�e�r�s� �h�t�t�p�:�/�/�w�w�w�.�c�o�m�m�o�n�-�l�i�s�p�.�n�e�t�/�p�r�o�j�e�c�t�/�e�l�e�p�h�a�n�t� �A�v�a�i�l�a�b�l�e� �v�i�a� �C�V�S� �a�n�d� �t�a�r� �a�r�c�h�i�v�e� �(�n�o�t� �a�s�d�f�-�i�n�s�t�a�l�l�a�b�l�e� �t�o�d�a�y�)� �A�d�d� �f�e�a�t�u�r�e� �r�e�q�u�e�s�t�s� �a�n�d� �l�o�g� �b�u�g� �r�e�p�o�r�t�s� �t�o� �t�h�e� �T�R�A�C� �p�r�o�j�e�c�t� �d�a�t�a�b�a�s�e� �a�t� �h�t�t�p�:�/�/�t�r�a�c�.�c�o�m�m�o�n�-�l�i�s�p�.�n�e�t�/�e�l�e�p�h�a�n�t� � �A�u�t�h�o�r�s� �O�r�i�g�i�n�a�l� �E�l�e�p�h�a�n�t� �I�m�p�l�e�m�e�n�t�a�t�i�o�n� �b�y� �A�n�d�r�e�w� �B�l�u�m�b�e�r�g� �a�n�d� �B�e�n� �L�e�e� �N�e�w� �E�l�e�p�h�a�n�t� �V�e�r�s�i�o�n�s� � 0�5�-� 0�7� �b�y� �I�a�n� �E�s�l�i�c�k� �a�n�d� �R�o�b�e�r�t� �R�e�a�d� �N�u�m�e�r�o�u�s� �p�a�t�c�h�e�s�,� �f�e�a�t�u�r�e�s�,� �e�x�a�m�p�l�e�s� �a�n�d� �s�i�g�n�i�f�i�c�a�n�t� �t�e�s�t�i�n�g� �s�u�p�p�o�r�t� �f�r�o�m� �a� �g�r�o�w�i�n�g� �u�s�e�r� �b�a�s�e�!� ���??��� ��������r�?�������r���������r�?�������r���������r� �������+������������?���� �� �n������������������������@���� �� �?����$��$�����(��(���?L��� �������+�������(��������������L����������������������$�������?��������?�����?��������������?��?��� ���4����?�����?��������������?��?���?���?���/�??����?��� ��������������������?��� �������������������?����������������������?����������������������?����������������������?����������������������?����������������������?����������������������?�������������� ��������?�������������� ��������?�������������� ��������?�������������� ��������?��� ����������� ��������?���!������������������P������???���������������������������������������������������������������������?p��?�����?? ���?����������������������^?`�? ���???�����???������??�33?�???�???�`�? �����?�???�����??��??���??�?���???�`�? ���???�����ff3�??��3?3�?����3?�??f�`�? ���???�����333�����???�???�MMM�???�`�? ���???�����???�����??f���?�?�?�???�`�? ���???�����???�����???��f?�?����?��`�? ���???�����???�����3??�???�?�?�???���?t����???��" ��d����?�|��������� �������???������????,����?��?��l?�-�?��������?������� ?�������? �����??����???��" ��d����?��|�??��?��� �������???������???? ����?��?%�� ??? ����?%��" ???@����?%�� ???`����?��?�?���� �?v����???��" �d����?��d�������� �������???����?????? ����?��?�� ? ����?��" ?@����?�� ?`����?��?�?����@�?n����???��" ��d����?��|��������� �������???������????����?�����  �������@@�������``�������??����P�?b������ �����������)����?? ��� ��)����??@��� ��)����??`��� ��)����???��� �`�?<������������������?�����������������?�������� ����p�?t������?1���" �d��?������?1��� �d��?�����?0���" �d�������?0��� �d�������?0���?��d������?�?t������?1���" �d��?������?1��� �d��?�����?0���" �d�������?0��� �d�������?0���?��d������� ?���??���?���������?���?(���� ?�����������??????� ?����������?f���"� ?������ ��?� ?0���?����?����?ff?�?���?ff?�?>?��?�����"?���?����?���?�?�?P�??���� ?������ ��?� ?H������?�`?z?�����?�����?�����?�����?����?���?���?���?� �����?���$???�?�����? ��������J�� ?T�����?���������? ���Click to edit Master title style��?���!�������? ���!���������?(��� ?������ ��?� ?B������?�?|??�����?�����?�����?�����?���?���?���?� �����?���????�?�����? �������J�� ??�����?��������?R���Click to edit Master text styles Second level Third level Fourth level Fifth level��?���!����� ���� ���� ���� ������? ���S���������?��� ?������ ��?� ?B������?�?_??�?_�?�??��?�?_�?�??��?���?���?���?� �����?���`?_�?�����? ��������J�� ??�����?��������?���*���?���������������������?���������? �������������?6������ �����@��?��?��� ��@ ��? ��?�����@��?��?���?��� ?������ ��?� ?B������?�`C??�?_�?�??��?�?_�?�??��?���?���?���?� �����?���`??�?�����? ������� �J�� ??�����?��������?���*���?���������������������?���������? �������������?6������ �����@��?��?��� ��@ ��? ��?�����@��?��?���?��� ?������ ��?� ?B������?�?"3�?�?_�?�??��?�?_�?�??��?���?���?���?� �����?���` ?�?�����? ��������J�� ??�����?��������?���*���?���������������������?���������? �������������?6������ �����@��?��?��� ��@ ��? ��?�����@��?��?���?B���� ?������ ��s� ?*���?����????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???� �?$���B�l�a�n�k� �P�r�e�s�e�n�t�a�t�i�o�n��??���?������?��?�� ���?��P�?���������??���?(���� ?�����������??????� ?����������??���� ?������ ��?� ?H������?��C~?�����?�����?�����?�����?����?���?���?���?� �����?�������??�?�����? ��������J�� ?0�����?��������?����������?����������� ���??���� ?������ ��?� ?B������?�@F~?�����?�����?�����?�����?���?���?���?� �����?���? ?/?�?�����? ��������J�� ?0�����?��������?����������?����������� ���?<���� ?������ ��c� ?$���?����???h�????�?�� ���?���? ���???�����???������??�33?�???�???��?8����?0�����?���_�_�_�P�P�T�1�0���?�����?.���??s��9$?�?b���?������ ���������?����?`��?������������??`�? �����?�����%�� x���?p�� �?������ ���?���?(���� ?�����������??????� ?���� ������??���� ?��� �� ��?� ?Z���?�????�����?�����?�����?�����?�����?����?���?���?����????�?���?�� �?�������?���?�??�?�����? ������� �J�� ?J�����?���������?6������ �����@��?��?��� ��@ ��? ��?�����@��?��?���??���� ?��� �� ��?� ?Z���?�????�����?�����?�����?�����?�����?�����?���?���?����????�?���?�� �?�������?���????�?�����? �������J�� ?F�����?��������?2������ �>��~��?��?��> ��~ ��?��?��>��~��?���?<���� ?��� ��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���?<���� ?���L��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ����������������?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ������������� �#�<�P�E�R�S�I�S�T�E�N�T�-�M�E�T�A�C�L�A�S�S� �F�R�I�E�N�D�>� � �(�m�a�k�e�-�i�n�s�t�a�n�c�e� �'�f�r�i�e�n�d� �:�n�a�m�e� �"�C�a�r�l�o�s�"� �:�b�i�r�t�h�d�a�y� �(�e�n�c�o�d�e�-�b�i�r�t�h�d�a�y� �'�(�1� �1� �1�9�7�2�)�)�)� �(�m�a�k�e�-�i�n�s�t�a�n�c�e� �'�f�r�i�e�n�d� �:�n�a�m�e� �"�A�d�r�i�a�n�a�"� �:�b�i�r�t�h�d�a�y� �(�e�n�c�o�d�e�-�b�i�r�t�h�d�a�y� �'�(�2�4� �4� �1�9�8�0�)�)�)� �(�m�a�k�e�-�i�n�s�t�a�n�c�e� �'�f�r�i�e�n�d� �:�n�a�m�e� �"�Z�a�i�d�"� �:�b�i�r�t�h�d�a�y� �(�e�n�c�o�d�e�-�b�i�r�t�h�d�a�y� �'�(�1�4� �8� �1�9�7�6�)�)�)� � �N�O�W� �S�O�M�E�O�N�E� �T�R�I�P�S� �O�V�E�R� �T�H�E� �P�O�W�E�R� �C�O�R�D�.�.�.� �R�e�s�t�a�r�t� �y�o�u�r� �L�I�S�P� �a�n�d� �d�o�:� � � � � � � �(�g�e�t�-�i�n�s�t�a�n�c�e�s�-�b�y�-�c�l�a�s�s� �'�f�r�i�e�n�d�s�)� �=�>� �(�#�<�C�a�r�l�o�s�>� �#�<�A�d�r�i�a�n�a�>� �#�<�Z�a�i�d�>�)� � � � � � � �(�m�a�p�-�c�l�a�s�s�-�i�n�d�e�x� �#�'�(�f�o�r�m�a�t� �t� �"� �n�a�m�e�:� �~�A� �b�i�r�t�h�d�a�t�e�:� �~�A�~�%�"� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �(�n�a�m�e� �f�r�i�e�n�d�)� �(�b�i�r�t�h�d�a�y� �f�r�i�e�n�d�)�)� � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � �'�f�r�i�e�n�d� � b�i�r�t�h�d�a�y�)� �n�a�m�e�:� �C�a�r�l�o�s� �b�i�r�t�h�d�a�t�e�:� �(�1� �1� �1�9�7�2�)� �n�a�m�e�:� �Z�a�i�d� �b�i�r�t�h�d�a�t�e�:� �(�1�4� �8� �1�9�7�6�)� �n�a�m�e�:� �A�d�r�i�a�n�a� �b�i�r�t�h�d�a�t�e�:� �(�2�4� �4� �1�9�8�0�)� �=�>� �n�i�l���?x���,��������y����� �D������ �?��??������ �"���� �� �"������ ������� �}������ ������� ����� �� ���??���������� �������������� �����������������������������������?��������������?��������������1������� ��������������$�������� ������������������������������ ������� ���������������������������� ����������������?6������ �����@��?��?��� ��@ ��? ��?�����@��?��?���?��? ?������ ��?� ?B���?�?N@?�����?�����?�����?�����?����?���?���?���?�� �?�����?���?�???� ??�����?��������?���Persistent Classes��?��������� ���������,���? �������������?6������ �����@��?��?��� ��@ ��? ��?�����@��?��?���?<���� ?������ ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���??���� ?���$��� ��?� ?<���?�?!,?����?����?���?�??�?����?���?��?�������?���?y-� ?B�����?��������?�����������������$�??������?����������� ���?<���� ?���$��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ������������?0?�����?�����?�����?�����?�����?�����?���?��?����????�?���?� �?�����?������?���????�?�����? �������J�� ?F�����?��������?2������ �>��~��?��?��> ��~ ��?��?��>��~��?���??���� ?���|��� ��?� ?<���?�?K%,?����?����?���?�??�?����?���?��?�������?���?y-� ?B�����?��������?�����������������$�??������?����������� ���?<���� ?���|��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���??���� ?���d��� ��?� ?<���?�???/?����?����?���?�??�?����?���?��?�������?���?y-� ?B�����?��������?�����������������$�??������?����������� ���?<���� ?���d��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���??���� ?���l��� ��?� ?<���?�`?.,?����?����?���?�??�?����?���?��?�������?���?y-� ?B�����?��������?�����������������$�??������?����������� ���?<���� ?���l��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���??���� ?���t��� ��?� ?<���?� }/,?����?����?���?�??�?����?���?��?�������?���?y-� ?B�����?��������?�����������������$�??������?����������� ���?<���� ?���t��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���?<���� ?���\��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���?<���� ?������ ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���?<���� ?��� ��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���?<���� ?���(��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ��������������~��?��?��> ��~ ��?��?��>��~��?���?<���� ?���T��� ��c� ?$���????�???h�?��?��� ���?���? ���???�����???������??�33?�???�???��?f���?^����?���_�_�_�P�P�T�1�0���?>����?.���?]?0G?������������+��������D?����'? ������������������������k0�????����=? ���@B?��� ����D??�����'? ����������������������������������=? ���@B?��������A?��������������������?�%?,�����(?������ ������������j��J?�� �p�??��?��?��??��??��??��??��� ??��??��??��?f��r?��??��??��V?��:?��$?��P?��|?��??��?��?��??��??��??����?������s �����?�����"����M?�??T���?(���?��?��?��?���������������������� ?����� �������?j�����?��������?V���h�t�t�p�:�/�/�w�w�w�.�c�o�m�m�o�n�-�l�i�s�p�.�n�e�t�/�p�r�o�j�e�c�t�/�e�l�e�p�h�a�n�t��?\�����?��������?H���h�t�t�p�:�/�/�t�r�a�c�.�c�o�m�m�o�n�-�l�i�s�p�.�n�e�t�/�e�l�e�p�h�a�n�t��??��/�?������������������������ ��� ��� ��� ��� ���������������������??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????�� ��������������������????.??�+,??D���????.??�+,???��H��������?������?������?������?������?������?��� ���?��� ���?������?��� ���?������?������?������?��� ���?��� ���?�����??��������On-screen Show�����?,����t��������������������������������� � ������� ������� ������� ���������������Arial����?????? ???????????????� ���Wingdings����Times New Roman����Blank Presentation�X���Elephant: A Persistent Metaobject Protocol and Object Oriented Database for Common Lisp��?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?????���?????????���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?���?�����??�� ��������������������?????Oh??�+'??0���?7�� ������P������X������x������?��� ���?��� ���?��� ���?������?������?������'��������PowerPoint Presentation���� ���Ian Eslick����� ���Ian Eslick��������3���@���??xa ���@���???c??���&��G���?6��????PICT6?�����?�?�? �??���H���H�������?�?������� �����?�?�?���???�����?�?��������H���H���� ����� �)lpJ?�����?�?�����?�?��� ????????????� ????????????� ????????????� ????????????� ????????????�.?????????tl?fls????????????tl?fls?????????�6??????s??????????????s??????s??????????????s??????�*??????t????????t??????t????????t??????�*??????s????????s??????s????????s??????�_?????y??????y????N????????y????y??????y????N????????y?????y????N??????�X?????f??????????N??'????r????f??????????N??'????r??????????N??'???�??????f??????&???f!%???m.-???6D\??t)R??@g6??2?X????f????f??????&???f!%???m.-???6D\??t)R??@g6??2?X????f??????&???f!%???m.-???6D\??t)R??@g6??2?X???�??????s?????X'????T??,?y+??????*??I??1??�??ly?'???????s????s?????X'????T??,?y+??????*??I??1??�??ly?'???????s?????X'????T??,?y+??????*??I??1??�??ly?'??????�??????s??????$???[[??(?y{??L??B??d????v?????e?'????t????s??????$???[[??(?y{??L??B??d????v?????e?'????t??????$???[[??(?y{??L??B??d????v?????e?'???�???????????????G{???y???Q??N??g??{?L?????c?'??????????????????G{???y???Q??N??g??{?L?????c?'?????????????G{???y???Q??N??g??{?L?????c?'???�?????????????&???h??F?yB????N??g?{y??!?????c?-???????????????????&???h??F?yB????N??g?{y??!?????c?-??????????????&???h??F?yB????N??g?{y??!?????c?-??????�??????|?????'N???g:5??y3+!???N??g??/LEQ?????c?g?>????|????|?????'N???g:5??y3+!???N??g??/LEQ?????c?g?>????|?????'N???g:5??y3+!???N??g??/LEQ?????c?g?>???�Q????????????y?????�???�???????????????y?????�???�???????????y?????�???�???�'????????y???????????y???????y???�+?????p????????p????p????????p???????�??????????????????????????�??????????????????????????�?????????�???????�??????????�???�????????�???????????�???????�??????????�???�????????�???????�???????�??????????�???�????????�???�?????u????C??9kf|??????????�????S??=???????u???????�????UjW???�???�b???u??u????C??9kf|??????????�????S??=???????u???????�????UjW???�???�b???u???C??9kf|??????????�????S??=???????u???????�????UjW???�???�b??S????m???t????_V??P?pm??Z?WZ???WZ?V}?ol?l?J?/????L??D??|o??2?\V??bj?upP???|o??gj?/?????????E?in?/??k??gj??di?b???m??m???t????_V??P?pm??Z?WZ???WZ?V}?ol?l?J?/????L??D??|o??2?\V??bj?upP???|o??gj?/?????????E?in?/??k??gj??di?b???m??t????_V??P?pm??Z?WZ???WZ?V}?ol?l?J?/????L??D??|o??2?\V??bj?upP???|o??gj?/?????????E?in?/??k??gj??di?b??R????�f???^?^??5Zb????ht??S??w?S????r???M???c??!???????`?W?j???g???eu??xu?`?W????c???H\???u?i???c?p?v????t???b??�f??�f???^?^??5Zb????ht??S??w?S????r???M???c??!???????`?W?j???g???eu??xu?`?W????c???H\???u?i???c?p?v????t???b??�f???^?^??5Zb????ht??S??w?S????r???M???c??!???????`?W?j???g???eu??xu?`?W????c???H\???u?i???c?p?v????t???b??R????�f?????K???V??????t???z?w??z???f???b???c??!??fv???o???j?w?d???Vu???u?o??????c?????????c???c?q?{????h??sb??�f??�f?????K???V??????t???z?w??z???f???b???c??!??fv???o???j?w?d???Vu???u?o??????c?????????c???c?q?{????h??sb??�f?????K???V??????t???z?w??z???f???b???c??!??fv???o???j?w?d???Vu???u?o??????c?????????c???c?q?{????h??sb??R????�f??Q??????V??C?hy?t??]t?w?]t??o?rg?b???]?????u????j??B?ry_?hk?uuj?u?j??k?|???????????|W?]??{c?k??sZ?b??�f??�f??Q??????V??C?hy?t??]t?w?]t??o?rg?b???]?????u????j??B?ry_?hk?uuj?u?j??k?|???????????|W?]??{c?k??sZ?b??�f??Q??????V??C?hy?t??]t?w?]t??o?rg?b???]?????u????j??B?ry_?hk?uuj?u?j??k?|???????????|W?]??{c?k??sZ?b??�?????�f??�???�????????????????????????�?????b???????????????????????????�f??�f??�???�????????????????????????�?????b???????????????????????????�f??�???�????????????????????????�?????b???????????????????????????�#????�f??????�f??�f??????�f??????�????�f????�f??�f????�f????�????�f????�f??�f????�f????�????�f????�f??�f????�f????�?????�f??�???�???�???�???�???�???�???????????�???�???�???�f??�f??�???�???�???�???�???�???�???????????�???�???�???�f??�???�???�???�???�???�???�???????????�???�???�???????�f??�c?? ??~j??u??????�?????yf???????�???c???*jb???????????H????hyl???�V??�???�f??�f??�c?? ??~j??u??????�?????yf???????�???c???*jb???????????H????hyl???�V??�???�f??�c?? ??~j??u??????�?????yf???????�???c???*jb???????????H????hyl???�V??�????????�f?? ?bK?l?J?{a*??|}???V?upP?p?|o??gj?/???Q???q?P???ol???L?/??rr?{a*???e??I?ui\?2?_d~?WX??\V??h^??q????/?ql??P???X?????k??M[vy??_M?g??bj???L???V??p?WZ?gd]???�f??�f?? ?bK?l?J?{a*??|}???V?upP?p?|o??gj?/???Q???q?P???ol???L?/??rr?{a*???e??I?ui\?2?_d~?WX??\V??h^??q????/?ql??P???X?????k??M[vy??_M?g??bj???L???V??p?WZ?gd]???�f?? ?bK?l?J?{a*??|}???V?upP?p?|o??gj?/???Q???q?P???ol???L?/??rr?{a*???e??I?ui\?2?_d~?WX??\V??h^??q????/?ql??P???X?????k??M[vy??_M?g??bj???L???V??p?WZ?gd]????????�f?? ??w?M???d?I??^c???^?u??xb?`?W????c???R???V????r???v???c?V?r?d?I???e??[???5?j???I???R???gx???L?U???c???b?????V???p?v?t?K?c???|?o???ev?????V??b?S??O?????�f??�f?? ??w?M???d?I??^c???^?u??xb?`?W????c???R???V????r???v???c?V?r?d?I???e??[???5?j???I???R???gx???L?U???c???b?????V???p?v?t?K?c???|?o???ev?????V??b?S??O?????�f?? ??w?M???d?I??^c???^?u??xb?`?W????c???R???V????r???v???c?V?r?d?I???e??[???5?j???I???R???gx???L?U???c???b?????V???p?v?t?K?c???|?o???ev?????V??b?S??O??????????�f?? u???b???k?W??????9?u???b?o??????c??C?????????f???v???c?Q???k?W???e??H?L?L?j?[?R???i?w?d??D?P?????c???V????Q????q?{???b?c?????k???Vv?????V??b??z?X?????�f??�f?? u???b???k?W??????9?u???b?o??????c??C?????????f???v???c?Q???k?W???e??H?L?L?j?[?R???i?w?d??D?P?????c???V????Q????q?{???b?c?????k???Vv?????V??b??z?X?????�f?? [393 lines skipped] From ieslick at common-lisp.net Tue Apr 24 03:02:27 2007 From: ieslick at common-lisp.net (ieslick) Date: Mon, 23 Apr 2007 23:02:27 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070424030227.4D6714D04B@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv23327/doc Modified Files: IntroTalk1.ppt user-guide.texinfo Log Message: More documentation edits; performance and feature enhancements for map-index (from-end, collect); fix bug in slot initialization under from-oid Binary files /project/elephant/cvsroot/elephant/doc/IntroTalk1.ppt 2007/04/23 04:58:15 1.1 and /project/elephant/cvsroot/elephant/doc/IntroTalk1.ppt 2007/04/24 03:02:26 1.2 differ --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/22 03:35:09 1.13 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/24 03:02:26 1.14 @@ -340,53 +340,186 @@ (initialization of slots, slot-access, etc). All persistent classes create objects that inherit from the - at code{persistent} class. The persistent class has two slots that -contain a unique object identifier (oid) and a reference to the - at code{store-controller} specification they are associated with. + at code{persistent} class. The @code{persistent} class provides two +slots that contain a unique object identifier (oid) and a reference to +the @code{store-controller} specification they are associated with. Persistent slots do not take up any storage space in memory, instead -the slot access protocol is changed into calls to the configured data -store via the store controller. Typically, data stores then perform -reads and writes to disk. +the @code{persistent-metaclass} slot access protocol redirects slot +accesses into calls to the store controller. Typically, the +underlying data store will then perform the necessary serialization, +deserialization to read and write data to disk. When a reference to a @code{persistent} instance itself is written to the database, for example as a key or value in a @code{btree}, only the unique ID and class of the instance is stored. When read, a persistent object instance is re-created (see below). This means that serialization of persistent objects is exceedingly cheap compared to -standard objects, but slot access time can be much more expensive. +standard objects. The subsection on instance creation below will +discuss the lifecycle of a persistent object in more detail. @subsection Persistent Class Definition To create persistent classes, the user needs to specify the @code{persistent-metaclass} to the class initarg @code{:metaclass}. -The only differences between standard and persistent class definitions -is the specification of a slot storage policy and an index policy. + + at lisp +(defclass my-pclass () + ((slot1 :accessor slot1 :initarg :slot1 :initform 1)) + (:metaclass persistent-metaclass)) + at end lisp + +The only differences between the syntax of standard and persistent +class definitions is the ability to specify a slot storage policy and +an index policy. Slot value storage policies are specified by a +boolean argument to the slot initargs @code{:persistent}, + at code{:transient} and @code{:indexed}. Slots are @code{:persistent} +and not @code{:indexed} by default. + The @code{defpclass} macro is provided as a convenience to hide the @code{:metaclass} slot option. -Slot value storage policies are specified by a boolean argument to the -new slot initargs @code{:persistent}, @code{:transient} and - at code{indexed}. Slots are @code{:persistent} by default. - @lisp -(defclass my-pclass () +(defpclass my-pclass () ((pslot1 :accessor pslot1 :initarg :pslot1 :initform 'one) (pslot2 :accessor pslot2 :initarg :pslot2 :initform 'two :persistent t) - (tslot1 :accessor tslot1 :initarg :tslot1 :initform 'three :transient t)) - (:metaclass persistent-metaclass)) + (tslot1 :accessor tslot1 :initarg :tslot1 :initform 'three :transient t))) @end lisp -In the definition above, @code{pslot1} and @code{pslot2} are -persistent while @code{tslot1} is transient and stored in memory. +In the definition above the class @code{my-pclass} is an instance of +the metaclass @code{persistent-metaclass}. According to this +definition @code{pslot1} and @code{pslot2} are persistent while + at code{tslot1} is transient and stored in memory. Slot storage class implications are straightforward. Persistent slot -writes are durably stored to disk. Transient slots are initialized on -instance creation according to initforms or initargs. Transient slot -values are never stored to nor loaded from the database. +writes are durably stored to disk and reads are made from disk and can +be part of a ACID compliant transaction . Transient slots are +initialized on instance creation according to initforms or initargs. +Transient slot values are never stored to nor loaded from the database +and their accesses cannot be protected by transactions. (Ordinary +multi-process synchronization would be required instead). + +Persistent classes have their metaobject protocols modified through +specializations on @code{persistent-metaclass}. These specializations +create special slot metaobjects: @code{transient-slot-definition}, + at code{persistent-slot-definition} and direct and effective versions of +each. For the MOP aficionado the highlights of the new class +initialization protocols are as follows: + + at itemize + at item @code{shared-initialize :around} ensures that this class inherits from + @code{persistent-object} and @code{persistent} if it doesn't + already and that the class option @code{:index} results in class indexes + being indexed;. + at item @code{direct-slot-initialization-class} returns the appropriate slot + metaobject based on the values of the @code{:transient} and @code{:persistent} + slot definition keywords. It also does some simple error checking for invalid + combinations, for example, indexed transient slots. + at item @code{effective-slot-definition-class} performs the same role as the above for + effective slots. + at item @code{slot-definition-allocation} returns the @code{:database} allocation for + persistent slot definitions so the underlying lisp will not allocate instance or + class storage under some lisps. + at item @code{compute-effective-slot-definition-initargs} performs some error checking + to ensure a subclass does not try to make an inherited persistent slot transient. + at item @code{finalize-inheritance} called before the first instance is created in order + to finalize the list of persistent slots to account for any + forward referenced classes in the inheritence list. Similarly the + list of indexed slots is computed. This function is also called by the class indexing + code if any calls are made that depend on knowing which slots are indexed. + at end itemize + +Reinitialization is discussed in the section on class redefinition. + + at subsection Instance Creation + +To the user, persistent objects are created as usual with a call to + at code{make-instance}. Initforms and slot initargs behave as the user +expects. The call to @code{make-instance} of a persistent class will +fail unless there is a default @code{store-controller} instance in the +variable @code{*store-controller*} or the @code{:sc} keyword argument +is provided a valid store controller object. The store controller is +required to provide a unique object id, initialize the specification +pointer of the instance and to store the values of any initialized +slots. The initialization process is as follows: + + at itemize + at item @code{initialize-instance :before} is called to initialize the + @code{oid} slot and the data store specification slot @code{dbcn-spc-pst}. + The oid is set by the argument @code{:from-oid} or by calling the store + controller for a new oid. + at item @code{shared-initialize :around} is called to ensure that the underlying + lisp does not bypass the metaobject protocol during slot + initialization by manually initializing the persistent slots + and passing the transient slots to the underlying lisp. + Finally it adds the instance to the class index so that any inverted indicies + are updated appropriately. + at end itemize + +Persistent slots are initialized under the following conditions: + + at enumerate + at item An initarg is provided to @code{make-instance} + at item The database slot value is unbound, an initform exists and from-oid was not provided + at end enumerate + + at subsection Persistent Class Lifecycle + + + +The initialize instance protocol and persistent slot initialization +provides us some insight into how persistent class instances are +recreated in a fresh lisp image. To get a persistent instance, the +instance must be retrieved from the database via the root or a class +index. When a reference to an instance is deserialized, the deserializer +looks in a + +Because the oid is already known +when we deserialize the class reference from the database, the use the + at code{:from-oid} argument to indicate that this object already exists. +Because any initializable values are set + + + The two properties of + at code{persistent} can be specified explicitly during instance +creation: + + at lisp +(make-instance 'my-pclass :from-oid 100 :sc *store-controller*) + at end lisp + +These three elements, class, oid and store controller is all that is +needed to create a new instance + + +If you do make an instance with a specified OID which already exists +in the database, @code{initargs} to @code{make-instanc} take +precedence over values in the database, which take precedences over +any @code{initforms} defined in the class. + +Think of it this way. If you get a value from a persistent index and that value is a persistent object, then you want object identity, the ability to inspect it's type, slot access, etc. The only way to do that is to create something in memory which provides this. The only way to that is to make sure your lisp has had the chance to construct everything it needs via make-instance. + +The :from-oid argument to make-instance overrides some of the normal make-instance behavior by inhibiting all initform initialization as the object's slots are assumed to already be valid, so you have exactly the behavior you want. + +There are two caveats to this nice model: transient slots and make-instance methods. + +Transient Slots: they have the semantics of ordinary lisp slots - you cannot expect them to persist between images or after their placeholder has been garbage collected. If you drop all references to an object with transient slots, those slots are reclaimed by the GC. If you try to access them again later, they are reset to their default state (unbound, initforms or make-instance computations). + +Make-instance specializers: If you override the typical make-instance via :before, :after, :around or specialized on your class, you have to make sure that you predicate on :from-oid (this will be documented in the new manual) to avoid resetting persistent values. If :from-oid is given a value, then you can assume the object is being reconstituted and you don't need to do anything special for the persistent slots. + +However, if you want to use persistent slots to compute the value of some transient slots, then a make-instance specializer is a good place to do it. indexed-btree is an example of this, an in-memory hash is cached in the transient slot for reads and writes are mirrored to a serialized hash in a persistent slot. + + at subsection Using Persistent Slots + +* What happens to persistent objects when store-controller is closed? + + at subsection Using Transient Slots + +The best policy for initializing transient slots is an @code{:after} +method on @code{initialize-instance}. During a given lisp session transient values will be cached regardless of how often you retrieve an object from the store as long as there is -a reference to it stored in memory or tere are zero references but the +a reference to it in memory or there are zero references but the object has not yet been collected by the lisp garbage collector. After collection, if you retrieve an object from the store its transient slots will be reset to the slot initforms from the class @@ -394,6 +527,11 @@ or persistent slots during the initial call to @code{make-instance} or by manually creating the instance from an oid (see below). +A good policy is to initialize transient values using an @code{:after} +method on @code{initialize-instance} to initialize transient values +using system defaults or from persistent slot values. That way +whether the slots are valid when the object is pulled from + @lisp (setf pobj1 (make-instance 'my-pclass :pslot1 1 :tslot3 3)) => # @@ -431,48 +569,6 @@ behavior of indexed classes and class slots are discussed in depth in @ref{Class Indices}. - at subsection Instance Creation - -To the user, persistent objects are created normally with a call to - at code{make-instance} with the desired slot initargs as illustrated in -the last section. However, this call will fail unless there is a -default @code{store-controller} instance in the variable - at code{*store-controller*} or the @code{:sc} keyword argument is -provided with a valid store controller object. - -An instance of the class is created and any initforms and initarg -values are written to transient or persistent slots as one would -expect. - -* Default store controller & instance creation -* What happens to persistent objects when store-controller is closed? - -The best policy for initializing transient slots is an @code{:after} -method on @code{initialize-instance}. - -The two properties of @code{persistent} can be specified explicitly -during instance creation: - - at lisp -(make-instance 'my-pclass :from-oid 100 :sc *store-controller*) - at end lisp - -These three elements, class, oid and store controller is all that is -needed to create a new instance - -If you do make an instance with a specified OID which already exists -in the database, @code{initargs} to @code{make-instanc} take -precedence over values in the database, which take precedences over -any @code{initforms} defined in the class. - -A good policy is to initialize transient values using an @code{:after} -method on @code{initialize-instance} to initialize transient values -using system defaults or from persistent slot values. That way -whether the slots are valid when the object is pulled from - -Also currently there is a bug where @code{initforms} are always -evaluated, so beware. (What is the current model here?) - @subsection Class Redefinition This section discusses the appropriate model a user should employ in From ieslick at common-lisp.net Tue Apr 24 03:02:28 2007 From: ieslick at common-lisp.net (ieslick) Date: Mon, 23 Apr 2007 23:02:28 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070424030228.623084D04B@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv23327/src/elephant Modified Files: classes.lisp classindex.lisp collections.lisp Log Message: More documentation edits; performance and feature enhancements for map-index (from-end, collect); fix bug in slot initialization under from-oid --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/22 03:35:09 1.29 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/24 03:02:27 1.30 @@ -129,7 +129,7 @@ (unwind-protect (progn ;; initialize the persistent slots ourselves - (initialize-persistent-slots class instance persistent-slot-inits initargs) + (initialize-persistent-slots class instance persistent-slot-inits initargs from-oid) ;; let the implementation initialize the transient slots (apply #'call-next-method instance transient-slot-inits initargs)) (uninhibit-indexing oid)) @@ -144,7 +144,7 @@ (setf (get-value oid class-index) instance)))) )))) -(defun initialize-persistent-slots (class instance persistent-slot-inits initargs) +(defun initialize-persistent-slots (class instance persistent-slot-inits initargs object-exists) (flet ((initialize-from-initarg (slot-def) (loop for initarg in initargs with slot-initargs = (slot-definition-initargs slot-def) @@ -157,7 +157,7 @@ (loop for slot-def in (class-slots class) unless (initialize-from-initarg slot-def) when (member (slot-definition-name slot-def) persistent-slot-inits :test #'eq) - unless (slot-boundp-using-class class instance slot-def) + unless (or object-exists (slot-boundp-using-class class instance slot-def)) do (let ((initfun (slot-definition-initfunction slot-def))) (when initfun --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/23 02:26:53 1.37 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/24 03:02:27 1.38 @@ -394,8 +394,8 @@ (declare (dynamic-extent map-fn)) (map-btree #'map-fn class-idx)))) -(defun map-class-index (fn class index &rest args &key start end value from-end) - "map-class-index maps a function of two variables, taking key +(defun map-inverted-index (fn class index &rest args &key start end value from-end) + "map-inverted-index maps a function of two variables, taking key and instance, over a subset of class instances in the order defined by the index. Specify the class and index by quoted name. The index may be a slot index or a derived index. --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/23 02:41:11 1.24 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/24 03:02:27 1.25 @@ -22,6 +22,10 @@ (in-package "ELEPHANT") +#-elephant-without-optimize +(eval-when (:compile-toplevel) + (declaim (optimize (speed 3) (safety 1) (space 1)))) + ;;; collection types ;;; we're slot-less (defclass persistent-collection (persistent) () @@ -382,7 +386,7 @@ (funcall fn k v) (return nil))))))))) -(defgeneric map-index (fn index &rest args &key start end value from-end) +(defgeneric map-index (fn index &rest args &key start end value from-end collect) (:documentation "Map-index is like map-btree but for secondary indices, it takes a function of three arguments: key, value and primary key. As with map-btree the keyword arguments start and end @@ -393,6 +397,72 @@ use the value keyword which will override any values of start and end.")) +(defmethod map-index (fn (index btree-index) &rest args + &key start end (value nil value-set-p) from-end collect) + (declare (dynamic-extent args)) + (unless (or (null start) (null end) (lisp-compare<= start end)) + (error "map-index called with start = ~A and end = ~A. Start must be less than or equal to end according to elephant::lisp-compare<=." + start end)) + (let ((sc (get-con index)) + (end (or value end)) + (results nil)) + (flet ((collector (k v pk) + (push (funcall fn k v pk) results))) + (let ((fn (if collect #'collector fn))) + (declare (dynamic-extent (function collector))) + (ensure-transaction (:store-controller sc) + (with-btree-cursor (cur index) + (labels ((continue-p (key) + ;; Do we go to the next value? + (or (if from-end (null start) (null end)) + (if from-end + (or (not (lisp-compare<= key start)) + (lisp-compare-equal key start)) + (lisp-compare<= key end)))) + (value-increment () + ;; Step to the next key value + (if from-end + (pprev-hack cur) + (cursor-pnext-nodup cur))) + (next-value () + ;; Handle the next key value + (multiple-value-bind (exists? skey val pkey) + (value-increment) + (if (and exists? (continue-p skey)) + (progn + (funcall fn skey val pkey) + (map-duplicates skey)) + (return-from map-index + (nreverse results))))) + (map-duplicates (key) + ;; Map all duplicates for key value + (multiple-value-bind (exists? skey val pkey) + (cursor-pnext-dup cur) + (if exists? + (progn + (funcall fn skey val pkey) + (map-duplicates key)) + (progn + (cursor-pset-range cur key) + (next-value)))))) + (declare (dynamic-extent (function next-value) (function next-value-increment) + (function continue-p) (function map-duplicates))) + (multiple-value-bind (exists? skey val pkey) + (cond (value-set-p + (cursor-pset cur value)) + ((and (not from-end) (null start)) + (cursor-pfirst cur)) + ((and from-end (null end)) + (cursor-last-range-hack cur)) + (t (if from-end + (cursor-pset-range cur end) + (cursor-pset-range cur start)))) + (if (and exists? (continue-p skey)) + (progn + (funcall fn skey val pkey) + (map-duplicates skey)) + nil))))))))) + (defun pprev-hack (cur) "Get the first duplicate instance of the prior value off the current cursor" (let ((e? (cursor-pprev-nodup cur))) @@ -411,57 +481,6 @@ (cursor-pnext cur) (cursor-pfirst cur)))))) - -(defmethod map-index (fn (index btree-index) &rest args &key start end (value nil value-set-p) from-end) - (declare (dynamic-extent args) - (ignorable args)) - (unless (or (null start) (null end) (lisp-compare<= start end)) - (error "map-index called with start = ~A and end = ~A. Start must be less than or equal to end according to elephant::lisp-compare<=." - start end)) - (let ((sc (get-con index)) - (end (or value end))) - (ensure-transaction (:store-controller sc) - (with-btree-cursor (cur index) - (labels ((continue-p (key) ;; Do we got to the next value? - (or (if from-end (null start) (null end)) - (if from-end - (or (not (lisp-compare<= key start)) - (lisp-compare-equal key start)) - (lisp-compare<= key end)))) - (value-increment () ;; Step to the next key value - (if from-end - (pprev-hack cur) - (cursor-pnext-nodup cur))) - (next-value () ;; Handle the next key value - (multiple-value-bind (exists? skey val pkey) - (value-increment) - (if (and exists? (continue-p skey)) - (progn - (funcall fn skey val pkey) - (map-duplicates skey)) - (return-from map-index nil)))) - (map-duplicates (key) ;; Map all duplicates for key value - (loop as (exists? skey val pkey) = (multiple-value-list (cursor-pnext-dup cur)) - while exists? do (funcall fn skey val pkey)) - (cursor-pset-range cur key) - (next-value))) - (declare (dynamic-extent next-value next-value-increment continue-p map-duplicates)) - (multiple-value-bind (exists? skey val pkey) - (cond (value-set-p - (cursor-pset cur value)) - ((and (not from-end) (null start)) - (cursor-pfirst cur)) - ((and from-end (null end)) - (cursor-last-range-hack cur)) - (t (if from-end - (cursor-pset-range cur end) - (cursor-pset-range cur start)))) - (if (and exists? (continue-p skey)) - (progn - (funcall fn skey val pkey) - (map-duplicates skey)) - nil))))))) - ;; =============================== ;; Some generic utility functions ;; =============================== From ieslick at common-lisp.net Tue Apr 24 12:58:10 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 08:58:10 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070424125810.C0C684051@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv14127/doc Modified Files: make-ref.lisp user-guide.texinfo Log Message: Default inihibit annoying warnings --- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/21 17:22:35 1.9 +++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/04/24 12:58:10 1.10 @@ -34,7 +34,7 @@ ;; (sb-texinfo:generate-includes #p"/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/" (sb-texinfo:generate-includes include-dir-path (find-package :elephant) - (find-package :elephant-backend) + (find-package :elephant-data-store) (find-package :elephant-memutil) (find-package :elephant-system)))) --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/24 03:02:26 1.14 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/24 12:58:10 1.15 @@ -398,12 +398,17 @@ and their accesses cannot be protected by transactions. (Ordinary multi-process synchronization would be required instead). +The @code{:index} option tells Elephant whether to maintain an +inverted index that maps slot values to their parent objects. The +behavior of indexed classes and class slots are discussed in depth in + at ref{Class Indices}. + Persistent classes have their metaobject protocols modified through specializations on @code{persistent-metaclass}. These specializations -create special slot metaobjects: @code{transient-slot-definition}, - at code{persistent-slot-definition} and direct and effective versions of -each. For the MOP aficionado the highlights of the new class -initialization protocols are as follows: +include the creation of special slot metaobjects: + at code{transient-slot-definition}, @code{persistent-slot-definition} +and direct and effective versions of each. For the MOP aficionado the +highlights of the new class initialization protocols are as follows: @itemize @item @code{shared-initialize :around} ensures that this class inherits from @@ -432,15 +437,15 @@ @subsection Instance Creation -To the user, persistent objects are created as usual with a call to - at code{make-instance}. Initforms and slot initargs behave as the user -expects. The call to @code{make-instance} of a persistent class will -fail unless there is a default @code{store-controller} instance in the -variable @code{*store-controller*} or the @code{:sc} keyword argument -is provided a valid store controller object. The store controller is -required to provide a unique object id, initialize the specification -pointer of the instance and to store the values of any initialized -slots. The initialization process is as follows: +Persistent objects are created just like standard objects, with a call +to @code{make-instance}. Initforms and slot initargs behave as the +user expects. The call to @code{make-instance} of a persistent class +will fail unless there is a default @code{store-controller} instance +in the variable @code{*store-controller*} or the @code{:sc} keyword +argument is provided a valid store controller object. The store +controller is required to provide a unique object id, initialize the +specification pointer of the instance and to store the values of any +initialized slots. The initialization process is as follows: @itemize @item @code{initialize-instance :before} is called to initialize the @@ -455,82 +460,107 @@ are updated appropriately. @end itemize -Persistent slots are initialized under the following conditions: +Persistent slots are initialized only under the following conditions: - at enumerate + at itemize @item An initarg is provided to @code{make-instance} - at item The database slot value is unbound, an initform exists and from-oid was not provided - at end enumerate - - at subsection Persistent Class Lifecycle - + at item The database slot value is unbound, an initform exists and from-oid was not specified + at end itemize +After initialization the persistent instance is added to its host +store controller's object cache. This cache is a weak hash table that +maps oids to object instances. So after initialization the following +state has been created: -The initialize instance protocol and persistent slot initialization -provides us some insight into how persistent class instances are -recreated in a fresh lisp image. To get a persistent instance, the -instance must be retrieved from the database via the root or a class -index. When a reference to an instance is deserialized, the deserializer -looks in a + at itemize + at item @strong{Placeholder Instance:} An instance of the class is in memory, containing storage for + the oid, the specification reference, lisp instance data and any + transient slot values. We call this the placeholder instance which + mediates access to persistent values, but does not itself persist. + at item @strong{Cached Reference:} A weak reference to the instance is in the store controller object cache + at item @strong{Memory References:} A normal reference to the instance is (maybe) retained by the caller of + @code{make-instance}. + at item @strong{Database Slot Values:} The data store contains the persistent slot values that were initialized, + indexed by the object id and slot name. + at item @strong{Database References:} If the resulting placeholder instance was written to a persistent slot, + added to a btree or the class is indexed, a @strong{reference} +to the instance was written into the data store. Today this reference +consists of an oid and a class name. If this reference is reachable, +then the persistent object can be reconstructed using the + at code{:from-oid} argument. + at end itemize -Because the oid is already known -when we deserialize the class reference from the database, the use the - at code{:from-oid} argument to indicate that this object already exists. -Because any initializable values are set +If you mnanually create an object using an OID which already exists in +the database, @code{initargs} to @code{make-instance} take precedence +over existing values in the database, which in turn take precedence +over any @code{initforms} defined in the class. + + at subsection Persistent Instance Lifecycle + +The distributed nature of persistent instance storage results in some +interesting behaviors, especially with respect to transient slots. +The prior section detailed the state of the system after the original +initialization of an object. The object can then be in a number of +different states: + at itemize + at item @strong{Resident:} The canonical state of an in-use persistent + object as described in the initialization section above. + at item @strong{Unreferenced, Unreclaimed:} All memory references to the + object have been dropped but the placeholder instance has not yet + been garbage collected. The weak pointer still exists in the cache. + If a database reference is fetched from the data store, the cached + value will be used. + at item @strong{Non-resident:} The object only exists as reachable database + references and slot values. This is the state after garbage collection + of the placeholder instance. + at item @strong{Recreated:} An intermediary state where a non resident object + is fetched from the data store and its placeholder object must be + recreated prior to the object enter the resident state. + at end itemize - The two properties of - at code{persistent} can be specified explicitly during instance -creation: +The garbage collection of the placeholder instance is an important +feature. This means that we can have more objects in our system than +are currently resident in memory. If this were not the case, what +would be the point of an object database? + +The recreated state deserves to be discussed in more detail. We +learned earlier that the database reference contains the oid and class +of the object, and of course we know the store-controller the +reference is stored into at footnote{If you attempt to store an object +from one store into another, the system will issue an error condition +called @code{cross-reference-error}}, so this information is +sufficient to reconstruct the placeholder instance. + +When the reference is deserialized, its oid is used to look up the +object in the store controller's object cache. If this fails, then +the instance is created with a call much like this: @lisp -(make-instance 'my-pclass :from-oid 100 :sc *store-controller*) +(make-instance 'pclass :from-oid 2000 :sc *store-controller*) @end lisp -These three elements, class, oid and store controller is all that is -needed to create a new instance - - -If you do make an instance with a specified OID which already exists -in the database, @code{initargs} to @code{make-instanc} take -precedence over values in the database, which take precedences over -any @code{initforms} defined in the class. - -Think of it this way. If you get a value from a persistent index and that value is a persistent object, then you want object identity, the ability to inspect it's type, slot access, etc. The only way to do that is to create something in memory which provides this. The only way to that is to make sure your lisp has had the chance to construct everything it needs via make-instance. - -The :from-oid argument to make-instance overrides some of the normal make-instance behavior by inhibiting all initform initialization as the object's slots are assumed to already be valid, so you have exactly the behavior you want. - -There are two caveats to this nice model: transient slots and make-instance methods. - -Transient Slots: they have the semantics of ordinary lisp slots - you cannot expect them to persist between images or after their placeholder has been garbage collected. If you drop all references to an object with transient slots, those slots are reclaimed by the GC. If you try to access them again later, they are reset to their default state (unbound, initforms or make-instance computations). - -Make-instance specializers: If you override the typical make-instance via :before, :after, :around or specialized on your class, you have to make sure that you predicate on :from-oid (this will be documented in the new manual) to avoid resetting persistent values. If :from-oid is given a value, then you can assume the object is being reconstituted and you don't need to do anything special for the persistent slots. - -However, if you want to use persistent slots to compute the value of some transient slots, then a make-instance specializer is a good place to do it. indexed-btree is an example of this, an in-memory hash is cached in the transient slot for reads and writes are mirrored to a serialized hash in a persistent slot. - - at subsection Using Persistent Slots - -* What happens to persistent objects when store-controller is closed? +The @code{:from-oid} argument to @code{make-instance} overrides some +of the normal make-instance behavior by inhibiting all initform +initialization as the object's slots are assumed to be properly +initialized from the original call to @code{make-instance}. @subsection Using Transient Slots -The best policy for initializing transient slots is an @code{:after} -method on @code{initialize-instance}. +What about transient slots? Transients slots are tied to the +placeholder object where their storage is allocated. While the +persistent slots are permanently stored in the data store, transient +slots can be garbage collected when all memory references have been +dropped, even if database references exist. -During a given lisp session transient values will be cached regardless -of how often you retrieve an object from the store as long as there is -a reference to it in memory or there are zero references but the -object has not yet been collected by the lisp garbage collector. -After collection, if you retrieve an object from the store its +After collection, if you retrieve an object from the store, its transient slots will be reset to the slot initforms from the class -definition. You can only use @code{:initargs} to initialize transient -or persistent slots during the initial call to @code{make-instance} or -by manually creating the instance from an oid (see below). +definition. You can only reliably use @code{:initargs} to initialize +transient or persistent slots during the initial call to + at code{make-instance} or when manually creating the instance from an oid. -A good policy is to initialize transient values using an @code{:after} -method on @code{initialize-instance} to initialize transient values -using system defaults or from persistent slot values. That way -whether the slots are valid when the object is pulled from +Here is an example illustrating the ephemeral nature of transient +slots: @lisp (setf pobj1 (make-instance 'my-pclass :pslot1 1 :tslot3 3)) @@ -560,14 +590,56 @@ @end lisp The implications of this behavior is that you need to think carefully -about how to use transient values. Essentially you cannot make +about how to use employ transient values. Essentially you cannot make assumptions about the state of transient values in objects loaded from -the store unless you know that they were loaded and cannot be GC'ed. +the store unless you know that they were loaded at some point in time +and cannot be GC'ed (i.e. they are stored in a list or hash table). -The @code{:index} option tells Elephant whether to maintain an -inverted index that maps slot values to their parent objects. The -behavior of indexed classes and class slots are discussed in depth in - at ref{Class Indices}. +A good policy is to initialize transient values using an @code{:after} +method on @code{initialize-instance}. This allows you to initialize +transient values using either system defaults or persistent slot +values. That way you can ensure that the transient slots are always +in a consistent state when accessed by the application, regardless of +when the placeholder object was recreated. + +In general, transient slots are a good place for intermediate values +in a computation or to cache frequently read items to avoid +deserialization overhead. @code{indexed-btree} is an example of this +approach, an in-memory hash is cached in the transient slot for reads +and writes are mirrored to a serialized hash in a persistent slot. +The @code{:after} method just copies the persistent hash value to the +transient slot. + + at subsection Using Persistent Slots + +Persistent slot use is straightforward. You can read from them, write +to them or make them unbound. Remember that every access goes to the +data store. This makes reads relatively expensive as they may result +in a disk seek. Writes can be doubly expensive, especially outside a +transaction, as the write will result in a synchronous disk synch +operation. + +Reads and writes require the home store controller to be valid and +open. The placeholder object's specification pointer is used to +retrieve the @code{store-controller} object. If this object is closed +or mising, the system will give you a restart option to reopen the +controller and continue. + +Persistent slot behavior is implemented by overloading the relevant +MOP functions controlling slot access: + + at itemize + at item @code{slot-value-using-class} + at item @code{(setf slot-value-using-class)} + at item @code{slot-boundp-using-class} + at item @code{slot-makunbound-using-class} + at end itemize + +Each of these functions retrieves the home store-controller for the +instance and then calls a method specialized on the class of that +store controller. This method is responsible for mapping the oid and +slotname of the slot access to the appropriate value in the data +store. @subsection Class Redefinition @@ -580,52 +652,12 @@ - What if you connect to an old database with a new class specification? (ref to class indicies behavior) - at subsection Synchronizing Code and Database - - - at subsection Persistent objects and the MOP - -This section provides increased detail on the use of the MOP to create -and support persistent objects. - -new slot types -- transient-slot -- persistent-slot - -slot instantiation protocols -- effective-slot-definition -- direct-slot-definition-class -- compute-effective-slot-definition -- compute-effective-slot-definition-initargs - -class initialization protocols -- shared-initialize :around (class) -- finalize-inheritance :around (class) -- reinitialize-instance :around (class) - -instance initialization protocols -- initialize-instance :before (instance) -- shared-initialize :around +MOP: - update-instance-for-redefinied-class - update-instance-for-different-class -slot access protocol -- slot-value-using-class (setf) -- slot-boundp-using-class -- slot-makunbound-using-clas - - at subsection Overriding methods on persistent objects - -* slot types -* caching -* slot access protocol - -Readers, writers, accessors, and @code{slot-value-using-class} are -employed in redirecting slot accesses to the database, so override -these with care. Because @code{slot-value, slot-boundp, -slot-makunbound} are not generic functions, they are not guaranteed by -the specification to work properly with persistent slots. However the -proper behavior has been verified on SBCL, Allegro and Lispworks. + at subsection Synchronizing Code and Database + @node Class Indices @comment node-name, next, previous, up From ieslick at common-lisp.net Tue Apr 24 12:58:11 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 08:58:11 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070424125811.3A54A5062@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv14127/src/elephant Modified Files: classindex.lisp variables.lisp Log Message: Default inihibit annoying warnings --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/24 03:02:27 1.38 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/24 12:58:10 1.39 @@ -99,7 +99,8 @@ (defun ensure-finalized (class) (when (not (class-finalized-p class)) - (warn "Manually finalizing class ~A" (class-name class)) + (when *warn-on-manual-class-finalization* + (warn "Manually finalizing class ~A" (class-name class))) (finalize-inheritance class))) (defun cache-existing-class-index (class btree sc) --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/12 02:47:33 1.15 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/24 12:58:10 1.16 @@ -60,6 +60,13 @@ (defvar *current-transaction* nil "The transaction which is currently in effect.") +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;;; Enables warnings of various kinds + +(defvar *warn-on-manual-class-finalization* nil + "Issue a printed warnings when the class mechanism has + to finalize a class to access indexing information") + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; Forward references ;; From ieslick at common-lisp.net Tue Apr 24 16:39:30 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 12:39:30 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070424163930.6AB3637018@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv14475/doc Modified Files: user-guide.texinfo Log Message: Handle error conditions in change-class protocol; more docs --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/24 12:58:10 1.15 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/24 16:39:30 1.16 @@ -10,10 +10,9 @@ * Serialization details:: The devil hides in the details. * Persistent Classes and Objects:: All the dirt on persistent objects. * Class Indices:: In-depth discussion about indexing persistent indices. - at c * Querying persistent instances:: Retrieving instances of classes. * Using BTrees:: Using the native btree. * Using Cursors:: Low-level access to BTrees. -* BTree Indices:: Alternative ways to index collections. +* The BTree index:: Alternative ways to reference objects in btrees * Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. * Multi-threaded Applications:: What considerations are required for safe multi-threading @@ -643,21 +642,70 @@ @subsection Class Redefinition -This section discusses the appropriate model a user should employ in -thinking about the behavior of persistent object when their clases are -redefined. - -- What happens when you redefine a class online? -- Drop & add slots? Change slot status? -- What if you connect to an old database with a new class specification? - (ref to class indicies behavior) - -MOP: -- update-instance-for-redefinied-class -- update-instance-for-different-class +Class redefinition is problematic in the current (0.9) version of +Elephant. The usual CLOS mechanisms are properly implemented, but +updating instances will only work for those instances that are in +memory at the time. Instances that are non-resident will not be +updated. This is usually not as big a problem as it seems, because +the slot values are stored independently. An outline of the update +procedure follows: + +The function @code{update-instance-for-redefined-class} is called by +CLOS whenever @code{defclass} is re-evaluated and results in a change +in the list of slots. - at subsection Synchronizing Code and Database +For transient slots the behavior is the same as it is in CLOS for +all in-memory slots. + at itemize + at item Added slots: are added to the object and their initforms + called just as if they were created without initargs + at item Discarded slots: are dropped and their values lost + at end itemize + +Persistent slots have a slightly different behavior, as only resident +(those with valid placeholder objects) objects are updated. + + at itemize + at item Added slots (resident): are added to the object and the initforms + are called only on in-memory objects, as in an empty call to + @code{make-instance} + at item Added slots (non-resident): the added slots will have unbound values + at item Discarded slots (resident): slots are dropped from the class and become + inaccessible, but their values are not deleted from the database. This + is a precautionary measure as losing persistent data because of an + accidental re-evaluation while editing a defclass could be painful. If + you add the slot back, the original value will be accessible regardless of + the initform. + at item Discarded slots (non-resident): This has the same behavior as resident objects, + as no side effects are made on the objects or their slots + at end itemize + +There are additional considerations for matching class indexing +options in the class object to the actual indices in the database. +The following section will discuss synchronizing these if they +diverge. + + at emph{(Note: release 0.9.1 should fix this by providing an oid->class map that allows +the system to cheaply iterate over all objects and update them appropriately. This +hasn't been done yet due to performance implications. See Trac system for the appropriate +tickets)} + + at subsection Support for @code{change-class} + +Elephant also supports the @code{change-class} by overloading + at code{update-instance-for-different-class}. The handling of slots in +this case is identical to the class redefinition above. Persistent +and transient slot values are retained if their name matches a +slotname in the new class and initforms are called on newly added +slots. Valid initargs for any slot will override this default behavior +and set the slot value to the initarg value. + +Because the instance is guaranteed to be resident, the operation has none of the +resident/non-resident conflicts above. + +Changing a persistent instance to a non-persistent class is not +allowed and will result in an error. @node Class Indices @comment node-name, next, previous, up @@ -706,111 +754,43 @@ somewhat user customizable; documentation for this exists in the source file referenced above. - at c @node Querying persistent instances - at c @comment node-name, next, previous, up - at c @section Querying persistent instances - at c - at c A SQL select-like interface is in the works, but for now queries are - at c limited to manual mapping over class instances or doing small queries - at c with @code{get-instances-*} functions. One advantage of this is that - at c it is easy to estimate the performance costs of your queries and to - at c choose standard and derived indices that give you the ordering and - at c performance you want. - at c - at c There is, however, a quick and dirty query API example that is not - at c officially supported in the release but is intended to invite comment. - at c This is an example of a full query system that would automatically - at c perform joins, use the appropriate indices and perhaps even adaptively - at c suggest or add indices to facilitate better performance on common - at c queries. - at c - at c There are two functions @ref{Function elephant:get-query-instances} - at c and @ref{Function elephant:map-class-query} which accept a set of - at c constraints instead of the familiar value or range arguments. - at c - at c We'll use the classes @code{person} and @code{department} to - at c illustrate how to perform queries over a set of objects that may be - at c constrainted by their relationships to other objects. - at c - at c @lisp - at c (defpclass person () - at c ((name :initarg :name :index t) - at c (salary :initarg :salary :index t) - at c (department :initarg :dept))) - at c - at c (defmethod print-object ((p person) stream) - at c (format stream "#" (slot-value p 'name))) - at c - at c (defun print-name (inst) - at c (format t "Name: ~A~%" (slot-value inst 'name))) - at c - at c (defpclass department () - at c ((name :initarg :name) - at c (manager :initarg :manager))) - at c - at c (defmethod print-object ((d department) stream) - at c (format stream "#" - at c (slot-value d 'name) - at c (when (slot-boundp d 'manager) - at c (slot-value (slot-value d 'manager) 'name)))) - at c @end lisp - at c - at c Here we have a simple employee database with managers (also of type - at c person) and departments. This simple system will provide fodder for - at c some reasonably complex constraints. Let's create a few departments. - at c - at c @lisp - at c (setf marketing (make-instance 'department :name "Marketing")) - at c (setf engineering (make-instance 'department :name "Engineering")) - at c (setf sales (make-instance 'department :name "Sales")) - at c @end lisp - at c - at c And manager @code{people} for the departments. - at c - at c @lisp - at c (make-instance 'person :name "George" :salary 140000 :department marketing) - at c (setf (slot-value marketing 'manager) *) - at c - at c (make-instance 'person :name "Sally" :salary 140000 :department engineering) - at c (setf (slot-value engineering 'manager) *) - at c - at c (make-instance 'person :name "Freddy" :salary 180000 :department sales) - at c (setf (slot-value sales 'manager) *) - at c @end lisp - at c - at c And of course we need some folks to manage - at c - at c @lisp - at c (defparameter *names* - at c '("Jacob" "Emily" "Michael" "Joshua" "Andrew" "Olivia" "Hannah" "Christopher")) - at c - at c (defun random-element (list) - at c "Choose a random element from the list and return it" - at c (nth (random (length list)) list)) - at c - at c (with-transaction () - at c (loop for i from 0 upto 40 do - at c (make-instance 'person - at c :name (format nil "~A~A" (random-elephant *names*) i) - at c :salary (floor (+ (* (random 1000) 100) 30000)) - at c :department (case (random 3) - at c (0 marketing) - at c (1 engineering) - at c (2 sales))))) - at c @end lisp - at c - at c - at c Now we can look at a few queries. - at c - at c @lisp - at c (defun get-managers () - at c (get-query-instances `((person - at c - at c For those familiar with SQL, if an instance of @code{person} has a - at c pointer to an instance of @code{department} then that relation can be - at c used to perform a join. Of course joins in the object world won't - at c return a table, instead they will return conjunctions of objects that - at c satisfy a mutual set of constraints. + at subsection Synchronizing Classes and Data Stores + +Sometimes you may change a defclass form and then connect to a +database with instances that do not match the current defclass +definition. Because of the defclass behavior above, there is no need +to detect this case as the behavior will be as if all instances were +non-resident at redefinition time. However, this is an issue for +indexed classes as the cost of indexing is high. There is a +synchronization policy which updates either the class or the online +class indexing mechanism at the time you try to perform an index +operation (i.e. when @code{find-class-index} is called). + +A policy is selected by setting the value of + at code{*default-indexed-class-synch-policy*} with the appropriate +policy: + + at itemize + at item :class - The class is the master, and indices are deleted for any slots + that are no longer indexed + at item :db - The database is the master and the class indexing annotations are + updated so that the slots that satisfy @code{class-indexedp-by-name} + are isomorphic to the existing indices in the db. + at item :union - This does what you would expect, updates the class to match any + existing indices and creates new indices. + at end itemize + +Derived slots can be problematic as they may depend on slot values +that no longer exist in the changed defclass. This will result in an +error, so for now you will have to manage any mismatches such as this +yourself. + + at emph{Note: release 0.9.1 should fix both mismatches and performance issues related +to derived indices by allowing the user to provide hints as to which slot values the +index depends. This will allow the system to only update when the appropriate slots +change and to delete or inhibit derived indicies when slots are deleted. We will also +improve error handling for this case, so you can delete the derived index and continue +performing the write to a persistent object that flagged the error.} @node Using BTrees @comment node-name, next, previous, up @@ -829,15 +809,17 @@ blocks of data is relatively inexpensive after a seek and comparisons on objects that are stored in memory is cheap. + + @node Using Cursors @comment node-name, next, previous, up @section Using Cursors -Empty. +- initialized, not-initialized - at node BTree Indicies + at node The BTree Index @comment node-name, next, previous, up - at section BTree Indicies + at section The BTree Index Empty. @@ -1103,10 +1085,123 @@ @section CL-SQL Data Store + @node Postmodern Data Store @comment node-name, next, previous, up @section Postmodern Data Store +The postmodern data store is not yet integrated. It should be documented +for the forthcoming release 0.9.1 or 0.9.2. + @node Native Lisp Data Store @comment node-name, next, previous, up @section Native Lisp Data Store + +The native lisp data store is vaporware at this time. + + at c @node Querying persistent instances + at c @comment node-name, next, previous, up + at c @section Querying persistent instances + at c + at c A SQL select-like interface is in the works, but for now queries are + at c limited to manual mapping over class instances or doing small queries + at c with @code{get-instances-*} functions. One advantage of this is that + at c it is easy to estimate the performance costs of your queries and to + at c choose standard and derived indices that give you the ordering and + at c performance you want. + at c + at c There is, however, a quick and dirty query API example that is not + at c officially supported in the release but is intended to invite comment. + at c This is an example of a full query system that would automatically + at c perform joins, use the appropriate indices and perhaps even adaptively + at c suggest or add indices to facilitate better performance on common + at c queries. + at c + at c There are two functions @ref{Function elephant:get-query-instances} + at c and @ref{Function elephant:map-class-query} which accept a set of + at c constraints instead of the familiar value or range arguments. + at c + at c We'll use the classes @code{person} and @code{department} to + at c illustrate how to perform queries over a set of objects that may be + at c constrainted by their relationships to other objects. + at c + at c @lisp + at c (defpclass person () + at c ((name :initarg :name :index t) + at c (salary :initarg :salary :index t) + at c (department :initarg :dept))) + at c + at c (defmethod print-object ((p person) stream) + at c (format stream "#" (slot-value p 'name))) + at c + at c (defun print-name (inst) + at c (format t "Name: ~A~%" (slot-value inst 'name))) + at c + at c (defpclass department () + at c ((name :initarg :name) + at c (manager :initarg :manager))) + at c + at c (defmethod print-object ((d department) stream) + at c (format stream "#" + at c (slot-value d 'name) + at c (when (slot-boundp d 'manager) + at c (slot-value (slot-value d 'manager) 'name)))) + at c @end lisp + at c + at c Here we have a simple employee database with managers (also of type + at c person) and departments. This simple system will provide fodder for + at c some reasonably complex constraints. Let's create a few departments. + at c + at c @lisp + at c (setf marketing (make-instance 'department :name "Marketing")) + at c (setf engineering (make-instance 'department :name "Engineering")) + at c (setf sales (make-instance 'department :name "Sales")) + at c @end lisp + at c + at c And manager @code{people} for the departments. + at c + at c @lisp + at c (make-instance 'person :name "George" :salary 140000 :department marketing) + at c (setf (slot-value marketing 'manager) *) + at c + at c (make-instance 'person :name "Sally" :salary 140000 :department engineering) + at c (setf (slot-value engineering 'manager) *) + at c + at c (make-instance 'person :name "Freddy" :salary 180000 :department sales) + at c (setf (slot-value sales 'manager) *) + at c @end lisp + at c + at c And of course we need some folks to manage + at c + at c @lisp + at c (defparameter *names* + at c '("Jacob" "Emily" "Michael" "Joshua" "Andrew" "Olivia" "Hannah" "Christopher")) + at c + at c (defun random-element (list) + at c "Choose a random element from the list and return it" + at c (nth (random (length list)) list)) + at c + at c (with-transaction () + at c (loop for i from 0 upto 40 do + at c (make-instance 'person + at c :name (format nil "~A~A" (random-elephant *names*) i) + at c :salary (floor (+ (* (random 1000) 100) 30000)) + at c :department (case (random 3) + at c (0 marketing) + at c (1 engineering) + at c (2 sales))))) + at c @end lisp + at c + at c + at c Now we can look at a few queries. + at c + at c @lisp + at c (defun get-managers () + at c (get-query-instances `((person + at c + at c For those familiar with SQL, if an instance of @code{person} has a + at c pointer to an instance of @code{department} then that relation can be + at c used to perform a join. Of course joins in the object world won't + at c return a table, instead they will return conjunctions of objects that + at c satisfy a mutual set of constraints. + From ieslick at common-lisp.net Tue Apr 24 16:39:31 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 12:39:31 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070424163931.25E8E3A01C@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv14475/src/elephant Modified Files: classes.lisp Log Message: Handle error conditions in change-class protocol; more docs --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/24 03:02:27 1.30 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/24 16:39:30 1.31 @@ -177,13 +177,20 @@ (old-persistent-slots class)))) ;; Update new persistent slots, the others we get for free (same oid!) ;; Isn't this done by the default call-next-method? - (apply #'shared-initialize instance new-persistent-slots initargs)) + (apply #'shared-initialize instance new-persistent-slots initargs) + ) )) ;; ;; CLASS CHANGE PROTOCOL ;; +(defmethod change-class ((inst persistent) (class t) &rest rest) + (error "Changing a persistent instance's class to a non-persistent class is not currently allowed")) + +(defmethod change-class ((inst standard-object) (class persistent-metaclass) &rest rest) + (error "Changing a standard instance to a persistent instance is not supported")) + (defmethod update-instance-for-different-class :around ((previous persistent) (current persistent) &rest initargs &key) (let* ((old-class (class-of previous)) (new-class (class-of current)) From ieslick at common-lisp.net Wed Apr 25 02:27:56 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 22:27:56 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070425022756.DBF2A1E007@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv4997 Modified Files: TODO config.sexp Log Message: Export bdb performance tweaks; lots more documentation; new ops for libberkeley-db --- /project/elephant/cvsroot/elephant/TODO 2007/04/19 22:25:50 1.80 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/25 02:27:56 1.81 @@ -28,7 +28,6 @@ - More notes about transaction performance - Serious discussion of threading implications - Add notes about with/ensure-transaction usage (abort & commit behavior on exit) - - Add notes about optimize-storage - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) --- /project/elephant/cvsroot/elephant/config.sexp 2007/04/19 05:51:33 1.10 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/04/25 02:27:56 1.11 @@ -4,6 +4,7 @@ (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") (:pthread-lib . nil) + (:berkeley-db-cachesize . 20971520) (:clsql-lib . nil) (:compiler . :gcc)) @@ -46,4 +47,21 @@ ;; :compiler options are ;; :gcc (default: for unix platforms with /usr/bin/gcc) ;; :cygwin (for windows platforms with cygwin/gcc) -;; :msvc (unsupported) \ No newline at end of file +;; :msvc (unsupported) +;; +;; Additional supported parameters include: +;; +;; +;; :berkeley-db-cachesize - an integer indicating the number of bytes +;; for the page cache, default 10MB which is +;; about enough storage for 10k-30k indexed +;; persistent objects +;; +;; :map-using-degree2 - Boolean parameter that indicates whether map +;; operations lock down the btree for the entire +;; transaction or whether they allow other +;; transactions to add/delete/modify values +;; before the map operation is completed. The +;; map operation remains stable and any writes +;; are kept transactional, see user manual as +;; well as berkeley DB docs for more details From ieslick at common-lisp.net Wed Apr 25 02:27:57 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 22:27:57 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070425022757.C4D671E017@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv4997/doc Modified Files: reference.texinfo tutorial.texinfo user-guide.texinfo Log Message: Export bdb performance tweaks; lots more documentation; new ops for libberkeley-db --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/21 17:22:35 1.11 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/25 02:27:56 1.12 @@ -16,6 +16,7 @@ * Cursors:: Traversing BTrees. * Transactions:: Transactions. * Migration and Upgrading:: Migration and upgrading. +* Miscellaneous:: Other functions and data store specific functions @end menu @node Store Controllers @@ -109,6 +110,7 @@ @include includes/fun-elephant-find-item.texinfo @include includes/fun-elephant-map-pset.texinfo @include includes/fun-elephant-pset-list.texinfo + at include includes/fun-elephant-drop-pset.texinfo @c @node Query Interfaces @c @comment node-name, next, previous, up @@ -148,6 +150,7 @@ @include includes/fun-elephant-get-index.texinfo @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo + at include includes/fun-elephant-drop-pset.texinfo @node Cursors @comment node-name, next, previous, up @@ -161,33 +164,35 @@ @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo - at include includes/fun-elephant-cursor-current.texinfo - at include includes/fun-elephant-cursor-delete.texinfo @include includes/fun-elephant-cursor-duplicate.texinfo + at include includes/fun-elephant-cursor-current.texinfo @include includes/fun-elephant-cursor-first.texinfo - at include includes/fun-elephant-cursor-get-both-range.texinfo - at include includes/fun-elephant-cursor-get-both.texinfo @include includes/fun-elephant-cursor-last.texinfo - at include includes/fun-elephant-cursor-next-dup.texinfo - at include includes/fun-elephant-cursor-next-nodup.texinfo @include includes/fun-elephant-cursor-next.texinfo + at include includes/fun-elephant-cursor-prev.texinfo + at include includes/fun-elephant-cursor-set.texinfo + at include includes/fun-elephant-cursor-set-range.texinfo + at include includes/fun-elephant-cursor-get-both.texinfo + at include includes/fun-elephant-cursor-get-both-range.texinfo + at include includes/fun-elephant-cursor-delete.texinfo + at include includes/fun-elephant-cursor-put.texinfo + @include includes/fun-elephant-cursor-pcurrent.texinfo @include includes/fun-elephant-cursor-pfirst.texinfo - at include includes/fun-elephant-cursor-pget-both-range.texinfo - at include includes/fun-elephant-cursor-pget-both.texinfo @include includes/fun-elephant-cursor-plast.texinfo - at include includes/fun-elephant-cursor-pnext-dup.texinfo - at include includes/fun-elephant-cursor-pnext-nodup.texinfo @include includes/fun-elephant-cursor-pnext.texinfo - at include includes/fun-elephant-cursor-pprev-nodup.texinfo @include includes/fun-elephant-cursor-pprev.texinfo - at include includes/fun-elephant-cursor-prev-nodup.texinfo - at include includes/fun-elephant-cursor-prev.texinfo - at include includes/fun-elephant-cursor-pset-range.texinfo @include includes/fun-elephant-cursor-pset.texinfo - at include includes/fun-elephant-cursor-put.texinfo - at include includes/fun-elephant-cursor-set-range.texinfo - at include includes/fun-elephant-cursor-set.texinfo + at include includes/fun-elephant-cursor-pset-range.texinfo + at include includes/fun-elephant-cursor-pget-both.texinfo + at include includes/fun-elephant-cursor-pget-both-range.texinfo + + at include includes/fun-elephant-cursor-next-dup.texinfo + at include includes/fun-elephant-cursor-next-nodup.texinfo + at include includes/fun-elephant-cursor-pnext-dup.texinfo + at include includes/fun-elephant-cursor-pnext-nodup.texinfo + at include includes/fun-elephant-cursor-prev-nodup.texinfo + at include includes/fun-elephant-cursor-pprev-nodup.texinfo @node Transactions @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/21 17:22:35 1.17 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/25 02:27:56 1.18 @@ -442,19 +442,118 @@ The remaining problem outlined in the section on @ref{Serialization} is that operations which mutate collection types do not have persistent side effects. We have solved this problem for objects, but -not for collections such as as arrays, hashes or lists. Elephant's -solution to this problem is the @code{btree} class which provides -persistent addition, deletion and mutation of elements. - -The BTree stores arbitrarily sized sets of key-value pairs ordered by -key. Every key-value pair is stored independantly in Elephant just -like persistent object slots. They inherit all the important -properties of persistent objects: btree identity and fast -serialization / deserialization. They also resolve the mutated -substructure and nested aggregates problem for collections. Every -mutating write to a btree is an independent and persistent operation -and you can serialize or deserialize a btree without serializing any -of it's key-value pairs. +not for collections such as as arrays, hashes or lists. Elephant +provides two solutions to this problem: the @code{pset} and + at code{btree} classes. Each provides persistent addition, deletion and +mutation of elements, but the pset is a simple data structure that may +be more efficient in memory and time than the more general btree. + + at subsection Using PSets + +The persistent set maintains a persistent, unordered collection of +objects. They inherit all the important properties of persistent +objects: identity and fast serialization. They also resolve the +mutated substructure and nested aggregates problem for collections. +Every mutating write to a @code{pset} is an independent and persistent +operation and you can serialize or deserialize a @code{pset} without +serializing any of it's key-value pairs. + +The @code{pset} is also a very convenient data structure for enabling +a persistent slot contain a collection that can be updated without +deserializing and/or reserializing a list, array or hash table on +every access. + +Let's explore this data structure through a (very) simple social +networking example. + + at lisp +(defpclass person () + ((name :accessor person-name :initarg :name)) + ((friends :accessor person-friends :initarg :friends))) + at end lisp + +Our goal here is to store a list of friends that each person has, this +simple graph structure enables analyses such as who are the friends of +my friends, or do I know someone who knows X or what person has the +minimum degree of separation from everyone else? + +Without psets, we would have to do something like this: + + at lisp +(defmethod add-friend ((me person) (them person)) + (let ((friends (person-friends me))) + (pushnew them friends) + (setf (person-friends me) friends))) + +(defmethod remove-friend ((me person) (them person)) + (let ((remaining-friends (delete them (person-friends me)))) + (setf (person-friends me) remaining-friends))) + +(defmethod map-friends (fn (me person)) + (mapc fn (person-friends me))) + at end lisp + +Ouch! This results in a large amount of consing. We have to +deserialize and generate a freshly consed list every time we call + at code{person-friends} and then reserialize and discard it on every +call to @code{(setf person-friends)}. + +Instead, we can simply use a @code{pset} as the value of friends and +implement the add and remove friend operations as follows: + + at lisp +(defpclass person () + ((name :accessor person-name :initarg :name)) + ((friends :accessor person-friends :initarg :friends :initform (make-pset)))) + +(defmethod add-friend ((me person) (them person)) + (insert-item them (person-friends me))) + +(defmethod remove-friend ((me person) (them person)) + (remove-item them (person-friends me))) + +(defmethod map-friends (fn (me person)) + (map-pset fn (person-friends me))) + at end lisp + +If you want a list to be returned when the user calls person-friends +themselves, you can simply rejigger things like this: + + at lisp +(defpclass person () + ((name :accessor person-name :initarg :name)) + ((friends :accessor person-friends-set :initarg :friends :initform (make-pset)))) + +(defmethod person-friends ((me person)) + (pset-list (person-friends-set me))) + at end lisp + +If you just change the person-friends calls in our prior functions, +the new set of functions removes @code{(setf person-friends)}, which +doesn't make sense for a collection slot, allows users to get a list +of the friends for easy list manipulations and avoids all the consing +that plagued our earlier version. + +You can use a @code{pset} in any way you like just like a persistent +object. The only difference is the api used to manipulate it. +Instead of slot accessors, we use insert, remove, map and find. + +There is one drawback to persistent sets and that is that they are not +garbage collected. Over time, orphaned sets will eat up alot of disk +space. Therefore you need to explicitly free the space or resort to +more frequent uses of the migrate procedure to compact your database. +The pset supports the @code{drop-pset} + +However, given that persistent objects have the same explicit storage +property, using psets to create collection slots is a nice match. + + at subsection Using BTrees + +BTrees are collections of key-value pairs ordered by key with a log(N) +random access time and a rich iteration mechanism. Like persistent +sets, they solve all the collection problems of the prior sections. +Every key-value pair is stored independently in Elephant just like +persistent object slots. The primary interface to @code{btree} objects is through @code{get-value}. You use @code{setf} @code{get-value} to store --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/24 16:39:30 1.16 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/25 02:27:57 1.17 @@ -9,20 +9,22 @@ * The Store Controller:: Behind the curtain. * Serialization details:: The devil hides in the details. * Persistent Classes and Objects:: All the dirt on persistent objects. -* Class Indices:: In-depth discussion about indexing persistent indices. -* Using BTrees:: Using the native btree. -* Using Cursors:: Low-level access to BTrees. -* The BTree index:: Alternative ways to reference objects in btrees +* Class Indices:: In-depth discussion about indexing objects. +* Persistent Sets:: Using the persistent set. +* Persistent BTrees:: Using the native btree. +* BTree Cursors:: Low-level access to BTrees. +* BTree Indexing:: Alternative ways to reference objects in btrees. +* Index Cursors:: Low-level access to BTree indices. +* Multi-threaded Applications:: What considerations are required for safe multi-threading * Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls. * Multi-repository Operation:: Specifying repositories. -* Multi-threaded Applications:: What considerations are required for safe multi-threading * Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines? * Repository Migration and Upgrade:: How to move objects from one repository to another. -* Garbage Collection:: How to recover storage and OIDs in long-lived repositories. * Performance Tuning:: How to get the most from Elephant. +* Garbage Collection:: How to recover storage and OIDs in long-lived repositories. * Berkeley DB Data Store:: Commands and concerns specific to the :BDB data store -* CL-SQL Data Store:: Commands and concerns specific to the :CLSQL data store -* Postmodern Data Store:: +* CLSQL Data Store:: Commands and concerns specific to the :CLSQL data store +* Postmodern Data Store:: * Native Lisp Data Store:: @end menu @@ -46,7 +48,7 @@ @itemize @item Berkeley DB: '(:BDB "/path/to/datastore/directory/") - at item CL-SQL: '(:CLSQL ( )) + at item CLSQL: '(:CLSQL ( )) @end itemize Valid CLSQL database tags for @code{} are @@ -84,7 +86,7 @@ @end lisp The data store sections of the user guide (@ref{Berkeley DB Data -Store} and @ref{CL-SQL Data Store}) list all the data-store specific +Store} and @ref{CLSQL Data Store}) list all the data-store specific options to various elephant functions. When you finish your application, @code{close-store} will close the @@ -792,37 +794,155 @@ improve error handling for this case, so you can delete the derived index and continue performing the write to a persistent object that flagged the error.} - at node Using BTrees + at node Persistent Sets + at comment node-name, next, previous, up + at section Persistent Sets + +Persistent sets are fairly straightforward and are well-introduced by +the tutorial, please review the tutorial or read the reference section +for persistent sets. + + at node Persistent BTrees @comment node-name, next, previous, up - at section Using BTrees + at section Persistent BTrees -A BTree is a data structure designed for on-disk databases. -Traditional binary trees are a tree structure that stores a key and -value and two links in each node. To get to a value, you compare your -query key to the node key. If it is equal, return the value in this -node. If it is less, follow the first link and if it is greater, -follow the second link. The problem here is that every link requires -a disk seek. +A BTree is a data structure designed for on-disk databases. It's +design goal is to minimize the number of disk seeks while traversing +the tree structure. In contrast to a binary tree, the BTree exploits +the properties of memory/disk data heirarchies. Disk seeks are +expensive while loading large blocks of data is relatively inexpensive +and in-memory scanning of a block of memory is much cheaper than a +disk seek. This means a few, large nodes containing many keys is +a more balanced data structure than + +The BTree, or derivatives, are the basis of most record-oriented +database including SQL servers, Berkeley DB and many others. Elephant +directly exposes the BTree structure to the user so the user can +decide how best to manage and traverse it. Many of Elephant's other +facilities, such as the class indexing discussed above, are +implemented on top of the BTree. + +The basic interface to the BTree is via the @code{get-value} method. +Both the key and the value are serialized and then the BTree is +traversed according to the sorted order of the key and the value +inserted in its sorted order. Insertion, access and deletion (via + at code{remove-kv}) are all O(log N) complexity operations. + +Sorting in BTrees requires some discussion. The sorting constraints +on btrees are dictated by the original implementation on Berkeley DB. +The Berkeley DB data store sorts keys based on their serialized +representation. The CLSQL implementation has to sort based on the +deserialized lisp value, so sorted traversals require reading all the +objects into memory. This places some limitations on systems that +exploit the CLSQL implementation (@pxref{CLSQL Data Store} for more +information). + +Sorting is done first by primitive type (string, standard-class, +array, etc) and then by value within that type. The type order and +internal sorting constraint is: -The BTree exploits the properties of memory/disk data heirarchies. -The key properties are that disk seeks are expensive, loading large -blocks of data is relatively inexpensive after a seek and comparisons -on objects that are stored in memory is cheap. + at enumerate + at item Numbers. All numbers are sorted as a class by their numeric value. Effectively +all numbers are coerced into a double float and sorted relative to each other. + at item Strings. Because the serializer stores strings in variable width structures. Each width type is sorted separately, then sorted lexically. (NOTE: This should get fixed for 1.0. Strings should be sorted together) + at item Pathnames. Sorted by their string radix then lexically. + at item Symbols. Sorted by string radix, then lexically. + at item Aggregates. Sorted by type in the following order, then arbitrarily internally. Persistent instance references, cons, hash-table, standard objects, arrays, structs and then nil. + at end enumerate +String comparisons are case insensitive today, so @code{"Adam" = +"adam" > "Steve" }. When unicode support is finalized, comparisons +will be case sensitive. + +Like persistent sets, BTrees are not garbage collected so to recover +the storage of a BTree, just run the function @code{drop-btree} to +delete all the key-value pairs and return their storage to the +database for reuse. The oid used by the btree, however, will not be +recovered. + + at c *** FINISH *** + at node BTree Cursors + at comment node-name, next, previous, up + at section BTree Cursors +Aside from getting, setting and dropping key-value pairs from the +database, you can also traverse the BTree structure one key-value pair +at a time. + +Creating cursors (inside transactions) +Duplicating cursors + +Initializing cursors to location +- first, last, set +Traversing +- next, prev, (set) +Side effects +- delete +- put +- Can also use setf get-value while traversing - at node Using Cursors +Large cursor operations (locks, resources?) + + at c *** FINISH *** + at node BTree Indexing @comment node-name, next, previous, up - at section Using Cursors + at section BTree Indexing -- initialized, not-initialized - at node The BTree Index + at c *** FINISH *** + at node Index Cursors @comment node-name, next, previous, up - at section The BTree Index + at section Index Cursors + +Index cursors are just like BTree cursors except you can get the main +BTree value instead of the index value. There are also a parallel set +of operations such as @code{cursor-pnext} instead of + at code{cursor-next} which returns @code{exists}, @code{key}, @code{primary-btree-value} +and @code{index-value = primary-btree-key}. + +Operations that have the same behavior, but return primary btree values and keys are: + + at c need table here + + at itemize + at item cursor-first -> cursor-pfirst, cursor-last -> cursor-plast + at item cursor-current -> cursor-pcurrent + at item cursor-next -> cursor-pnext, cursor-prev -> cursor-pprev + at item cursor-set-range + at end itemize + +The big difference between btree cursors and index cursors is that +indices can have duplicate key values. This means we have to choose +between incrementing over elements, unique key-values or only within +a duplicate segment. There are cursor operations for each: + + at itemize + at item Simple move. Standard btree operations plus @code{cursor-pnext} and @code{cursor-pprev}. + at item Move to next key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup}. + at item Move to next duplicate. + at end itemize + + at c FINISH + at node Multi-threaded Applications + at comment node-name, next, previous, up + at section Multi-threaded Applications + +Berkeley DB plays well with threads and processes. The store +controller is thread-safe by default, that is, can be shared amongst +threads. This is enabled by the @code{:thread} keyword argument which +defaults to true. Transactions may not be shared amongst threads +except serially. One thing which is NOT thread and process safe is +recovery, which should be run when no one is else is talking to the +database environment. + +The following shared regions of Elephant are protected by standard locks: +- buffer stream pool +- access to serializer circular buffers +- writes to connection db +- where else? -Empty. + at c *** FINISH *** @node Transaction Details @comment node-name, next, previous, up @section Transaction Details @@ -838,7 +958,7 @@ ;; ;; User and designer considerations: ;; - *current-transaction* is reserved for use by dynamic transaction context. The default global -;; value must always be null (no transaction). Each backend can set it to a different parameter +;; value must always be null (no transaction). Each data storeend can set it to a different parameter ;; within the dynamic context of an execute-transaction. ;; - Any closures returned from within a transaction cannot bind *current-transaction* ;; - Only a normal return value will result in the transaction being committed, any non-local exit @@ -848,45 +968,7 @@ ;; knowing that the transaction will be aborted ;; - at node Multi-threaded Applications - at comment node-name, next, previous, up - at section Multi-threaded Applications - -Sleepycat plays well with threads and processes. The store controller -is thread-safe by default, that is, can be shared amongst threads. -This is set by the @code{:thread} key. Transactions may not be shared -amongst threads except serially. One thing which is NOT thread and -process safe is recovery, which should be run when no one is else is -talking to the database environment. Consult the Sleepycat docs for -more information. - -Elephant uses some specials to hold parameters and buffers. If you're -using a natively threaded lisp, you can initialize these specials to -thread-local storage by using the @code{run-elephant-thread} function, -assuming your lisp creates thread-local storage for let-bound -specials. (This functionality is currently broken) - -Persisting ordinary aggregate types (e.g. NOT persistent classes or -btrees) suffers from something called "merge-conflicts." Since -updating one value of an aggregate object requires the entire object -to be written to the database, in heavily threaded situations you may -overwrite changes another thread or process has committed. This is -not protected by transactions! - -Consider two processes operating on the same cons: - - at code{ ------start---read--update-car--write--commit---------------- --start------read--update-cdr-----------------write--commit-- -} - -Although the first process successfully committed its transaction, its -work (writing to the car) will be erased by the second transaction -(which writes both the car and cdr.) - -Persistent classes and persistent collections do not suffer from -merge-conflicts, since each slot / entry is a separate database entry. - + at c *** FINISH *** @node Multi-repository Operation @comment node-name, next, previous, up @section Multi-repository Operation @@ -895,7 +977,7 @@ actual database connections. If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CL-SQL connection specification. +If it is a list, it is a assumed to be a CLSQL connection specification. For example: @lisp ELE-TESTS> *testdb-path* @@ -934,7 +1016,15 @@ @comment node-name, next, previous, up @section Multiple Processes and Distributed Applications -Can we do this? What do we have to say about this? +Just start up two lisp images and connect to the same database. +Transactions will ensure there is no interaction between processes. +This has not been extensively tested, but should work without any +problem. Any field experience will get reflected in this section of the manual + +Distributed applications may be supported if the underlying SQL server +or an appropriate Berkeley DB database is used. We provide no +documentation nor have we heard of this use-case. This remains +fertile ground for future investigation. @node Repository Migration and Upgrade @comment node-name, next, previous, up @@ -1070,19 +1160,153 @@ @comment node-name, next, previous, up @section Garbage Collection -GC is not implemented, but migration (@pxref{Repository Migration and -Upgrade}) will consolidate storage and recover OIDs which emulates GC. -No online solution is currently supported. - +Garbage collection is not implemented as part of the persistent object +protocol. However, the migration (@pxref{Repository Migration and +Upgrade}) mechanism will consolidate storage and recover OIDs which is +an effective offline GC. No online solution is currently anticipated. @node Berkeley DB Data Store @comment node-name, next, previous, up @section Berkeley DB Data Store +This section briefly describes special facilities of the Berkeley DB +data store and explains how persistent objects map onto it. Elephant +was originally written targeting only Berkeley DB. As such, the +design of Elephant was heavily influenced by the Berkeley DB architecture. + +Berkeley DB is a C library that very efficiently implements a database +by allowing the application to directly manipulate the memory pools +and disk storage without requiring communication through a server as +in many relational database applications. The library supports +multi-threaded and multi-process transactions through a shared memory +region that provides for shared buffer pools, shared locks, etc. Each +process in a multi-process application is independently linked to the +library, but shares the memory pool and disk storage. + +The following subsections discuss places where Berkeley DB provides +additional facilities to the Elephant interfaces described above. + + at subsection Architecture Overview + +The Berkeley DB data store (indicated by a @code{:BDB} in the data +store specification) supports the Elephant protocols using Berkeley DB +as a backend. The primary features of the BDB library that are used +are BTree databases, the transactional subsystem, a shared buffer pool +and unique ID sequences. + +All data written to the data store ends up in a BTree slot using a +transaction. There are two databases, one for persistent slot values +and one for btrees. The mapping of Elephant objects is quite simple. + +Persistent slots are written to a btree using a unique key and the +serialized value being written. The key is the oid of the persistent +object concatenated to the serialized name of the slot being written. +This ordering groups slots together on the disk + + at subsection Opening a Store + +When opening a store there are several special options you can invoke: + + at itemize + at item @strong{@code{:recover}} tells Berkeley DB to run recovery on the + underlying database. This is reasonably cheap if you do not need + to run recovery, but can take a very long time if you let your log + files get too long. This option must be run in a single-threaded + mode before other threads or processes are accessing the same database. + at item @strong{@code{:recover-fatal}} runs Berkeley DB catastrophic recovery (see BDB documentation). + at item @strong{@code{:thread}} set this to nil if you want to run single threaded, + it avoids locking overhead on the environment. The default is + to run @emph{free-threaded}. + at item The @strong{@code{:deadlock-detect}} launches a background process via + the run-shell commands of lisp. This background process connects to a Berkeley + DB database and runs a regular check for deadlock, freeing locks as appropriate + when it finds them. This can avoid a set of annoying crashes in Berkeley DB, + the very crashes that, in part, motivated Franz to abandon AllegroStore and write + the pure-Lisp AllegroCache. + at end itemize + + at subsection Starting a Transaction + +Berkeley DB transactions have a number of additional keyword +parameters that can help you tune performance or change the semantics +in Berkeley DB applications. They are summaried briefly here, see the +BDB docs for detailed information: + + at itemize + at item @strong{@code{:degree-2}} This option provides for cursor stability, that is whatever + object the cursor is currently at will not change, however prior +values read may change. This can significantly enhance performance if +you frequently map over a btree as it doesn't lock the entire btree, +just the current element. All transactions running concurrently over +the btree can commit without restarting. The global parameter + at code{*map-using-degree2*} determines the default behavior of this +option. It is set to true by default so that map has similar +semantics to lists. This violates both @emph{Atomicity and +Consistency} depending on how it is used. + at item @strong{@code{:read-uncommitted}} Allows reading data that has been written by other [79 lines skipped] From ieslick at common-lisp.net Wed Apr 25 02:28:01 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 22:28:01 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-bdb Message-ID: <20070425022801.40BAB20014@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv4997/src/db-bdb Modified Files: bdb-collections.lisp bdb-controller.lisp bdb-transactions.lisp berkeley-db.lisp libberkeley-db.c Log Message: Export bdb performance tweaks; lots more documentation; new ops for libberkeley-db --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-collections.lisp 2007/04/12 02:47:23 1.22 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-collections.lisp 2007/04/25 02:27:58 1.23 @@ -676,8 +676,7 @@ (error "cursor-get-both-range not implemented on secondary indices. Use cursor-pget-both-range.")) (defmethod cursor-put ((cursor bdb-secondary-cursor) value &rest rest) - "Puts are forbidden on secondary indices. Try adding to -the primary." + "Puts are forbidden on secondary indices. Try adding to the primary." (declare (ignore rest value) (ignorable cursor)) (error "Puts are forbidden on secondary indices. Try adding to the primary.")) --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/04/21 17:22:36 1.34 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-controller.lisp 2007/04/25 02:27:58 1.35 @@ -83,13 +83,14 @@ :name "%ELEPHANT"))))) (setf (controller-environment sc) env) (db-env-set-flags env 0 :auto-commit t) + (db-env-set-cachesize env 0 elephant::*berkeley-db-cachesize* 1) + (db-env-set-timeout env 100000 :set-transaction-timeout t) + (db-env-set-timeout env 100000 :set-lock-timeout t) (db-env-open env (namestring (second (controller-spec sc))) :create t :init-rep nil :init-mpool t :thread thread :init-lock t :init-log t :init-txn t :recover recover :recover-fatal recover-fatal ) - (db-env-set-timeout env 100000 :set-transaction-timeout t) - (db-env-set-timeout env 100000 :set-lock-timeout t) (let ((metadata (db-create env)) (db (db-create env)) (btrees (db-create env)) @@ -111,26 +112,30 @@ ;; Open main class, slot-value and index databases (setf (controller-db sc) db) (db-open db :file "%ELEPHANT" :database "%ELEPHANTDB" - :auto-commit t :type DB-BTREE :create t :thread thread) + :auto-commit t :type DB-BTREE :create t :thread thread + :read-uncommitted t) (setf (controller-btrees sc) btrees) (db-bdb::db-set-lisp-compare btrees (controller-serializer-version sc)) (db-open btrees :file "%ELEPHANT" :database "%ELEPHANTBTREES" - :auto-commit t :type DB-BTREE :create t :thread thread) + :auto-commit t :type DB-BTREE :create t :thread thread + :read-uncommitted t) (setf (controller-indices sc) indices) (db-bdb::db-set-lisp-compare indices (controller-serializer-version sc)) (db-bdb::db-set-lisp-dup-compare indices (controller-serializer-version sc)) (db-set-flags indices :dup-sort t) (db-open indices :file "%ELEPHANT" :database "%ELEPHANTINDICES" - :auto-commit t :type DB-BTREE :create t :thread thread) + :auto-commit t :type DB-BTREE :create t :thread thread + :read-uncommitted t) (setf (controller-indices-assoc sc) indices-assoc) (db-bdb::db-set-lisp-compare indices-assoc (controller-serializer-version sc)) (db-bdb::db-set-lisp-dup-compare indices-assoc (controller-serializer-version sc)) (db-set-flags indices-assoc :dup-sort t) (db-open indices-assoc :file "%ELEPHANT" :database "%ELEPHANTINDICES" - :auto-commit t :type DB-UNKNOWN :thread thread) ;; :rdonly t) + :auto-commit t :type DB-UNKNOWN :thread thread + :read-uncommitted t) (db-bdb::db-fake-associate btrees indices-assoc :auto-commit t) (let ((db (db-create env))) --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/12 02:47:23 1.13 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/25 02:27:58 1.14 @@ -25,7 +25,7 @@ &key transaction parent environment (retries 100) - degree-2 dirty-read txn-nosync txn-nowait txn-sync) + degree-2 read-uncommitted txn-nosync txn-nowait txn-sync) (let ((env (if environment environment (controller-environment sc)))) (loop for count fixnum from 1 to retries @@ -36,7 +36,7 @@ (db-transaction-begin env :parent (if parent parent +NULL-VOID+) :degree-2 degree-2 - :dirty-read dirty-read + :read-uncommitted read-uncommitted :txn-nosync txn-nosync :txn-nowait txn-nowait :txn-sync txn-sync)))) @@ -65,7 +65,7 @@ txn-nosync txn-nowait txn-sync - dirty-read + read-uncommitted degree-2 &allow-other-keys) (assert (not *current-transaction*)) @@ -74,7 +74,7 @@ :txn-nosync txn-nosync :txn-nowait txn-nowait :txn-sync txn-sync - :dirty-read dirty-read + :read-uncommitted read-uncommitted :degree-2 degree-2)) --- /project/elephant/cvsroot/elephant/src/db-bdb/berkeley-db.lisp 2007/04/12 02:47:23 1.11 +++ /project/elephant/cvsroot/elephant/src/db-bdb/berkeley-db.lisp 2007/04/25 02:27:58 1.12 @@ -1526,6 +1526,28 @@ :flags (set-lock-timeout set-transaction-timeout) :documentation "Gets the timout.") +(def-function ("db_env_set_cachesize" %db-env-set-cachesize) + ((env :pointer-void) + (gbytes :unsigned-int) + (bytes :unsigned-int) + (ncache :int)) + :returning :int) + +(wrap-errno db-env-set-cachesize (env gbytes bytes ncache) + :documentation "Sets the size of the buffer pool cache + for elephant database data. Set large if you can!") + +(def-function ("db_env_get_cachesize" %db-env-get-cachesize) + ((env :pointer-void) + (gbytes :unsigned-int :out) + (bytes :unsigned-int :out) + (ncache :int :out)) + :returning :int) + +(wrap-errno db-env-get-cachesize (env) :outs 4 + :documentation "Return the current cache size of + the BDB environment buffer pool") + (def-function ("db_env_set_lk_detect" %db-env-set-lock-detect) ((env :pointer-void) (detect :unsigned-int)) --- /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/04/12 02:47:23 1.11 +++ /project/elephant/cvsroot/elephant/src/db-bdb/libberkeley-db.c 2007/04/25 02:27:58 1.12 @@ -661,6 +661,14 @@ return env->get_timeout(env, timeoutp, flags); } +int db_env_set_cachesize(DB_ENV *env, u_int32_t gbytes, u_int32_t bytes, int ncache) { + return env->set_cachesize(env, gbytes, bytes, ncache); +} + +int db_env_get_cachesize(DB_ENV *env, u_int32_t *gbytes, u_int32_t *bytes, int *ncache) { + return env->get_cachesize(env, gbytes, bytes, ncache); +} + int db_env_set_lk_detect(DB_ENV *env, u_int32_t detect) { return env->set_lk_detect(env, detect); } @@ -1002,7 +1010,7 @@ #define S2_FILL_POINTER_P 0x40 #define S2_ADJUSTABLE_P 0x80 -#define type_numeric2(c) (((c)<9) || ((c)==14) || ((c)==30)) +#define type_numeric2(c) (((c)<9) || ((c)==22)) /****** Serialized BTree keys have the form: From ieslick at common-lisp.net Wed Apr 25 02:28:18 2007 From: ieslick at common-lisp.net (ieslick) Date: Tue, 24 Apr 2007 22:28:18 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070425022818.D41AB34052@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv4997/src/elephant Modified Files: classes.lisp collections.lisp controller.lisp package.lisp pset.lisp variables.lisp Log Message: Export bdb performance tweaks; lots more documentation; new ops for libberkeley-db --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/24 16:39:30 1.31 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/25 02:28:01 1.32 @@ -186,10 +186,12 @@ ;; (defmethod change-class ((inst persistent) (class t) &rest rest) - (error "Changing a persistent instance's class to a non-persistent class is not currently allowed")) + (cerror "Ignore and continue?" + "Changing a persistent instance's class to a non-persistent class is not currently allowed")) (defmethod change-class ((inst standard-object) (class persistent-metaclass) &rest rest) - (error "Changing a standard instance to a persistent instance is not supported")) + (cerror "Ignore and continue?" + "Changing a standard instance to a persistent instance is not supported")) (defmethod update-instance-for-different-class :around ((previous persistent) (current persistent) &rest initargs &key) (let* ((old-class (class-of previous)) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/24 03:02:27 1.25 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/25 02:28:01 1.26 @@ -354,7 +354,7 @@ (defun lisp-compare-equal (a b) (equal a b)) -(defgeneric map-btree (fn btree &rest args &key start end value) +(defgeneric map-btree (fn btree &rest args &key start end value &allow-other-keys) (:documentation "Map btree maps over a btree from the value start to the value of end. If values are not provided, then it maps over all values. BTrees do not have duplicates, but map-btree can also be used with indices @@ -365,9 +365,9 @@ ;; function orders by type tag and nil is the highest valued type tag so nils are the last ;; possible element in a btree ordered by value. -(defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p)) +(defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p) &allow-other-keys) (let ((end (if value-set-p value end))) - (ensure-transaction (:store-controller (get-con btree)) + (ensure-transaction (:store-controller (get-con btree) :degree-2 *map-using-degree2*) (with-btree-cursor (curs btree) (multiple-value-bind (exists? key value) (cond (value-set-p @@ -386,7 +386,7 @@ (funcall fn k v) (return nil))))))))) -(defgeneric map-index (fn index &rest args &key start end value from-end collect) +(defgeneric map-index (fn index &rest args &key start end value from-end collect &allow-other-keys) (:documentation "Map-index is like map-btree but for secondary indices, it takes a function of three arguments: key, value and primary key. As with map-btree the keyword arguments start and end @@ -398,7 +398,8 @@ and end.")) (defmethod map-index (fn (index btree-index) &rest args - &key start end (value nil value-set-p) from-end collect) + &key start end (value nil value-set-p) from-end collect + &allow-other-keys) (declare (dynamic-extent args)) (unless (or (null start) (null end) (lisp-compare<= start end)) (error "map-index called with start = ~A and end = ~A. Start must be less than or equal to end according to elephant::lisp-compare<=." @@ -410,7 +411,7 @@ (push (funcall fn k v pk) results))) (let ((fn (if collect #'collector fn))) (declare (dynamic-extent (function collector))) - (ensure-transaction (:store-controller sc) + (ensure-transaction (:store-controller sc :degree-2 *map-using-degree2*) (with-btree-cursor (cur index) (labels ((continue-p (key) ;; Do we go to the next value? --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/23 02:26:53 1.49 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/25 02:28:01 1.50 @@ -133,6 +133,11 @@ name (asdf:find-system :elephant))) +(defun initialize-user-parameters () + (loop for (keyword variable) in *user-configurable-parameters* do + (awhen (get-user-configuration-parameter keyword) + (setf variable it)))) + ;; ;; COMMON STORE CONTROLLER FUNCTIONALITY ;; @@ -465,6 +470,8 @@ their *store-controller* to a given dynamic context or wrap each store-specific op in a transaction using with or ensure transaction" (assert (consp spec)) + ;; Ensure that parameters are set + (initialize-user-parameters) (let ((controller (get-controller spec))) (apply #'open-controller controller args) (if *store-controller* --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/22 03:35:09 1.32 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/25 02:28:02 1.33 @@ -208,7 +208,7 @@ #:persistent #:persistent-object #:persistent-metaclass #:defpclass #:persistent-collection #:drop-pobject - #:pset #:make-pset #:insert-item #:remove-item #:map-pset #:find-item #:pset-list + #:pset #:make-pset #:insert-item #:remove-item #:map-pset #:find-item #:pset-list #:drop-pset #:btree #:make-btree #:get-value #:remove-kv #:existsp --- /project/elephant/cvsroot/elephant/src/elephant/pset.lisp 2007/04/12 02:47:33 1.2 +++ /project/elephant/cvsroot/elephant/src/elephant/pset.lisp 2007/04/25 02:28:02 1.3 @@ -54,6 +54,9 @@ (:documentation "Construct an empty default pset or backend specific pset. This is an internal function used by make-pset")) +(defgeneric drop-pset (pset) + (:documentation "Release pset storage to database for reuse")) + ;; NOTE: Other operators? ;; - Efficient union, intersection and difference fn's exploiting an underlying ;; sorted order? @@ -117,6 +120,16 @@ (push item list)) (pset-btree pset))) list)) + +(defmethod drop-pset ((pset default-pset)) + (ensure-transaction (:store-controller *store-controller*) + (with-btree-cursor (cur (pset-btree pset)) + (loop for exists? = (cursor-first cur) + then (cursor-next cur) + while exists? + do (cursor-delete cur))))) + + --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/24 12:58:10 1.16 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/25 02:28:02 1.17 @@ -39,11 +39,32 @@ error") ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;;;; General support for user configurable parameters + +(defvar *user-configurable-parameters* + '((:map-using-degree2 *map-using-degree2*) + (:berkeley-db-cachesize *berkeley-db-cachesize*))) + +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;; Optimization parameters (defvar *circularity-initial-hash-size* 50 "This is the default size of the circularity cache used in the serializer") +(defparameter *map-using-degree2* t + "This parameter enables an optimization for the Berkeley DB data store + that allows a map operator to walk over a btree without locking all + read data, it only locks written objects and the current object") + +(defvar *berkeley-db-cachesize* 10485760 + "This parameter controls the size of the berkeley db data store page + cache. This parameter can be increased by to 4GB on 32-bit machines + and much larger on other machines. Using the db_stat utility to identify + cache hit frequency on your application is a good way to tune this number. + The default is 20 megabytes specified in bytes. If you need to specify + Gigbyte + cache sizes, talk to the developers! This is ignored for + existing databases that were created with different parameters") + ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Legacy Thread-local specials From rread at common-lisp.net Wed Apr 25 03:18:14 2007 From: rread at common-lisp.net (rread) Date: Tue, 24 Apr 2007 23:18:14 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070425031814.230903F003@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv12845 Modified Files: scenarios.texinfo Log Message: Fixing a typo. --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/21 17:22:35 1.7 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/25 03:18:13 1.8 @@ -803,7 +803,7 @@ system are also implemented as DCM objects. Binary objects, such as uploaded PDFs, can be attached to objects as comments and are stored directly in Elephant. Konsenti is based on utf-8, and unicode -characters outside of the ISO-9959-1 character set are routinely +characters outside of the ISO-8859-1 character set are routinely stored in Elephant. Konsenti uses Postgres as a backend for licensing reasons; but use of other data stores is possible. From rread at common-lisp.net Thu Apr 26 03:10:13 2007 From: rread at common-lisp.net (rread) Date: Wed, 25 Apr 2007 23:10:13 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070426031013.0E1AB20016@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv21396 Modified Files: user-guide.texinfo Log Message: Added CLSQL section. --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/25 02:27:57 1.17 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/26 03:10:13 1.18 @@ -1308,7 +1308,40 @@ @comment node-name, next, previous, up @section CLSQL Data Store - +Elephant uses Kevin Rosenberg's excellent @uref{http://clsql.b9.com, CLSQL} + CLSQL lisp binding to relational databases +(it does not use the ORM functionality offered by that package.) +CLSQL interfaces to many databases (Postgres, MySQL, Oracle, +ODBC, SQLite3, Microsoft SQL Server (via ODBC)). Right now, Elephant has +been tested with Postgress and SQLite3. Probably getting it to work with +one of the others will take a small amount of debugging; in principle there +is no reason it won't work out of the box. We invite users to try +other database, and will quickly incorporate patches needed to make them +work. + +Because CLSQL is very generic, the CLSQL interface does not offer any +special feature as discussed in the previous section @ref{Berkeley DB Data Store}. + + at subsection Basic CLSQL Implementation + +The CLSQL uses base64 encoding to store binary data as text directly. This +has the advantage that it works with all databases, which tend to differ +widely in their treatment of Binary Large Objects (BLOBs.) It imposes some +obvious overhead. + +The CLSQL implementation is structurally exactly the same as the BDB implementation. +A single table is created to hold all (key,value) pairs. An index on the key column +provides efficient key lookup. No additional indexing offered by the underlying +databases is used. This has the advantage that the API is exactly the same as the +BDB api, and all of the functional indexes, cursors, and secondary indexes work exactly +the same way. It does not exploit the performance that a database-specific solution +would offer (see @ref{Postmodern Data Store} for an example of such a system. + +Our basic strategy is to leave the CLSQL interface as simple as possible, in order to +work with as many databases as possible. When there is enough motivation to support +a backend that is specific to one database (and therefore probably faster), such an +interface can be placed into the ``contrib'' directory and migrated into the main +code base as time allows the complete integration with the test suite. @node Postmodern Data Store @comment node-name, next, previous, up @@ -1317,6 +1350,9 @@ The postmodern data store is not yet integrated. It should be documented for the forthcoming release 0.9.1 or 0.9.2. +This backend will presumably be much faster, when used against PostGres, than the +generic CLSQL store. + @node Native Lisp Data Store @comment node-name, next, previous, up @section Native Lisp Data Store From rread at common-lisp.net Fri Apr 27 02:09:16 2007 From: rread at common-lisp.net (rread) Date: Thu, 26 Apr 2007 22:09:16 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/contrib/rread/dcm Message-ID: <20070427020916.2E10113038@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/contrib/rread/dcm In directory clnet:/tmp/cvs-serv22391 Modified Files: dcm.lisp Log Message: Minor keyword change --- /project/elephant/cvsroot/elephant/src/contrib/rread/dcm/dcm.lisp 2007/02/20 15:54:21 1.2 +++ /project/elephant/cvsroot/elephant/src/contrib/rread/dcm/dcm.lisp 2007/04/27 02:09:16 1.3 @@ -393,10 +393,10 @@ (defmethod initialize-btree ((dir elephant-director) c) (let* ((name (format nil "DCM-SPECIAL-~A" c)) (sc (slot-value dir 'root)) - (bt (get-from-root name :store-controller sc))) + (bt (get-from-root name :sc sc))) (format t "bt of name ~A is: ~A~%" name bt) (unless bt - (setf bt (add-to-root name (make-btree sc) :store-controller sc))) + (setf bt (add-to-root name (make-btree sc) :sc sc))) (setf (slot-value dir 'dcm-btree) bt)) dir ) @@ -409,7 +409,7 @@ ;; excellent idea. ;; (defun empty-out-corrupted-btree (c sc) ;; (let* ((name (format nil "DCM-SPECIAL-~A" (class-name c))) -;; (bt (get-from-root name :store-controller sc))) +;; (bt (get-from-root name :sc sc))) ;; "delete from keyvalue where clct_id = ") ;; ) From ieslick at common-lisp.net Fri Apr 27 13:32:14 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 09:32:14 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070427133214.0E7981120@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv32343 Modified Files: TODO Log Message: Bug fixes to change-class; drop-btree; enable :from-end and :collect on map-btree (not map-class though); export and documentation edits --- /project/elephant/cvsroot/elephant/TODO 2007/04/25 02:27:56 1.81 +++ /project/elephant/cvsroot/elephant/TODO 2007/04/27 13:32:11 1.82 @@ -31,10 +31,6 @@ - Add notes about deadlock-detect - Add notes about checkpoint (null in SQL?) -Additional features to document in Trac: -- Add error/restart conditions when lose data on: class redef, index drop, etc. -- :association slot option? - 0.6.1 - Features COMPLETED to date ---------------------------------- From ieslick at common-lisp.net Fri Apr 27 13:32:16 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 09:32:16 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-bdb Message-ID: <20070427133216.EC13C310A5@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv32343/src/db-bdb Modified Files: bdb-transactions.lisp Log Message: Bug fixes to change-class; drop-btree; enable :from-end and :collect on map-btree (not map-class though); export and documentation edits --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/25 02:27:58 1.14 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/27 13:32:14 1.15 @@ -57,7 +57,12 @@ (db-transaction-abort txn))))) (when success (return (values-list result))))) - finally (error "Too many retries in transaction")))) + finally (cerror "Retry transaction again?" + 'transaction-retry-count-exceeded + :format-control "Transaction exceeded the ~A retries limit" + :format-arguments (list retries) + :count retries)))) + (defmethod controller-start-transaction ((sc bdb-store-controller) &key From ieslick at common-lisp.net Fri Apr 27 13:32:28 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 09:32:28 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070427133228.7218C3109D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv32343/src/elephant Modified Files: classes.lisp collections.lisp package.lisp query.lisp transactions.lisp Log Message: Bug fixes to change-class; drop-btree; enable :from-end and :collect on map-btree (not map-class though); export and documentation edits --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/25 02:28:01 1.32 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/27 13:32:16 1.33 @@ -185,13 +185,17 @@ ;; CLASS CHANGE PROTOCOL ;; -(defmethod change-class ((inst persistent) (class t) &rest rest) - (cerror "Ignore and continue?" - "Changing a persistent instance's class to a non-persistent class is not currently allowed")) - -(defmethod change-class ((inst standard-object) (class persistent-metaclass) &rest rest) - (cerror "Ignore and continue?" - "Changing a standard instance to a persistent instance is not supported")) +(defmethod change-class :around ((previous persistent) (new-class standard-class) &rest initargs) + (declare (ignorable initargs)) + (unless (subtypep (type-of new-class) 'persistent-metaclass) + (error "Persistent instances cannot be changed to non-persistent classes in change-class")) + (call-next-method)) + +(defmethod change-class :around ((previous standard-object) (new-class persistent-metaclass) &rest initargs) + (declare (ignorable initargs)) + (unless (subtypep (type-of previous) 'persistent) + (error "Standard classes cannot be changed to non-persistent classes in change-class")) + (call-next-method)) (defmethod update-instance-for-different-class :around ((previous persistent) (current persistent) &rest initargs &key) (let* ((old-class (class-of previous)) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/25 02:28:01 1.26 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/27 13:32:17 1.27 @@ -70,6 +70,18 @@ (defmethod optimize-layout ((bt t) &key &allow-other-keys) t) +(defgeneric drop-btree (bt) + (:documentation "Delete all key-value pairs from the btree and + render it an invalid object in the data store")) + +(defmethod drop-btree ((bt btree)) + (ensure-transaction (:store-controller *store-controller*) + (with-btree-cursor (cur bt) + (loop for (exists? key) = (multiple-value-list (cursor-first cur)) + then (multiple-value-list (cursor-next cur)) + while exists? + do (remove-kv key bt))))) + ;; ;; Btrees that support secondary indices ;; @@ -161,17 +173,17 @@ (defclass cursor () ((oid :accessor cursor-oid :type fixnum :initarg :oid) -;; (intialized-p cursor) means that the cursor has -;; a legitimate position, not that any initialization -;; action has been taken. The implementors of this abstract class -;; should make sure that happens under the sheets... -;; According to my understanding, cursors are initialized -;; when you invoke an operation that sets them to something -;; (such as cursor-first), and are uninitialized if you -;; move them in such a way that they no longer have a legimtimate -;; value. (initialized-p :accessor cursor-initialized-p - :type boolean :initform nil :initarg :initialized-p) + :type boolean :initform nil :initarg :initialized-p + :documentation "Predicate indicating whether +the btree in question is initialized or not. Initialized means +that the cursor has a legitimate position, not that any +initialization action has been taken. The implementors of this +abstract class should make sure that happens under the +sheets... Cursors are initialized when you invoke an operation +that sets them to something (such as cursor-first), and are +uninitialized if you move them in such a way that they no longer +have a legimtimate value.") (btree :accessor cursor-btree :initarg :btree)) (:documentation "A cursor for traversing (primary) BTrees.")) @@ -240,13 +252,13 @@ (defgeneric cursor-delete (cursor) (:documentation - "Delete by cursor. The cursor is at an invalid position -after a successful delete.")) + "Delete by cursor. The cursor is at an invalid position, +and uninitialized, after a successful delete.")) (defgeneric cursor-put (cursor value &key key) (:documentation - "Put by cursor. Currently doesn't properly move the -cursor.")) + "Overwrite value at current cursor location. Currently does + not properly move the cursor.")) (defclass secondary-cursor (cursor) () (:documentation "Cursor for traversing secondary indices.")) @@ -354,7 +366,7 @@ (defun lisp-compare-equal (a b) (equal a b)) -(defgeneric map-btree (fn btree &rest args &key start end value &allow-other-keys) +(defgeneric map-btree (fn btree &rest args &key start end value from-end collect &allow-other-keys) (:documentation "Map btree maps over a btree from the value start to the value of end. If values are not provided, then it maps over all values. BTrees do not have duplicates, but map-btree can also be used with indices @@ -365,26 +377,47 @@ ;; function orders by type tag and nil is the highest valued type tag so nils are the last ;; possible element in a btree ordered by value. -(defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p) &allow-other-keys) - (let ((end (if value-set-p value end))) +(defmethod map-btree (fn (btree btree) &rest args &key start end (value nil value-set-p) + from-end collect &allow-other-keys) + (let ((end (if value-set-p value end)) + (results nil)) (ensure-transaction (:store-controller (get-con btree) :degree-2 *map-using-degree2*) (with-btree-cursor (curs btree) - (multiple-value-bind (exists? key value) - (cond (value-set-p - (cursor-set curs value)) - ((null start) - (cursor-first curs)) - (t (cursor-set-range curs start))) - (if exists? - (funcall fn key value) - (return-from map-btree nil)) - (loop - (multiple-value-bind (exists? k v) - (cursor-next curs) - (declare (dynamic-extent exists? k v)) - (if (and exists? (or (null end) (lisp-compare<= k end))) - (funcall fn k v) - (return nil))))))))) + (flet ((continue-p (key) + ;; Do we go to the next value? + (or (if from-end (null start) (null end)) + (if from-end + (or (not (lisp-compare<= key start)) + (lisp-compare-equal key start)) + (lisp-compare<= key end)))) + (collector (k v) + (push (funcall fn k v) results))) + (let ((fn (if collect #'collector fn))) + (declare (dynamic-extent (function continue-p) (function collector))) + (multiple-value-bind (exists? key value) + (cond (value-set-p + (cursor-set curs value)) + ((and (not from-end) (null start)) + (cursor-first curs)) + ((and from-end (null end)) + (cursor-last curs)) + (t (if from-end + (cursor-set-range curs end) + (cursor-set-range curs start)))) + (declare (dynamic-extent exists? k v)) + (if exists? + (funcall fn key value) + (return-from map-btree nil)) + (loop + (multiple-value-bind (exists? k v) + (if from-end + (cursor-prev curs) + (cursor-next curs)) + (declare (dynamic-extent exists? k v)) + (if (and exists? (continue-p k)) + (funcall fn k v) + (return nil))))))))) + results)) (defgeneric map-index (fn index &rest args &key start end value from-end collect &allow-other-keys) (:documentation "Map-index is like map-btree but for secondary indices, it --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/25 02:28:02 1.33 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/27 13:32:17 1.34 @@ -216,12 +216,11 @@ #:btree-index #:add-index #:get-index #:remove-index #:map-indices #:get-primary-key #:primary #:key-form #:key-fn - #:with-btree-cursor #:map-btree #:map-index + #:with-btree-cursor #:map-btree #:map-index #:drop-btree #:empty-btree-p #:dump-btree #:btree-keys #:btree-differ-p #:cursor #:secondary-cursor #:make-cursor #:make-simple-cursor - #:cursor-close #:cursor-init - #:cursor-duplicate #:cursor-current #:cursor-first + #:cursor-close #:cursor-duplicate #:cursor-current #:cursor-first #:cursor-last #:cursor-next #:cursor-next-dup #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup #:cursor-set #:cursor-set-range #:cursor-get-both @@ -229,8 +228,8 @@ #:cursor-pcurrent #:cursor-pfirst #:cursor-plast #:cursor-pnext #:cursor-pnext-dup #:cursor-pnext-nodup #:cursor-pprev #:cursor-pprev-nodup #:cursor-pset - #:cursor-pset-range #:cursor-pget-both - #:cursor-pget-both-range + #:cursor-pset-range #:cursor-pget-both #:cursor-pget-both-range + #:cursor-initialized-p #:find-class-index #:find-inverted-index #:enable-class-indexing #:disable-class-indexing @@ -240,7 +239,7 @@ #:report-indexed-classes #:class-indexedp-by-name - #:map-class #:map-class-index + #:map-class #:map-inverted-index #:get-instances-by-class #:get-instance-by-value #:get-instances-by-value --- /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/12 02:47:33 1.4 +++ /project/elephant/cvsroot/elephant/src/elephant/query.lisp 2007/04/27 13:32:17 1.5 @@ -73,8 +73,8 @@ (if (find-inverted-index class slot) (if (= (length values) 1) (progn - (map-class-index fn class slot (first values) (first values)) - (map-class-index fn class slot (first values) (second values)))) + (map-inverted-index fn class slot (first values) (first values)) + (map-inverted-index fn class slot (first values) (second values)))) (map-class #'filter-by-relation class)) (map-class-query #'filter-by-relation (cdr constraints)))))) --- /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/12 02:47:33 1.11 +++ /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/27 13:32:17 1.12 @@ -95,13 +95,18 @@ ;; form)) (defun transaction-object-p (txnrec) - (consp txnrec)) + (and (not (null txnrec)) + (consp txnrec) + (subtypep (type-of (car txnrec)) 'store-controller))) (defun owned-txn-p (sc parent-txn-rec) (and parent-txn-rec (transaction-object-p parent-txn-rec) (eq sc (transaction-store parent-txn-rec)))) +(define-condition transaction-retry-count-exceeded () + ((retry-count :initarg :count))) + (defmacro with-transaction ((&rest keyargs &key (store-controller '*store-controller*) (parent '*current-transaction*) @@ -126,7 +131,7 @@ (defmacro ensure-transaction ((&rest keyargs &key (store-controller '*store-controller*) - (transaction '*current-transaction*) + (parent '*current-transaction*) (retries 200) &allow-other-keys) &body body) @@ -139,7 +144,7 @@ (sc (gensym))) `(let ((,txn-fn (lambda () , at body)) (,sc ,store-controller)) - (if (owned-txn-p ,sc ,transaction) + (if (owned-txn-p ,sc ,parent) (funcall ,txn-fn) (funcall #'execute-transaction ,store-controller ,txn-fn From ieslick at common-lisp.net Sat Apr 28 02:29:47 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:29:47 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc/oldfiles Message-ID: <20070428022947.36FAF3E057@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc/oldfiles In directory clnet:/tmp/cvs-serv16495/oldfiles Log Message: Directory /project/elephant/cvsroot/elephant/doc/oldfiles added to the repository From ieslick at common-lisp.net Sat Apr 28 02:31:07 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:31:07 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070428023107.348554508E@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv16753 Modified Files: CREDITS README Removed Files: BEN-RETIRING-TODO.txt ChangeLog INSTALL NEWS NOTES TODO TUTORIAL UPGRADE-BDB Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/CREDITS 2006/11/11 06:27:37 1.8 +++ /project/elephant/cvsroot/elephant/CREDITS 2007/04/28 02:31:06 1.9 @@ -57,3 +57,16 @@ Marco Baringer provided a partial patch for 64-bit support. +Pierre Thierry for a *features* hack to turn off optimization as well as +many suggestions and dicussions. + +Henrik Hjelte for a number of bug fixes and for testing as well as +his wonderful postmodern data store! + +Edi Weisz for Lispwork and Win32 patches + +Mac Chan for his Hutchentoot blog tutorial + +And many others for their comments, testing and suggestions -- we're sorry +if we missed crediting you here. + --- /project/elephant/cvsroot/elephant/README 2006/11/11 06:27:37 1.6 +++ /project/elephant/cvsroot/elephant/README 2007/04/28 02:31:06 1.7 @@ -1,89 +1,58 @@ -============================================== -Elephant -- an object database for Common Lisp -============================================== +================================================================================= +Elephant -- An persistent metaobject protocol and object database for Common Lisp +================================================================================= ------------- -Introduction ------------- - -Elephant is an object database for Common Lisp. It supports -storing CLOS objects and most lisp primitives, and access to -BTrees. It uses Sleepycat / Berkeley DB, a -widely-distributed embedded database; many unix systems have -it installed by default. Sleepycat is server-less, ACID -compliant, transactional, process and thread safe, and fast -relative to relational databases; hopefully Elephant -inherits these properties. - -Goals: - -* Transparency: most Lisp values are easy to persist without -much effort or special syntax. Talk to the DB with Lisp -code, not SQL or another domain-specific language. No -additional server to run. - -* Safety: ACID, transactions. Concurrent with good -multi-user and -thread semantics / isolation, locking and -deadlock detection. - -* Simplicity: be a small library with few surprises for the -programmer. Lisp and Berkeley DB together are an excellent -substrate, try to use their features as much as possible. - -* Performance: leverage Sleepycat performance and -reliability. In addition to fast concurrent / transactional -modes, elephant will (eventually) offer an accellerated -single-user mode. - -Join the Elephant mailing lists to ask your questions and -receive updates. They're on the Elephant website +Detailed information on Elephant with an introduction, supported platforms, +latest news and documentation can be found at: http://www.common-lisp.net/project/elephant ------------------- -Supported Lisps ------------------- -CMUCL Linux (lightly tested) -SBCL Linux Mac (heavily tested) -Allegro Linux Mac cygwin? (heavily tested) -CLISP Linux Mac cygwin (lightly tested) -MCL Mac (lightly tested) - ----------------------------- Copyright, License + Warrenty ----------------------------- -See LICENSE. +See LICENSE + +------------------- +Supported Platforms +------------------- + +See the website: http://www.common-lisp.net/project/elephant/platforms.html ------------ -Installation +Latest News ------------ -See INSTALL +See the website: http://www.common-lisp.net/project/elephant/news.html ----------------------- -Tutorial and Reference ----------------------- +-------------- +Documentation +-------------- -HTML docs and texinfo sources can be found in the docs/ directory. +HTML and PDF documentation can be found on the website, or built via a +Makefile (and SBCL) in the /doc directory + +------------ +Installation +------------ + +See the documentation, Chapter 3 (Installation) ------------------------ Design and Development ------------------------ -See NOTES for some design internals. +See the documentation, Chapter 7 (Elephant Architecture) and +Chapter 8 (Data Store API) for some design internals. + +If you want to contribute, see the Trac website for current +feature release plans and other things that need to be done. -If you want to contribute, see TODO for current feature release plans -and other things that need to be done. +http://trac.common-lisp.net/elephant ------- Authors ------- -See CREDITS. - ----- -News ----- +See CREDITS and LICENSE -See NEWS and ChangeLog. \ No newline at end of file From ieslick at common-lisp.net Sat Apr 28 02:31:14 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:31:14 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070428023114.27E9350035@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv16753/doc Modified Files: copying.texinfo data-store-reference.texinfo elephant.texinfo installation.texinfo reference.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/30 14:34:34 1.4 +++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/04/28 02:31:07 1.5 @@ -8,29 +8,30 @@ @section Elephant Licensing - at b{Elephant}: a persistent metaprotocol and object-oriented database -for Common Lisp. +Elephant is a persistent metaprotocol and object-oriented database for +Common Lisp. Detailed information and distributions can be found at + at uref{http://www.common-lisp.net/project/elephant}. -Homepage: @uref{http://www.common-lisp.net/project/elephant} +The program is released under the following license: @quotation -Elephant users are granted the rights to distribute and use this software -as governed by the terms of the Lisp Lesser GNU Public License - at uref{http://opensource.franz.com/preamble.html}, also known as the LLGPL. +Elephant users are granted the rights to distribute and use this +software as governed by the terms of the Lisp Lesser GNU Public +License @uref{http://opensource.franz.com/preamble.html}, also known +as the LLGPL. @end quotation Copyrights include: @quotation -Copyright (c) 2004 by Andrew Blumberg and Ben Lee - -Copyright (c) 2005-2007 by Robert L. Read - -Copyright (c) 2006-2007 by Ian Eslick +Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. @* +Version 0.5, Copyright @copyright{} 2006 Robert L. Read. @* +Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* +Portions copyright respective contributors (see @file{CREDITS}). @end quotation -Portions of this program (namely the C unicode string sorter) are -derived from IBM's @b{ICU}: @uref{http://oss.software.ibm.com/icu/, +Portions of the program (namely the C unicode string sorter) are +derived from IBM's ICU: @uref{http://oss.software.ibm.com/icu/, ICU Website} whose copyright and license follows below. @quotation @@ -75,6 +76,33 @@ are the property of their respective owners. @end quotation + at section Elephant Manual Copyright and Licensing + + at quotation +Permission is granted to copy, distribute and/or modify this document +under the terms of the GNU Free Documentation License. + at end quotation + +Copyrights include: + + at quotation +Original Version, Copyright @copyright{} 2004 Ben Lee. @* +Versions 0.5-0.6, Copyright @copyright{} 2006 Robert L. Read. @* +Current Version, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* + at end quotation + + at section 3rd Party Libraries + +Elephant depends on 3rd party lisp libraries. See their respective +distributions for detailed copyright and licensing information. The +following is a brief summary. + + at itemize + at item @b{uffi}: By Kevin Rosenberg, no significant restrictions + at item @b{cl-base64}: By Kevin Rosenberg, no significant restrictions + at item @b{rt}: By Richard Waters, MIT License + at end itemize + @section Data Store Licensing Considerations The Berkeley DB data store is based on the Berkeley DB C library, now @@ -100,5 +128,3 @@ @item MySQL: Dual licensing (similar to BDB), see @uref{http://www.mysql.com/company/legal/licensing/, the MySQL license page} @end itemize - - --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/27 03:14:55 1.7 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 02:31:07 1.8 @@ -34,7 +34,7 @@ * Cursors: DSR Cursors. Traversing BTrees. * Transactions: DSR Transactions. Transaction implementation. * Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. -* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries + at c * Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu @node DSR Registration @@ -147,8 +147,8 @@ implemented using map-btree and map-index. @itemize - at item @ref{Generic-Function elephant:get-value} - at item @ref{Generic-Function (setf elephant:get-value)} + at item @ref{Generic-Function elephant:get-value} (and @code{(setf get-value)}) + at c @item @ref{Generic-Function (setf elephant:get-value)} @item @ref{Generic-Function elephant:existsp} @item @ref{Generic-Function elephant:remove-kv} @item @ref{Generic-Function elephant:get-index} @@ -168,7 +168,7 @@ @cindex Cursors Data stores must subclass these cursor classes and implement all the -methods described in @ref{Cursors} except @ref{Macro +methods described in @ref{DSR Cursors} except @ref{Macro elephant:with-btree-cursor}. @include includes/class-elephant-cursor.texinfo @@ -220,9 +220,23 @@ @section Multithreading Considerations @cindex Multithreading -Generic locking utility functions +This expands slightly on the multithreading discussion in + at ref{Multi-threaded Applications}. -Variable behavior in multithreading situations +Elephant provides a set of generic locking functions in + at code{src/utils/locks.lisp} to help protect any shared structures. +There are standard locking functions (@code{ele-with-lock}) and then +a special locking interface called @code{ele-with-fast-lock} which +on some lisps provides a faster locking option than the standard OS +locks of the basic interface. (i.e. under Allegro this uses + at code{without-interrupts} because Allegro still runs in a single +OS process on all platforms, this is not true of SBCL). + +See the sections on Transaction handling, particularly the dynamic +behavior of @code{*current-transaction*}. Also read up on the +store controller section in the User Guide to better understand the +role of @code{*store-controller*}. At this time there are no other +global variables to worry about. @node DSR Handling Serialization @comment node-name, next, previous, up @@ -253,19 +267,19 @@ @include includes/fun-elephant-data-store-serialize-to-base64-string.texinfo @include includes/fun-elephant-data-store-deserialize-from-base64-string.texinfo - at node DSR Memory Utilities - at comment node-name, next, previous, up - at section Memory utilities - at cindex Memory utilities - -Details about memory utilities here. - - at node DSR Foreign Libraries - at comment node-name, next, previous, up - at section Foreign libraries - at cindex Foreign libraries + at c @node DSR Memory Utilities + at c @comment node-name, next, previous, up + at c @section Memory utilities + at c @cindex Memory utilities + + at c Details about memory utilities here. + + at c @node DSR Foreign Libraries + at c @comment node-name, next, previous, up + at c @section Foreign libraries + at c @cindex Foreign libraries -How foreign libraries are built and used via UFFI. What functions are -in the .asd files or main lisp code to build & load libraries? + at c How foreign libraries are built and used via UFFI. What functions are + at c in the .asd files or main lisp code to build & load libraries? --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/27 03:14:55 1.10 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 02:31:07 1.11 @@ -2,19 +2,27 @@ @c %**start of header @setfilename elephant.info @settitle Elephant User Manual + at setchapternewpage odd @c %**end of header - - at copying -Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -Version 0.5, Copyright @copyright{} 2006 Robert L. Read. -Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read + at copying @quotation +Elephant System @* +Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. @* +Version 0.5, Copyright @copyright{} 2006 Robert L. Read. @* +Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* +Portions copyright respective contributors (see @file{CREDITS}). @* + +Elephant Manual @* +Original Version, Copyright @copyright{} 2004 Ben Lee. @* +Versions 0.5-0.6, Copyright @copyright{} 2006 Robert L. Read. @* +Current Version, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @* + Permission is granted to copy, distribute and/or modify this document -under the terms of the GNU Free Documentation License. Portions of -Elephant are derived from ICU, IBM's Unicode library. Details about -copyright, license and warranty information can be found in - at file{LICENSE} or in the ``Copying'' section of this document. +under the terms of the GNU Free Documentation License. See the +Copyright and License chapter for details about copyright, license and +warranty for this manual and the Elephant system. + @end quotation @end copying --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/27 03:14:55 1.11 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 02:31:07 1.12 @@ -9,9 +9,10 @@ * Requirements:: Supported lisps and required libraries. * Configuring Elephant:: Setting up Elephant and the configuration file. * Loading Elephant:: Loading Elephant and the data store loading protocol. -* Berkeley DB:: Installing support for the Berkeley DB data store +* Berkeley DB:: Installing support for the Berkeley DB data store. * Berkeley DB Example:: An example of installing and running the Berkeley DB data store. -* CL-SQL:: Install and connecting to the CL-SQL data store +* Upgrading Berkeley DB Database:: How to upgrade to a new version of Berkeley DB. +* CL-SQL:: Install and connecting to the CL-SQL data store. * CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows * Test Suites:: How to run and interpret the output of the regression test suite @@ -275,6 +276,57 @@ @end lisp in your application. + at node Upgrading Berkeley DB Databases + at comment node-name, next, previous, up + at section Upgrading Berkeley DB Databases + +At regular intervals, Elephant will require a specific version of Berkeley DB +because it does not parse the header files which tend to change with each release. +The patches are usually minor, but sometimes Elephant also depends on new +features of Berkeley DB. In this case, you have to update both the Berkeley DB +database files as well as any data Elephant has built within it. That creates +some special constraints for upgrading databases. + + at subsection Upgrading to 0.9 + +This section outlines how to upgrade from Elephant code base version 0.6.0 +using Berkeley DB 4.3. + + at enumerate + at item Install BDB 4.5 (keep 4.3 around for now) + at item Update my-config.sexp to point to the appropriate BDB 4.5 directories + at item Upgrade your database directory to 4.5 + @itemize + @item Run db43_recover in your 0.6 database + @item Optional: run db43_archive -d to remove all logs not part of a checkpoint + This will make catastrophic recovery impossible, but reduces the amount of data you + have to backup. + @item Backup your db files and remaining logs + @item Run db45_checkpoint -1 in your 0.6 database directory + @end itemize + at item Migrate 0.6 data to a new 0.6.1 database + @itemize + @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} + @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} + @end itemize + at item Test your new application and report any bugs that arise to elephant-devel at common-lisp.net + at end itemize + + at emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} + + at emph{(NOTE: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit + version of your lisp to update to 0.6.1 and then open that database in your 64-bit + lisp. There should be no compatibility problems. Best to test your application on + a 32-bit lisp if you can, just to be sure.)} + + at subsection Upgrade from Elephant 0.5 + +Follow the upgrade procedures outlined in Elephant release 0.6.0 to migrate your database +from 0.5 to 0.6. Then follow the above procedures for upgrading from an 0.6 database. + + at emph{(NOTE: It may not take much work to make 0.9 upgrade directly from 0.5 +However there are so few (none?) 0.5 users that it wasn't deemed worth +the work given that there's an upgrade path available.)} @node CL-SQL @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/27 03:14:55 1.13 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 02:31:07 1.14 @@ -140,13 +140,15 @@ Persistent collections inherit from @ref{Class elephant:persistent-collection} and consist of the @ref{Class elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class elephant:btree-index} classes. The following operations are defined -on most of these classes. More information can be found in @ref{Using BTrees} -and @ref{Secondary Indices}. +on most of these classes. More information can be found in @ref{Persistent BTrees} +and @ref{BTree Indexing}. @include includes/fun-elephant-make-btree.texinfo - @include includes/fun-elephant-get-value.texinfo - at include includes/fun-elephant-setf-get-value.texinfo + +Values are written to a btree using the @code{setf} method on @code{get-value}. + at c @include includes/fun-elephant-setf-get-value.texinfo + @include includes/fun-elephant-remove-kv.texinfo @include includes/fun-elephant-existsp.texinfo @include includes/fun-elephant-drop-btree.texinfo @@ -178,7 +180,8 @@ @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo @include includes/fun-elephant-cursor-duplicate.texinfo - at include includes/fun-elephant-cursor-initialized-p.texinfo + at c fixme, not generated by docstrings + at c @include includes/fun-elephant-cursor-initialized-p.texinfo Each of the following methods return multiple values consisting of @code{(exists? key value)}. @@ -197,8 +200,8 @@ @node Index Cursor API @comment node-name, next, previous, up - at section Cursor API - at cindex Cursors + at section Index Cursor API + at cindex Cursor @cindex Index @cindex Indices @cindex API @@ -224,12 +227,15 @@ @include includes/fun-elephant-cursor-pget-both.texinfo @include includes/fun-elephant-cursor-pget-both-range.texinfo - at include includes/fun-elephant-cursor-next-dup.texinfo @include includes/fun-elephant-cursor-next-nodup.texinfo - at include includes/fun-elephant-cursor-pnext-dup.texinfo + at include includes/fun-elephant-cursor-next-dup.texinfo @include includes/fun-elephant-cursor-pnext-nodup.texinfo + at include includes/fun-elephant-cursor-pnext-dup.texinfo + @include includes/fun-elephant-cursor-prev-nodup.texinfo + at include includes/fun-elephant-cursor-prev-dup.texinfo @include includes/fun-elephant-cursor-pprev-nodup.texinfo + at include includes/fun-elephant-cursor-pprev-dup.texinfo @node Transaction API @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/27 03:14:55 1.9 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/28 02:31:07 1.10 @@ -711,14 +711,15 @@ hash table indirections with a little macro: @lisp -(defmacro def-snapshot-wrapper - (accessor-name (source-classname target-classname hashname uid)) +(defmacro def-snapshot-wrapper (accessor-name + (source-classname target-classname hashname uid)) (with-gensysms (obj key ref) `(progn (defmethod ,accessorname :around ((,obj ,source-classname)) (let ((,key (call-next-method))) (when ,key (gethash ,key ,hashname)))) - (defmethod (setf ,accessorname) :around (,ref (,obj ,source-classname)) + (defmethod (setf ,accessorname) :around + (,ref (,obj ,source-classname)) (cond ((subtypep (type-of ,ref) ,target-classname) (let ((,key (find-object ,ref ,hashname))) (if ,key --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/27 03:14:55 1.19 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/28 02:31:11 1.20 @@ -272,7 +272,7 @@ @item @strong{Merge-conflicts in heavily multi-process/threaded situations}. This is the common read-modify-write problem in all databases. We will talk -more about this in the @ref{Transactions} section. +more about this in the @ref{Using Transactions} section. @end enumerate @@ -761,7 +761,9 @@ (get-instances-by-range 'friends 'name "Adam" "Devin") => (# #) -(get-instances-by-range 'friend 'birthday (encode-date '(1 1 1974)) (encode-date '(31 12 1984))) +(get-instances-by-range 'friend 'birthday + (encode-date '(1 1 1974)) + (encode-date '(31 12 1984))) => (# #) (mapc #'print-friend *) @@ -1207,12 +1209,12 @@ and iterating of duplicate or unique values. @item @strong{Using the Map Operators} Mapping operators can be very efficient if properly utilized. - at item @strong{Using Multiple Stores} Multiple store controllers can -be open simultaneously. However it does make the code more complex -and you need to be careful about how you use them to avoid crashes -and other unpleasant side effects. - at item @strong{Custom Transaction Architecture} You can implement your -own version of @code{with-transaction} using the + at item @strong{Using Multiple Stores} +Multiple store controllers can be open simultaneously. However it +does make the code more complex and you need to be careful about how +you use them to avoid crashes and other unpleasant side effects. + at item @strong{Custom Transaction Architecture} +You can implement your own version of @code{with-transaction} using the underlying controller methods for starting, aborting and committing transactions. You had better know what you are doing, however! @item @strong{Handling Errors and Conditions} --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/27 03:14:55 1.19 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 02:31:11 1.20 @@ -188,7 +188,7 @@ @item @strong{Merge-conflicts in heavily multi-process/threaded situations}. This is the common read-modify-write problem in all databases. We will talk -more about this in the @ref{Transactions} section. +more about this in the @ref{Transaction Details} section. @item @strong{Byte Ordering}. The primitive elements such as integers are written to disk in @@ -908,7 +908,7 @@ Once a cursor is properly initialized, it can be incremented or decremented, a simple constant-time operation on BTrees. - at code{cursor-next} and @{cursor-prev} move the cursor a single step + at code{cursor-next} and @code{cursor-prev} move the cursor a single step forward or back across the sorted key-value pairs. @code{cursor-next} moves in ascending order, @code{cursor-prev} in descending order. @@ -960,10 +960,11 @@ class indexing capabilities previously described. An index is created by using the @code{add-index} function. This -function takes the @code{indexed-btree} you wish to index, an -index-name for later retrieval, a key-form which dictates how -the index populates it's keys as a function of the main btree's -keys and values. +function takes the @code{indexed-btree} you wish to index, an symbolic +name for the index and a key-form which dictates how the index +populates it's keys as a function of the main btree's keys and values. +(It is a function of three arguments: the index itself, the key and +the value). A simple, contrived example is shown in the figure below: @@ -1018,23 +1019,37 @@ Operations that have the same behavior, but return primary btree values and keys are: - at table @samp + at multitable @columnfractions .3 .1 .35 + at headitem BTree Cursor Function @tab @tab Index Cursor Function + @item @code{cursor-first} - at code{cursor-pfirst} - at samp{@code{cursor-first}} + at tab => + at tab @code{cursor-pfirst} + @item @code{cursor-last } - at code{cursor-plast} + at tab => + at tab @code{cursor-plast} + @item @code{cursor-current } - at code{cursor-pcurrent} + at tab => + at tab @code{cursor-pcurrent} + @item @code{cursor-next } - at code{cursor-pnext} + at tab => + at tab @code{cursor-pnext} + @item @code{cursor-prev} - at code{cursor-pprev} + at tab => + at tab @code{cursor-pprev} + @item @code{cursor-set} - at code{cursor-pset} + at tab => + at tab @code{cursor-pset} + @item @code{cursor-set-range} - at code{cursor-set-prange} - at end table + at tab => + at tab @code{cursor-set-prange} + at end multitable The big difference between btree cursors and index cursors is that indices can have duplicate key values. This means we have to choose @@ -1042,100 +1057,217 @@ a duplicate segment. There are cursor operations for each: @itemize - at item Simple move. Standard btree operations plus @code{cursor-pnext} and @code{cursor-pprev}. - at item Move to next key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup}. - at item Move to next duplicate. + at item Simple move. Standard btree operations work plus @code{cursor-pnext} and @code{cursor-pprev} + at item Move to a different key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup} + at item Move to next duplicate key value. @code{cursor-pnext-dup} and @code{cursor-pprev-dup} @end itemize - at c FINISH +After incrementing through a set of duplicate items using a + at code{xxx-dup} function, the last next operation returns nil +indicating there are no more duplicates. The consequence of this is +that the cursor is now uninitialized (@code{cursor-initialized-p}) and +needs to be reset by a set or set both call. + + @node Multi-threaded Applications @comment node-name, next, previous, up @section Multi-threaded Applications -Berkeley DB plays well with threads and processes. The store -controller is thread-safe by default, that is, can be shared amongst -threads. This is enabled by the @code{:thread} keyword argument which -defaults to true. Transactions may not be shared amongst threads -except serially. One thing which is NOT thread and process safe is -recovery, which should be run when no one is else is talking to the -database environment. - -The following shared regions of Elephant are protected by standard locks: -- buffer stream pool -- access to serializer circular buffers -- writes to connection db -- where else? +Elephant is thread-safe by design. Users should not have to think +about threading except to follow a couple of simple rules. + + at enumerate + at item Do not perform transactions across multiple threads + at item Do not perform add/remove index operations on indexed-btrees +in more than one thread. + at end enumerate + +This and common coding sense should be sufficient! Elephant's +internal design for thread safety employs a number of policies +to try to minimize using lisp locks and simplify analysis of +multi threaded interactions: + + at enumerate + at item @strong{Rely on the thread safety of the data store databases} + at item @strong{Ensure transaction isolation} + at item @strong{Minimize dependency on thread-local special variables} + at item @strong{Protect shared resources for a given store controller}. + at item @strong{A use policy for shared objects (above)} + at end enumerate + + at subsection Shared Resources + +Elephant has a few shared resources which are protected by standard locks. +These are: + at itemize + at item The store controller connection table + at item The instance cache + at item The circularity buffer pool for the serializer + at item The buffer-stream pool in memutils + at end itemize + +In some cases, and on some lisp platforms, we try to use a fast lock +strategy for frequently accessed items (the resource pools and +instance cache especially). + + at subsection Data Store Thread Safety and Transactions + +Both CLSQL and Berkelely DB backends are thread safe. In CLSQL this +is by ensuring that every thread has it's own handle into the SQL +libraries or sockets. Berkeley DB is reentrant and handles locking +internally. + +Elephant depends on these guarantees especially for the isolation +properties of transactions. All operations in the context of a +given transaction should be isolated and atomic. It is important +that a transaction not be shared across threads, however. + + at subsection Minimize Dependency on Thread-Local Specials + +Elephant uses several global variables as default arguments. Most +of these were removed leaving only a couple to handle: + + at itemize + at item @strong{@code{*store-controller*}}. Store controller objects can +be shared between threads and if a user resets this variable in a local +thread to another controller, there is no problem with that either. +Users of multiple concurrent stores can specify the store controller to +all elephant API commands that don't get it from a persistent object +implicitely. + at code{*current-transaction*}. This is always set to the proper null +value globally and should not be reset in local threads. Instead, +transactions take place in a dynamic context that rebinds this variable +as a special with the current transaction. This allows for a dynamic +transaction stack for data stores that can nest transactions or when +two datastores are both doing transactions concurrently. + at end itemize - at c *** FINISH *** @node Transaction Details @comment node-name, next, previous, up @section Transaction Details +Transactions are dynamic contexts in which all side effects to +persistent slots and other persistent objects such as BTrees are +guaranteed to have the ACID properties: atomicity, consistency, +isolation and durability. On a normal exit from context, the +side effects are committed as a group. On a non-local exit, +the transaction is aborted. + +For most users, the tutorial section @ref{Using Transactions} is the +best introduction to transactions. This section adds to that by +exposing some of the details of how it is implemented. + +To reiterate, there are a few important restrictions to adhere to: + + at itemize + at item @code{*current-transaction*} is reserved for use by the transaction system. Users should not override, manipulate or close over this variable. + at item The body of a transaction cannot throw, signal or jump without aborting the transaction. Any non-local exit is considered an aborting event. Catch signals inside the transaction and return a value instead. + at item The dynamic extent of a transaction body must stay within the same thread + at end itemize + + at subsection @code{with-transaction} internals + +The @code{with-transaction} macro wraps the body expression with an +anonymous lambda expression. This closure is passed to a call to the + at code{execute-transaction} generic function which is specialized to +the current data store. + +The only bookkeeping done by the macro is ensuring that the + at code{:parent} argument is checked for the current dynamic transaction +context. If it is not owned by the default or provided store +controller, then it is not passed to @code{execute-transaction}. This +maintains a continuous dynamic stack transactions through the +with/ensure transaction macros, but allows for a single leaf +transaction to another store controller. + +Be very careful about mixing transactions between store controller. +This facility was only added to ensure that migrate worked correctly. + +The macro processes keywords arguments @code{:store-controller} +(defaults to @code{*store-controller*}), @code{:parent} (defaults to + at code{*current-transaction*}) and @code{:retries} and passes the +remaining keywords to the call to @code{execute-transaction} allowing +the user to pass data store specific transaction keywords to their +preferred data store. The consumed keywords are analyzed and then +passed on to @code{execute-transaction}. + +Any non-standard keywords for a given data store will be ignored by +other data store implementation of @code{execute-transaction} so +portable programs should not use keywords that change the semantics of +the transaction. + + at code{ensure-transaction} only calls @code{execute-transaction} if +it needs to create a fresh transaction. If the transaction in + at code{*current-transaction*} exists and belongs to the store controller +passed to @code{ensure-transaction} then it merely calls the transaction +closure, relying on the environment that created the transaction to +handle any exit procedures and determining whether to abort or commit. + + at code{*current-transaction*} contains transaction records during the +dynamic execution of a transaction. These records capture any data +store specific bookkeeping as well as the store-controller that the +transaction is associated with. + + at subsection @code{execute-transaction} internals + +See the @ref{Elephant Architecture} section for details on how +execute-transaction works. It will provide some deeper insight +into the transaction system. + + at subsection Building your own transactional framework + +Data stores are required to implement three primitive transaction +methods: @code{controller-start-transaction}, + at code{controller-abort-transaction} and + at code{controller-commit-transaction}. These are wrappers for the data +store's primitive transaction mechanism. If you use these, it is up +to you to make sure that you properly manage nested transactions, +maintain the state of @code{*current-transaction*} handle any +automated retries you might want, and handle detecting + +If you use these, you are on your own - it is easy to make mistakes with +transactions and create very complex bugs that are hard to track down. +Most users are much better off sticking with the two transaction macros +and the underlying @code{execute-transaction} method. + + at subsection Analyzing Dynamic Transaction Behavior + You can trace @code{elephant::execute-transaction} to see the sequence -of calls to @code{execute-transaction} that occur dynamically and -detect where transactions are and are not happening. We may add some -transaction diagnosis and tracing tools in the future, such as -throwing a condition when @code{with-transaction} forms are nested -dynamically. - -;; Transaction architecture: -;; -;; User and designer considerations: -;; - *current-transaction* is reserved for use by dynamic transaction context. The default global -;; value must always be null (no transaction). Each data storeend can set it to a different parameter -;; within the dynamic context of an execute-transaction. -;; - Any closures returned from within a transaction cannot bind *current-transaction* -;; - Only a normal return value will result in the transaction being committed, any non-local exit -;; results in a transaction abort. If you want to do something more sophisticated, roll your own -;; using controller-start-transaction, etc. -;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc) -;; knowing that the transaction will be aborted -;; +of calls that occur dynamically and detect where and how many +transactions are and are not happening. + + at c IT WOULD BE GOOD TO EXPAND ON THIS REGARDING HOW TO SOLVE COMMON PROBLEMS - at c *** FINISH *** @node Multi-repository Operation @comment node-name, next, previous, up @section Multi-repository Operation -Elephant now keeps a small hashtables that maps ``database specifications'' into -actual database connections. +Elephant maintains a small hashtable that maps ``database +specifications'' into actual @code{store-controller} objects. + +The basic strategy is that the ``database specification'' object is +stored in every persistent object and collection so that the +repository can be found. In this way, objects that reside in +different repositories can coexist within the LISP object space, +allowing data migration or multiple user stores. + +All persistent instances store their oid and a store-controller +reference in internal slots. Slot access and other protocols use +this to provide access. This executes an auto-transaction or joins +a surrounding transaction if the @code{transaction-record} in + at code{*current-transaction*} matches the store. + +When operating with multiple stores and nested transactions there are +some subtle issues to work around: how to avoid writing one store with +a transaction created in the context of another. A nested or ensured +transaction is only indicated in the call to + at code{execute-transaction} if the store controllers match, otherwise a +new transaction for that store is created. + + at c A PICTURE OF THE DYNAMIC CONTEXT WOULD BE USEFUL HERE -If a database spec is a string, it is assumed to be a BerkeleyDB path. -If it is a list, it is a assumed to be a CLSQL connection specification. -For example: - at lisp -ELE-TESTS> *testdb-path* -"/home/read/projects/elephant/elephant/tests/testdb/" -ELE-TESTS> *testpg-path* -(:postgresql "localhost.localdomain" "test" "postgres" "") -ELE-TESTS> - at end lisp -The tests now have a function @code{do-all-tests-spec} that take a spec and -based on its type attempt to open the correct kind of store controller and -perform the tests. - -The routine @code{get-controller} takes this specifiation. - -The basic strategy is that the ``database specification'' object is stored in -every persistent object and collection so that the repository can be found. - -In this way, objects that reside in different repositories can coexist within -the LISP object space, allowing data migration. - -;; Multiple stores - -;; Multiple store considerations: -;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to -;; work around: how to avoid writing one store with a transaction created in the context of another. -;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct; -;; this requirement may relax in the future -;; - The following macros accomodate multiple stores by requiring that execute-transaction return a -;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller -;; is the store instance it is associated with. A nested or ensured transaction is only indicated -;; in the call to execute transaction if the store controllers match, otherwise a new transaction -;; for that store is created @node Multiple Processes and Distributed Applications @comment node-name, next, previous, up From ieslick at common-lisp.net Sat Apr 28 02:31:15 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:31:15 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc/oldfiles Message-ID: <20070428023115.2CF74671CA@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc/oldfiles In directory clnet:/tmp/cvs-serv16753/doc/oldfiles Added Files: INSTALL NEWS NOTES TODO TUTORIAL Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/doc/oldfiles/INSTALL 2007/04/28 02:31:14 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/INSTALL 2007/04/28 02:31:14 1.1 ------------ Requirements ------------ Supported Lisps: CMUCL 19a Linux SBCL 0.9.17/1.0+ Linux / Mac OSX Allegro CL 7.0/8.0 Linux / Mac OSX OpenMCL 0.14.2 LispWorks (port in-progress) Lisp libraries: ASDF - http://www.cliki.net/asdf UFFI 1.5.4+ - http://uffi.b9.com/ A Backend Database: 1) Oracle Berkeley DB 4.5 - http://www.oracle.com/database/berkeley-db.html 2) CLSQL - http://clsql.b9.com/ with an appropriate SQL installation. Tested with SQlite3 and Postgresql so far A C compiler, probably gcc or Visual Studio. Presumably you have this if you installed ------------------ Short Instructions ------------------ The new build system should work out of the box on most Un*x platforms that have asdf, uffi and either clsql or Berkeley DB installed in the usual places. Try: (asdf:operate 'asdf:load-op :elephant) Then: (open-store '( )) Where = { :bdb | :clsql } = { "fresh directory for BDB files" | '(:sqlite3 "db path") | '(:postgresql "db path") This should load all files, including compiling libraries, on most systems. For Win32, see the instructions below. (We'll improve the build process for Win32 if there is demand) ----------------- Long Instructions ----------------- For SBCL, CMUCL, Allegro 8.0+, MCL and CLISP: 0) Unpack Elephant. I put mine in the directory /usr/local/share/common-lisp/elephant-0.6.x/ 1) Install ASDF. Ensure that you have a recent version of ASDF installed as the load process now depends upon it. 2) Install UFFI 3) Install a backend: Either Berkeley DB 4.5, PostGresql, or SQLite 3. ------- SQL ------- For relational database systems, refering the formal documentation other the heading "SQL-BACK-END". ------------- Berkeley 4.5: ------------- (Note: 0.6.0 required BDB 4.3; to upgrade 0.6.0 to 0.6.1, upgrade BDB to 4.5, modify my-config.sexp appropriately then run 0.6.1+; your underlying Berekely DB files will automatically upgrade when the DB is opened. To use 0.6.1, you will have to manually migrate your 0.6.0 database to a fresh database created in 0.6.1) Under Un*x, you may actually already have this installed, though it may be compiled with funny options, so if things don't work you may want to try to start from scratch. FreeBSD has a port for this, as I'm sure do other BSDs (including DarwinPorts/Fink.) Take note of where libdb.so and db.h are installed, usually: /usr/local/BerkeleyDB.4.5/lib/libdb.so and /usr/local/BerkeleyDB.4.5/include/db.h, or /usr/local/lib/db45/libdb.so and /usr/local/include/db45/db.h.) a) Site specific configuration config.sexp Which contains an alist providing string paths pointing to the root of the Berkeley DB distribution :berkeley-db-root, the library to load :berkeley-db-lib and the pthreads library if you're running linux :pthread-lib. For Win32 (directions courtesy of Bill Clementson): --------------------------------------------------- Create an MSVC dll project and add src/db-bdb/libberkeley-db.c, src/db-bdb/libberkeley-db.def and the Berkeley DB libdb43.lib files to the project (should be in the build_win32/release folder) Add the Berkeley DB dbinc include files directory and the build_win32/release directory (where the Berkeley DB install instructions builds the Berkeley DB objects by default) to the build directories for the project Build the Elephant DLL file Since you've statically included libdb43.lib inside libberkeley-db.c, it may or may not be necessary to load libdb43.dll into Lisp (see below.) 4) Compile and load Elephant: The new backend load process should work automatically on Un*x systems but if there are probolems with loading foreign libraries, then you can test your C tools setup with 'make' in the elephant root directory. This will build the common memutils library in src/memutil/libmemutil.so/dylib that all backends require. There is a new two-phase load process. The first requires that you use asdf to load the main elephant front-end: (asdf:operate 'asdf:load-op :elephant) This will load and compile Elephant. This will also automatically load UFFI. When you call (open-store ) inside lisp it will automatically load the remaining dependencies for the specified backend via ASDF. To test the load process explicitly the following asdf files are provided: if you are using Berkeley DB, type: (asdf:operate 'asdf:load-op :ele-bdb) if you are using CL-SQL, type: (asdf:operate 'asdf:load-op :ele-clsql) if you are using SQLite3, type: (asdf:operate 'asdf:load-op :ele-sqlite3) 5) Make the documentation: Execute: make In the doc directory should be build the HTML version of the texinfo files. ------- Testing ------- Elephant uses RT for regression testing, available at: http://www.cliki.net/RT Once RT is installed (asdf:operate 'asdf:load-op :elephant-tests) (in-package :ele-tests) (setf *default-spec* ) Where = { *testsqlite3-spec* | *testpg-spec* | *testbdb-spec* } (do-backend-tests) This will test the standalone API for your backend. Currently all tests are passing on 0.6.0. There will be a set of migration tests that will be 'ignored' but the final message should indicate no failing tests. This should take less than 5 minutes on decent hardware. The tests are not idempotent, so if you run the tests a second time, they are likely to fail. To avoid this, for example if you are debugging tests, just run the script delscript.sh (or do the equivalent on Win32) in the elephant/tests directory. Elephant allows migration between repositories. To test this: (do-migration-tests *default-spec* ) where is a different *testXXXXX-spec* variable to test migration to that backend. This should take less than 2 minutes on decent hardware. A backend is considered "green" if it can pass both the backend tests and the migration tests. --- /project/elephant/cvsroot/elephant/doc/oldfiles/NEWS 2007/04/28 02:31:14 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/NEWS 2007/04/28 02:31:14 1.1 April, 2006 - Elephant 0.6.0 released by Robert Read and Ian Eslick. Supports class slot indexing and benefits from a clean refactoring of backends and a host of other small changes. This is a solid BETA release. November 30, 2005 - Elephant 0.3.0 released by the new maintainer, Robert L. Read, providing support for relational database backends, repository migration, and multi-repository operation. As of this release, the documentation provides a lot of information about installation and getting things working; I wouldn't at all claim that it is complete, smooth, or well organized. The more notes I get about the use of Elephant, the more inclined I will be to invest time in improving the documentation. October 7, 2004 - Elephant 0.2.1 released. Thanks to Bill Clementson, Elephant should compile on Win32 now. Also, a few minor fixups. September 19, 2004 - Elephant 0.2 released. This is an BETA release. New features: - Secondary indices and cursors - PPC Darwin OpenMCL / SBCL - Doc strings and improved documentation - An RT-based test suite - many bugfixes This release has been tested on CMUCL 19a, SBCL 0.8.14 and Allegro 6.2 on x86 Linux and FreeBSD, and OpenMCL 0.14.2-p1 and SBCL 0.8.14 on PPC Darwin. September 2, 2004 - The bad news: there was a bug in 0.1 which made OID generation inside of manual transactions deadlock. The good news: this is fixed, and I've added OpenMCL support. So I'm releasing 0.1-p1. August 30, 2004 - Elephant 0.1 was released August 30th, 2004. This is an ALPHA quality release, so claims about correctness, performance and safety should be taken with a grain of salt. This release has been tested on CMUCL 19a, SBCL 0.8.13 and Allegro 6.2 on x86 Linux and FreeBSD. OpenMCL and Lispworks versions will come soon. As a proof of concept I've compiled and run CL-IRC http:://www.common-lisp.net/project/cl-irc making all objects and slots persistent, except for the socket-streams. It runs, and saves everything except for the socket-streams. --- /project/elephant/cvsroot/elephant/doc/oldfiles/NOTES 2007/04/28 02:31:15 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/NOTES 2007/04/28 02:31:15 1.1 ------- GENERAL ------- this has been optimized for use with CMUCL / SBCL / Allegro. OpenMCL has been minimally supported. Lispworks is a target as well but less so as the developers don't have access to it. Theoretically one can port this to any lisp with a decent FFI and MOP. However since those are two of the less standardized bits of Lisp, in practice this might be difficult. >From top to bottom, here are the implementation layers: ELEPHANT package persistent meta-object, persistent collections controller serializer memutils package UFFI / implementation specific stuff libsleepycat.so Sleepycat 4.2/3 While I loath specials, since you can't change the signature of slot accessors, in order to pass parameters to the database / serializer, specials are needed. Also specials will probably play nice with threaded lisps. ----------------------- CLASSES AND METACLASSES ----------------------- Persistent classes which the user defines are declared and instrumented by using the persistent-metaclass. Ideally creating persistent versions of class, slot-defintion, et al would be enough, but in reality various implementations do things in different ways. CMUCL / SBCL: their's a bit of work to make class slot allocation and reader / writer / slot-boundp work right. Allegro: is using slot-boundp instead of slot-boundp-using-class inside of shared-initialize, which necessitates some work. CMUCL doesn't do non-standard allocation types correctly, so we've created our own slot definition keyword :transient. In the future this will change. Andrew will add some notes here in the future. ----------- COLLECTIONS ----------- While we support serializing and persisting a wide class of Lisp data types, there are problems with persisting aggregate types (conses, lists, arrays, objects, hash-tables...) 1) not automatic: there's no way for elephant to know when you've changed a value in an aggregate object, so you have to manually restore it back into the slot to get it saved. example 1: you put a cons into the database. you change it's car. this is not saved unless you resave the cons into the database. example 2: slot-1 of obj A (saved in the database) contains a cons. you change the car of the cons. this is not reflected into the database unless you resave A. 2) merge-conflicts: changing one value and saving an aggregate will write out the whole aggregate, possibly blowing away changes other threads have made behind your back. this is not protected by transactions! 3) consing, non-lazy and expensive (de)serialization: you have to serialize/deserialize the entire aggregate every time you save it or restore it. This is pretty fast all things considered, but it's probably better to use persistent collections. 4) you have to store the entire collection in memory, whereas one of the points of the database to store large collections of objects..... For these and other reasons, we provide a hash-table-like interface to Berkeley BTrees. These have many advantages over ordinary hash-tables from the point of view of persistence. There is a separate table for BTrees. This is because we use a hand coded C function for sorting, which understands a little of the serialized data. It can handle numbers (up to 64-bit bignums -- they are approximated by floats) and strings (case-insensitive for 8-bit, code-point-order for 16-bit Unicode.) It should be fast but we don't want a performance penalty on objects. Secondary indices are mostly handled on the lisp side, because of our weird table layout (see below) and to avoid crossing FFI boundaries. Some unscientific microbenchmarks indicated that there was no performance benefit on CMUCL / SBCL, and only minor benefit (asymptotically nil) on OpenMCL. They have a separate table. Actually two handles are opened on this table: one which is plain, and one which is associated to the primary btree table by a no-op indexing function. Since we maintain the secondary keys ourselves, the associated handle is good for gets / cursor traversals. We use the unassociated handle for updates. ---------- CONTROLLER ---------- The controller is accessed through the special *store-controller*. The controller keeps track of 1) the environment handle 2) the DB handle(s) 3) the instance cache 4) the root object The environment handle and DB handle currently aren't really exposed. Eventually they should be, so that tuning flags can be set on them. OIDs are generated by a bit of C code, which isn't great, nor that safe (to get acceptable performance i use DB_TXN_NOSYNC.) Waiting for Sleepycat 4.3. The instance cache is implemented as a values-weak hash-table. This is a hash-table where the values can be collected, and when they are, the entire key-value entry is deleted. There are implementations of this on the various platforms. The instance cache is there to make deserialization of persistant objects faster. Since we [249 lines skipped] --- /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:15 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:15 1.1 [498 lines skipped] --- /project/elephant/cvsroot/elephant/doc/oldfiles/TUTORIAL 2007/04/28 02:31:15 NONE +++ /project/elephant/cvsroot/elephant/doc/oldfiles/TUTORIAL 2007/04/28 02:31:15 1.1 [923 lines skipped] From ieslick at common-lisp.net Sat Apr 28 02:31:15 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:31:15 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-bdb Message-ID: <20070428023115.D3EB17113B@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-bdb In directory clnet:/tmp/cvs-serv16753/src/db-bdb Modified Files: bdb-transactions.lisp Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/27 13:32:14 1.15 +++ /project/elephant/cvsroot/elephant/src/db-bdb/bdb-transactions.lisp 2007/04/28 02:31:15 1.16 @@ -31,15 +31,13 @@ for count fixnum from 1 to retries for success of-type boolean = nil do - (let ((txn - (if transaction transaction - (db-transaction-begin env + (let ((txn (db-transaction-begin env :parent (if parent parent +NULL-VOID+) :degree-2 degree-2 :read-uncommitted read-uncommitted :txn-nosync txn-nosync :txn-nowait txn-nowait - :txn-sync txn-sync)))) + :txn-sync txn-sync))) (declare (type pointer-void txn)) (let (result) (let ((*current-transaction* (make-transaction-record sc txn)) From ieslick at common-lisp.net Sat Apr 28 02:31:31 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:31:31 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070428023131.AB5707208D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv16753/src/elephant Modified Files: classindex.lisp collections.lisp controller.lisp package.lisp transactions.lisp Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/24 12:58:10 1.39 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 02:31:15 1.40 @@ -84,11 +84,23 @@ (let ((class (find-class class-name nil))) (when class (indexed class)))) +(define-condition persistent-class-not-indexed (error) + ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj))) + +(defun signal-class-not-indexed (class) + (cerror "Ignore and continue?" + 'persistent-class-not-indexed + :format-control "Class ~A is not enabled for indexing" + :format-arguments (list (class-name class)) + :class class)) + +;; (define-condition + (defmethod find-class-index ((class persistent-metaclass) &key (sc *store-controller*) (errorp t)) (ensure-finalized class) (if (not (indexed class)) (when errorp - (error "Class ~A is not an indexed class" class)) + (signal-class-not-indexed class)) (if (class-index-cached? class) (%index-cache class) ;; we've got a cached reference, just return it (multiple-value-bind (btree found) @@ -110,31 +122,26 @@ (synchronize-class-to-store class :sc sc :method method) btree)) -(define-condition persistent-class-not-indexed (error) - ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj))) - (defun cache-new-class-index (class sc) "If not cached or persistent then this is a new class, make the new index" (if (indexed class) (enable-class-indexing class (indexing-record-slots (indexed-record class)) :sc sc) - (signal 'persistent-class-not-indexed - :class class - :format-control "Class ~A is not enabled for indexing" - :format-arguments (list (class-name class))))) + (signal-class-not-indexed class))) (defmethod find-inverted-index ((class symbol) slot &key (null-on-fail nil)) (find-inverted-index (find-class class) slot :null-on-fail null-on-fail)) (defmethod find-inverted-index ((class persistent-metaclass) slot &key (null-on-fail nil)) - (let* ((cidx (find-class-index class)) + (let* ((cidx (find-class-index class :errorp (not null-on-fail))) (idx (or (get-index cidx slot) (get-index cidx (make-derived-name slot))))) (if idx idx (if null-on-fail nil - (error "Inverted index ~A not found for class ~A with - persistent slots: ~A" slot (class-name class) (car (%persistent-slots class))))))) + (cerror "Ignore and continue?" + "Inverted index ~A not found for class ~A with persistent slots: ~A" + slot (class-name class) (car (%persistent-slots class))))))) (defmethod find-inverted-index-names ((class persistent-metaclass)) (let ((names nil)) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/27 13:32:17 1.27 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 02:31:15 1.28 @@ -312,6 +312,7 @@ primary key greater or equal to the pkey argument. Returns has-tuple / secondary key / value / primary key.")) + (defgeneric cursor-next-dup (cursor) (:documentation "Move to the next duplicate element (with the same key.) @@ -322,11 +323,6 @@ "Move to the next non-duplicate element (with different key.) Returns has-pair key value.")) -(defgeneric cursor-prev-nodup (cursor) - (:documentation - "Move to the previous non-duplicate element (with -different key.) Returns has-pair key value.")) - (defgeneric cursor-pnext-dup (cursor) (:documentation "Move to the next duplicate element (with the same key.) @@ -338,12 +334,53 @@ key.) Returns has-tuple / secondary key / value / primary key.")) + +(defgeneric cursor-prev-dup (cursor) + (:documentation + "Move to the previous duplicate element (with the same key.) +Returns has-pair key value.")) + +(defmethod cursor-prev-dup ((cur cursor)) + "Default implementation. Plan is to update both backends when BDB 4.6 comes out" + (when (cursor-initialized-p cur) + (multiple-value-bind (exists? skey-cur) + (cursor-current cur) + (declare (ignore exists?)) + (multiple-value-bind (exists? skey value) + (cursor-prev cur) + (if (lisp-compare-equal skey-cur skey) + (values exists? skey value) + (setf (cursor-initialized-p cur) nil)))))) + +(defgeneric cursor-prev-nodup (cursor) + (:documentation + "Move to the previous non-duplicate element (with +different key.) Returns has-pair key value.")) + +(defgeneric cursor-pprev-dup (cursor) + (:documentation + "Move to the previous duplicate element (with the same key.) +Returns has-tuple / secondary key / value / primary key.")) + +(defmethod cursor-pprev-dup ((cur cursor)) + "Default implementation. Plan is to update both backends when BDB 4.6 comes out" + (when (cursor-initialized-p cur) + (multiple-value-bind (exists? skey-cur) + (cursor-current cur) + (declare (ignore exists?)) + (multiple-value-bind (exists? skey value pkey) + (cursor-pprev cur) + (if (lisp-compare-equal skey-cur skey) + (values exists? skey value pkey) + (setf (cursor-initialized-p cur) nil)))))) + (defgeneric cursor-pprev-nodup (cursor) (:documentation "Move to the previous non-duplicate element (with different key.) Returns has-tuple / secondary key / value / primary key.")) + (defmacro with-btree-cursor ((var bt) &body body) "Macro which opens a named cursor on a BTree (primary or not), evaluates the forms, then closes the cursor." @@ -439,6 +476,7 @@ start end)) (let ((sc (get-con index)) (end (or value end)) + (from-end (and from-end (not value-set-p))) (results nil)) (flet ((collector (k v pk) (push (funcall fn k v pk) results))) @@ -454,11 +492,12 @@ (lisp-compare-equal key start)) (lisp-compare<= key end)))) (value-increment () - ;; Step to the next key value + ;; Step to the next key value; + ;; from-end duplicate cursor is already there (if from-end - (pprev-hack cur) + (cursor-current cur) (cursor-pnext-nodup cur))) - (next-value () + (map-values () ;; Handle the next key value (multiple-value-bind (exists? skey val pkey) (value-increment) @@ -468,18 +507,23 @@ (map-duplicates skey)) (return-from map-index (nreverse results))))) + (next-duplicate (key) + (if from-end + (pprev-dup-hack cur key) + (cursor-pnext-dup cur))) (map-duplicates (key) ;; Map all duplicates for key value (multiple-value-bind (exists? skey val pkey) - (cursor-pnext-dup cur) + (next-duplicate key) (if exists? (progn (funcall fn skey val pkey) (map-duplicates key)) (progn - (cursor-pset-range cur key) - (next-value)))))) - (declare (dynamic-extent (function next-value) (function next-value-increment) + (unless from-end + (cursor-pset cur key)) + (map-values)))))) + (declare (dynamic-extent (function map-values) (function next-duplicate) (function continue-p) (function map-duplicates))) (multiple-value-bind (exists? skey val pkey) (cond (value-set-p @@ -487,9 +531,9 @@ ((and (not from-end) (null start)) (cursor-pfirst cur)) ((and from-end (null end)) - (cursor-last-range-hack cur)) + (cursor-plast cur)) (t (if from-end - (cursor-pset-range cur end) + (pset-range-for-descending cur end) (cursor-pset-range cur start)))) (if (and exists? (continue-p skey)) (progn @@ -497,23 +541,24 @@ (map-duplicates skey)) nil))))))))) -(defun pprev-hack (cur) - "Get the first duplicate instance of the prior value off the current cursor" - (let ((e? (cursor-pprev-nodup cur))) - (when e? - (let ((e? (cursor-pprev-nodup cur))) - (if e? - (cursor-pnext cur) - (cursor-pfirst cur)))))) - -(defun cursor-last-range-hack (cur) - "Get the first duplicate instance of the last value of the cursor's index" - (let ((e? (cursor-plast cur))) - (when e? - (let ((e? (cursor-pprev-nodup cur))) - (if e? - (cursor-pnext cur) - (cursor-pfirst cur)))))) +(defun pset-range-for-descending (cur end) + (if (cursor-pset cur end) + (progn + (cursor-next-nodup cur) + (cursor-pprev cur)) + (progn + (cursor-pset-range cur end) + (cursor-pprev cur)))) + +(defun pprev-dup-hack (cur key) + "Go back one step in a duplicate set, returns nil + if previous element is a different key. More efficient than + the current default implementation of cursor-pprev-dup" + (multiple-value-bind (exists? skey value pkey) + (cursor-pprev cur) + (when (lisp-compare-equal key skey) + (values exists? key value pkey)))) + ;; =============================== ;; Some generic utility functions --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/25 02:28:01 1.50 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/28 02:31:15 1.51 @@ -136,7 +136,7 @@ (defun initialize-user-parameters () (loop for (keyword variable) in *user-configurable-parameters* do (awhen (get-user-configuration-parameter keyword) - (setf variable it)))) + (setq variable it)))) ;; ;; COMMON STORE CONTROLLER FUNCTIONALITY @@ -165,7 +165,7 @@ "This is an instance cache and part of the metaclass protocol. Data stores should not override the default behavior.") - (instance-cache-lock :accessor instance-cache-lock :initform (ele-make-lock) + (instance-cache-lock :accessor instance-cache-lock :initform (ele-make-fast-lock) :documentation "Protection for updates to the cache from multiple threads. Do not override.") @@ -202,14 +202,16 @@ (defun cache-instance (sc obj) "Cache a persistent object with the controller." (declare (type store-controller sc)) - (ele-with-lock ((instance-cache-lock sc)) + (ele-with-fast-lock ((instance-cache-lock sc)) (setf (get-cache (oid obj) (instance-cache sc)) obj))) (defun get-cached-instance (sc oid class-name) "Get a cached instance, or instantiate!" (declare (type store-controller sc) (type fixnum oid)) - (let ((obj (get-cache oid (instance-cache sc)))) + (let ((obj + (ele-with-fast-lock ((instance-cache-lock sc)) + (get-cache oid (instance-cache sc))))) (if obj obj ;; Should get cached since make-instance calls cache-instance (make-instance class-name :from-oid oid :sc sc)))) --- /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/27 13:32:17 1.34 +++ /project/elephant/cvsroot/elephant/src/elephant/package.lisp 2007/04/28 02:31:16 1.35 @@ -222,12 +222,12 @@ #:cursor #:secondary-cursor #:make-cursor #:make-simple-cursor #:cursor-close #:cursor-duplicate #:cursor-current #:cursor-first #:cursor-last #:cursor-next #:cursor-next-dup - #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup + #:cursor-next-nodup #:cursor-prev #:cursor-prev-nodup #:cursor-prev-dup #:cursor-set #:cursor-set-range #:cursor-get-both #:cursor-get-both-range #:cursor-delete #:cursor-put #:cursor-pcurrent #:cursor-pfirst #:cursor-plast #:cursor-pnext #:cursor-pnext-dup #:cursor-pnext-nodup - #:cursor-pprev #:cursor-pprev-nodup #:cursor-pset + #:cursor-pprev #:cursor-pprev-dup #:cursor-pprev-nodup #:cursor-pset #:cursor-pset-range #:cursor-pget-both #:cursor-pget-both-range #:cursor-initialized-p @@ -267,6 +267,7 @@ ;; Various error conditions #:cross-reference-error #:controller-lost-error + #:persistent-class-not-indexed #:map-class-query #:get-query-instances --- /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/27 13:32:17 1.12 +++ /project/elephant/cvsroot/elephant/src/elephant/transactions.lisp 2007/04/28 02:31:16 1.13 @@ -120,7 +120,7 @@ If nested, the backend must support nested transactions." (let ((sc (gensym))) `(let ((,sc ,store-controller)) - (funcall #'execute-transaction ,store-controller + (funcall #'execute-transaction ,sc (lambda () , at body) :parent (if (owned-txn-p ,sc ,parent) (transaction-object ,parent) @@ -146,10 +146,9 @@ (,sc ,store-controller)) (if (owned-txn-p ,sc ,parent) (funcall ,txn-fn) - (funcall #'execute-transaction ,store-controller + (funcall #'execute-transaction ,sc ,txn-fn :parent nil - :transaction nil :retries ,retries ,@(remove-keywords '(:store-controller :parent :transaction :retries) keyargs)))))) From ieslick at common-lisp.net Sat Apr 28 02:31:41 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 22:31:41 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070428023141.DC4567E005@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv16753/tests Modified Files: elephant-tests.lisp testconditions.lisp Log Message: Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft --- /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/03/30 14:34:35 1.31 +++ /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/04/28 02:31:31 1.32 @@ -300,9 +300,15 @@ (defmacro signals-condition (&body body) `(handler-case - (progn , at body) - (condition () t) - (:no-error (&rest rest) (declare (ignore rest)) nil))) + (progn , at body) + (condition () t) + (:no-error (&rest rest) (declare (ignore rest)) nil))) + +(defmacro signals-specific-condition ((condition) &body body) + `(handler-case + (progn , at body) + (,condition () t) + (:no-error (&rest rest) (declare (ignore rest)) nil))) (defmacro signals-error (&body body) `(handler-case --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/22 03:02:35 1.2 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 02:31:31 1.3 @@ -28,8 +28,17 @@ (values (is-not-null (add-to-root 'inst1 inst1 :sc sc1)) (is-not-null (add-to-root 'inst2 inst2 :sc sc2)) - (signals-condition (add-to-root 'inst1 inst1 :sc2)) - (signals-condition (add-to-root 'inst2 inst2 :sc1)))) + (signals-specific-condition (cross-reference-error) + (add-to-root 'inst1 inst1 :sc sc2)) + (signals-specific-condition (cross-reference-error) + (add-to-root 'inst2 inst2 :sc sc1)))) (close-store sc1) (close-store sc2)))) - t t t t) \ No newline at end of file + t t t t) + +(deftest unindexed-class-condition + (let* ((sc (open-store *test-spec-primary* :recover t :deadlock-detect nil)) + (inst (make-instance 'pfoo :slot1 1 :sc sc))) + (signals-specific-condition (persistent-class-not-indexed) + (find-class-index 'pfoo :sc sc))) + t) \ No newline at end of file From ieslick at common-lisp.net Sat Apr 28 03:07:39 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 23:07:39 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070428030739.C03E55004B@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv23936/tests Modified Files: testconditions.lisp Log Message: Fixed test bug; cleaned up get-instances-by-xxx fns to use new map operators --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 02:31:31 1.3 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 03:07:39 1.4 @@ -20,7 +20,7 @@ (format t "~%Second store spec missing: ignoring") (values t t t t)) (let (*store-controller* - (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect nil)) + (sc1 *store-controller*) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil))) (unwind-protect (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) @@ -32,13 +32,14 @@ (add-to-root 'inst1 inst1 :sc sc2)) (signals-specific-condition (cross-reference-error) (add-to-root 'inst2 inst2 :sc sc1)))) - (close-store sc1) (close-store sc2)))) t t t t) (deftest unindexed-class-condition (let* ((sc (open-store *test-spec-primary* :recover t :deadlock-detect nil)) (inst (make-instance 'pfoo :slot1 1 :sc sc))) - (signals-specific-condition (persistent-class-not-indexed) - (find-class-index 'pfoo :sc sc))) + (unwind-protect + (signals-specific-condition (persistent-class-not-indexed) + (find-class-index 'pfoo :sc sc)) + (close-store sc))) t) \ No newline at end of file From ieslick at common-lisp.net Sat Apr 28 03:07:39 2007 From: ieslick at common-lisp.net (ieslick) Date: Fri, 27 Apr 2007 23:07:39 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070428030739.4DE9B5004D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv23936/src/elephant Modified Files: classindex.lisp controller.lisp Log Message: Fixed test bug; cleaned up get-instances-by-xxx fns to use new map operators --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 02:31:15 1.40 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 03:07:38 1.41 @@ -389,7 +389,7 @@ ;; USER MAPPING API ;; ====================== -(defun map-class (fn class) +(defun map-class (fn class &key collect) "Perform a map operation over all instances of class. Takes a function of one argument, a class instance" (let* ((class (if (symbolp class) @@ -400,9 +400,9 @@ (declare (ignore k)) (funcall fn v))) (declare (dynamic-extent map-fn)) - (map-btree #'map-fn class-idx)))) + (map-btree #'map-fn class-idx :collect collect)))) -(defun map-inverted-index (fn class index &rest args &key start end value from-end) +(defun map-inverted-index (fn class index &rest args &key start end (value nil value-p) from-end collect) "map-inverted-index maps a function of two variables, taking key and instance, over a subset of class instances in the order defined by the index. Specify the class and index by quoted @@ -431,7 +431,10 @@ (declare (ignore pkey)) (funcall fn key value))) (declare (dynamic-extent wrapper)) - (map-index #'wrapper index :start start :end end :value value :from-end from-end)))) + (if value-p + (map-index #'wrapper index :value value :collect collect) + (map-index #'wrapper index :start start :end end :from-end from-end :collect collect))))) + ;; ================= @@ -454,30 +457,27 @@ nil to start or end indicates, respectively, the lowest or highest value in the index")) + +(defun identity2 (k v) + (declare (ignore k)) + v) + +(defun identity3 (k v pk) + (declare (ignore k pk)) + v) + (defmethod get-instances-by-class ((class symbol)) (get-instances-by-class (find-class class))) (defmethod get-instances-by-class ((class persistent-metaclass)) - (let ((instances nil)) - (flet ((accum (c) - (declare (dynamic-extent c)) - (push c instances))) - (map-class #'accum class) - (nreverse instances)))) + (map-class #'identity class :collect t)) (defmethod get-instances-by-value ((class symbol) slot-name value) (get-instances-by-value (find-class class) slot-name value)) (defmethod get-instances-by-value ((class persistent-metaclass) slot-name value) (declare (type (or string symbol) slot-name)) - (let ((instances nil)) - (declare (type list instances)) - (flet ((collector (k v pk) - (declare (ignore k pk)) - (push v instances))) - (declare (dynamic-extent collector)) - (map-index #'collector (find-inverted-index class slot-name) :value value)) - (nreverse instances))) + (map-inverted-index #'identity2 class slot-name :value value :collect t)) (defmethod get-instance-by-value ((class symbol) slot-name value) (let ((list (get-instances-by-value (find-class class) slot-name value))) @@ -495,15 +495,7 @@ (defmethod get-instances-by-range ((class persistent-metaclass) idx-name start end) (declare (type (or fixnum null) start end) (type symbol idx-name)) - (let ((instances nil)) - (declare (type list instances)) - (flet ((collector (k v pk) - (declare (ignore k pk)) - (push v instances))) - (declare (dynamic-extent collector)) - (map-index #'collector (find-inverted-index class idx-name) - :start start :end end)) - (nreverse instances))) + (map-inverted-index #'identity2 class idx-name :start start :end end :collect t)) (defun drop-instances (instances &key (sc *store-controller*)) "Removes a list of persistent objects from all class indices --- /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/28 02:31:15 1.51 +++ /project/elephant/cvsroot/elephant/src/elephant/controller.lisp 2007/04/28 03:07:38 1.52 @@ -220,7 +220,7 @@ "Reset the instance cache (flush object lookups). Useful for testing. Does not reclaim existing objects so there will be duplicate instances with identical functionality" - (ele-with-lock ((instance-cache-lock sc)) + (ele-with-fast-lock ((instance-cache-lock sc)) (setf (instance-cache sc) (make-cache-table :test 'eql)))) From ieslick at common-lisp.net Sat Apr 28 14:24:57 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 10:24:57 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070428142457.88305431BF@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv26611 Modified Files: Makefile data-store-reference.texinfo elephant.texinfo installation.texinfo intro.texinfo reference.texinfo user-guide.texinfo Log Message: Final doc cleanup; added contents to PDF (finally) --- /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6 +++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/28 14:24:57 1.7 @@ -9,4 +9,5 @@ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo pdf: includes-stuff - texi2dvi --pdf elephant.texinfo + texi2pdf --texinfo=@setcontentsaftertitlepage --texinfo=@setshortcontentsaftertitlepage \ + elephant.texinfo --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 02:31:07 1.8 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 14:24:57 1.9 @@ -28,12 +28,12 @@ * Registration: DSR Registration. Register the data store for the appropriate controller specifications. * Store Controllers: DSR Store Controllers. Subclassing the store controller. * Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. -* Memory Utilities: DSR Memory Utilities. Writing primitive C types. * Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access. * Collections: DSR Collections. BTrees and indices. * Cursors: DSR Cursors. Traversing BTrees. * Transactions: DSR Transactions. Transaction implementation. * Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations. + at c * Memory Utilities: DSR Memory Utilities. Writing primitive C types. @c * Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries @end menu @@ -241,7 +241,7 @@ @node DSR Handling Serialization @comment node-name, next, previous, up @section Handling Serialization - at cindex Serialization + at cindex Serializer Data stores must initialize @ref{Class elephant:store-controller} with internal serializer functions. Packages @code{elephant-serializer1} --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 02:31:07 1.11 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 14:24:57 1.12 @@ -2,9 +2,10 @@ @c %**start of header @setfilename elephant.info @settitle Elephant User Manual - at setchapternewpage odd @c %**end of header + at setchapternewpage odd + @copying @quotation Elephant System @* @@ -44,7 +45,6 @@ @top Copyright @insertcopying - at end ifnottex @ifhtml @menu @@ -74,13 +74,14 @@ * Variable Index:: * Colophon:: @end menu - @end ifhtml @node Table of Contents @unnumbered @comment node-name, next, previous, up - at contents + at contents + + at end ifnottex @include intro.texinfo @include tutorial.texinfo @@ -131,4 +132,8 @@ @uref{http://www.common-lisp.net/project/elephant/}) and submit a patch. + at iftex + at shortcontents + at contents + at end iftex @bye --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 02:31:07 1.12 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 14:24:57 1.13 @@ -11,7 +11,7 @@ * Loading Elephant:: Loading Elephant and the data store loading protocol. * Berkeley DB:: Installing support for the Berkeley DB data store. * Berkeley DB Example:: An example of installing and running the Berkeley DB data store. -* Upgrading Berkeley DB Database:: How to upgrade to a new version of Berkeley DB. +* Upgrading Berkeley DB Databases:: How to upgrade to a new version of Berkeley DB. * CL-SQL:: Install and connecting to the CL-SQL data store. * CL-SQL Example:: An example of using the CL-SQL data store. * Elephant on Windows:: More details about running Elephant on Windows @@ -309,8 +309,8 @@ @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} @end itemize - at item Test your new application and report any bugs that arise to elephant-devel at common-lisp.net - at end itemize + at item Test your new application and report any bugs that arise to @email{elephant-devel@@common-lisp.net} + at end enumerate @emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} --- /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/02 00:51:06 1.6 +++ /project/elephant/cvsroot/elephant/doc/intro.texinfo 2007/04/28 14:24:57 1.7 @@ -3,7 +3,9 @@ @node Introduction @comment node-name, next, previous, up @chapter Introduction - at cindex Introduction + at cindex History + at cindex Goals + Elephant is a persistent object protocol and database for Common Lisp. The persistent protocol component of elephant overrides class --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 02:31:07 1.14 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 14:24:57 1.15 @@ -7,22 +7,22 @@ @cindex API Reference @menu -* Store Controllers:: Connecting to a data store. -* Persistent Object API:: Defining persistent classes and creating and manipulating persistent indices. -* Persistent Object Indexing API:: Convenient indexing of persistent classes. -* Persistent Set API:: Maintaining persistent collections the easy way. +* Store Controllers: Store Controller API. Connecting to a data store. +* Persistent Objects: Persistent Object API. Defining persistent classes and creating and manipulating persistent indices. +* Persistent Object Indexing: Persistent Object Indexing API. Convenient indexing of persistent classes. +* Persistent Sets: Persistent Set API. Maintaining persistent collections the easy way. @c * Query Interfaces:: Finding instances. -* BTree API:: BTrees and indices, a low level persistent data structure. -* Cursor API:: Traversing BTrees. -* Index Cursor API:: Traversing BTree Indicies. -* Transaction API:: Transaction functions. -* Migration and Upgrading API:: Migration and upgrading. -* Miscellaneous API:: Other functions and data store specific functions +* BTrees: BTree API. BTrees and indices, a low level persistent data structure. +* BTree Cursors: BTree Cursor API. Traversing BTrees. +* Index Cursors: Index Cursor API. Traversing BTree Indicies. +* Transactions: Transaction API. Transaction functions. +* Migration and Upgrading: Migration and Upgrading API. Migration and upgrading. + at c * Miscellaneous API:: Other functions and data store specific functions @end menu @node Store Controller API @comment node-name, next, previous, up - at section Store Controller API + at section Store Controllers @cindex Store Controller @cindex API @@ -49,7 +49,7 @@ @node Persistent Object API @comment node-name, next, previous, up - at section Persistent Object API + at section Persistent Objects @cindex Persistent Objects @cindex API @@ -63,7 +63,7 @@ @node Persistent Object Indexing API @comment node-name, next, previous, up - at section Persistent Object Indexing API + at section Persistent Object Indexing @cindex Persistent Object Indexing @cindex API @@ -100,8 +100,8 @@ @node Persistent Set API @comment node-name, next, previous, up - at section Persistent Set API - at cindex Persistent Set API + at section Persistent Sets + at cindex Persistent Sets Persistent sets are a simple persistent collection abstraction. They maintain an unordered collection of objects. Unlike the normal @@ -132,7 +132,7 @@ @node BTree API @comment node-name, next, previous, up - at section BTree API + at section BTrees @cindex BTree @cindex API @@ -167,9 +167,9 @@ @include includes/fun-elephant-map-index.texinfo - at node Cursor API + at node BTree Cursor API @comment node-name, next, previous, up - at section Cursor API + at section Btree Cursors @cindex Cursors @cindex API @@ -200,7 +200,7 @@ @node Index Cursor API @comment node-name, next, previous, up - at section Index Cursor API + at section Index Cursors @cindex Cursor @cindex Index @cindex Indices @@ -239,7 +239,7 @@ @node Transaction API @comment node-name, next, previous, up - at section Transaction API + at section Transactions @cindex Transaction API @cindex API @@ -256,7 +256,7 @@ @node Migration and Upgrading API @comment node-name, next, previous, up - at section Migration and Upgrading API + at section Migration and Upgrading @cindex Migration and Upgrading @cindex API --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 02:31:11 1.20 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 14:24:57 1.21 @@ -882,7 +882,7 @@ setting a second cursor to the same location. Cursors can be in two states: initialized and uninitialized. - at ref{Cursor API} for details. + at ref{BTree Cursor API} for details. To initialize a cursor, you have to use one of the initializing functions to select a key-value pair in the btree. @@ -1068,6 +1068,7 @@ that the cursor is now uninitialized (@code{cursor-initialized-p}) and needs to be reset by a set or set both call. + at xref{Index Cursor API} for further details. @node Multi-threaded Applications @comment node-name, next, previous, up From ieslick at common-lisp.net Sat Apr 28 14:24:57 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 10:24:57 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc/oldfiles Message-ID: <20070428142457.F1CA547005@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc/oldfiles In directory clnet:/tmp/cvs-serv26611/oldfiles Modified Files: TODO Log Message: Final doc cleanup; added contents to PDF (finally) --- /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:14 1.1 +++ /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 14:24:57 1.2 @@ -12,24 +12,9 @@ TASKS TO GET TO FINAL RELEASE: Bugs: -- Fix awkward serializer API -- Support for asdf-install? - - problem with other .asd files in root - -Lisp Support: -- Validate OpenMCL pre-1.1 on Mac OS X -- Validate OpenMCL 1.1 and/or 64-bit on Mac OS X? -- Verify db_deadlock for other lisps (launch and kill background program I/F) - -Documentation: -- Update install, build and test procedures -- Upgrade, migration and other system level issues -- Performance and design issues - - More notes about transaction performance - - Serious discussion of threading implications - - Add notes about with/ensure-transaction usage (abort & commit behavior on exit) - - Add notes about deadlock-detect - - Add notes about checkpoint (null in SQL?) +- Validate OpenMCL 1.0 on Mac OS X +- Lispworks on Linux +- Verify install, build and test documentation and procedures 0.6.1 - Features COMPLETED to date ---------------------------------- From ieslick at common-lisp.net Fri Apr 27 03:14:56 2007 From: ieslick at common-lisp.net (ieslick) Date: Thu, 26 Apr 2007 23:14:56 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070427031456.2159F3700D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv2708 Modified Files: data-store-reference.texinfo docstrings.lisp elephant-design.texinfo elephant.texinfo installation.texinfo reference.texinfo scenarios.texinfo tutorial.texinfo user-guide.texinfo Log Message: Latest edits, plus integration with Roberts --- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/02 00:51:06 1.6 +++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/27 03:14:55 1.7 @@ -8,16 +8,16 @@ This reference includes functions that need to be overridden, classes inherited from or other action taken to implement support for a new -data store backend. Included are the exported elephant functions that -need methods defined on them as well as the backend-only functions -exported in backends.lisp. Some functions here are utilities from the +data store. Included are the exported elephant functions that +need methods defined on them as well as the data-store-only functions +exported in data-store-api.lisp. Some functions here are utilities from the main elephant package that support store implementations, but are not required. Migration, class indices and query interfaces are implemented on top of the store API and require no special support by implementors. -Because the number of backend implementors is small, this is a minimal -documentation set intended to serve as an initial guide and a +Because the number of data store implementors is small, this is a +minimal documentation set intended to serve as an initial guide and a reference. However, it is anticipated that some interaction will be needed with the developers to properly harden a datastore for release. @@ -25,7 +25,7 @@ relevant to them. @menu -* Registration: DSR Registration. Register the backend to parse controller specifications. +* Registration: DSR Registration. Register the data store for the appropriate controller specifications. * Store Controllers: DSR Store Controllers. Subclassing the store controller. * Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects. * Memory Utilities: DSR Memory Utilities. Writing primitive C types. @@ -43,22 +43,22 @@ @cindex Registration and Initialization Elephant looks at the first element of the specification list to -determine which backend code base to use. The master table for this -information is @code{*elephant-backends*} in elephant/controller.lisp. -This will need to be augmented for every backend with the +determine which data store module to use. The master table for this +information is @code{*elephant-data-stores*} in elephant/controller.lisp. +This will need to be augmented for every data store with the specification keyword tag to be used (such as @code{:BDB} or @code{:CLSQL}) and the required asdf dependencies. -In addition, the backend source should use an eval-when statement to +In addition, the data store source should use an eval-when statement to call the following function: - at include includes/fun-elephant-backend-register-backend-con-init.texinfo + at include includes/fun-elephant-data-store-register-data-store-con-init.texinfo -If the backend requires any special user-specified configuration, +If the data store requires any special user-specified configuration, augment the key types in config.sexp with what you need and use the following function to access. - at include includes/fun-elephant-backend-get-user-configuration-parameter.texinfo + at include includes/fun-elephant-data-store-get-user-configuration-parameter.texinfo @node DSR Store Controllers @comment node-name, next, previous, up @@ -69,25 +69,25 @@ which are called by open-store and close-store respectively. @include includes/class-elephant-store-controller.texinfo - at include includes/fun-elephant-backend-open-controller.texinfo - at include includes/fun-elephant-backend-close-controller.texinfo - at include includes/fun-elephant-backend-connection-is-indeed-open.texinfo + at include includes/fun-elephant-data-store-open-controller.texinfo + at include includes/fun-elephant-data-store-close-controller.texinfo + at include includes/fun-elephant-data-store-connection-is-indeed-open.texinfo For upgrading and opening legacy databases it is important that a store be able to indicate which version of elephant was used to create it. This governs the chosen serializer, mappings between elephant symbols used in an old vs. new version, etc. Because this is called to initialize the serializer, it must directly implemented by the -backend without using the serializer. +data store without using the serializer. - at include includes/fun-elephant-backend-database-version.texinfo + at include includes/fun-elephant-data-store-database-version.texinfo There are some utilities for serializing simple data without a serializer using the memutil package. - at include includes/fun-elephant-backend-serialize-database-version-key.texinfo - at include includes/fun-elephant-backend-serialize-database-version-value.texinfo - at include includes/fun-elephant-backend-deserialize-database-version-value.texinfo + at include includes/fun-elephant-data-store-serialize-database-version-key.texinfo + at include includes/fun-elephant-data-store-serialize-database-version-value.texinfo + at include includes/fun-elephant-data-store-deserialize-database-version-value.texinfo @node DSR Persistent Objects and Slot Access @@ -106,22 +106,22 @@ that functions such as map-btree do not need a store-controller argument. (NOTE: Should this function be user visible?) - at include includes/fun-elephant-backend-get-con.texinfo + at include includes/fun-elephant-data-store-get-con.texinfo All objects require a unique object identifier. During new object -creation the backend is asked to produce a unique id. +creation the data store is asked to produce a unique id. - at include includes/fun-elephant-backend-next-oid.texinfo + at include includes/fun-elephant-data-store-next-oid.texinfo These functions are called by the metaclass protocol to implement the appropriate operations on persistent class slots. Unless protected by a transaction, the side effects of these functions should be atomic, persistent and visible to other threads on completion. - at include includes/fun-elephant-backend-persistent-slot-writer.texinfo - at include includes/fun-elephant-backend-persistent-slot-reader.texinfo - at include includes/fun-elephant-backend-persistent-slot-boundp.texinfo - at include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo + at include includes/fun-elephant-data-store-persistent-slot-writer.texinfo + at include includes/fun-elephant-data-store-persistent-slot-reader.texinfo + at include includes/fun-elephant-data-store-persistent-slot-boundp.texinfo + at include includes/fun-elephant-data-store-persistent-slot-makunbound.texinfo @node DSR Collections @comment node-name, next, previous, up @@ -137,9 +137,9 @@ @include includes/class-elephant-btree-index.texinfo @include includes/class-elephant-indexed-btree.texinfo -To create the backend-appropriate type of btree, the backend +To create the data store-appropriate type of btree, the data store implements this method (and possibly related methods) aginst their store-controller. - at include includes/fun-elephant-backend-build-btree.texinfo + at include includes/fun-elephant-data-store-build-btree.texinfo Most of the user-visible operations over BTrees must be implemented. Class indexing functions such as @code{map-class} and @@ -181,19 +181,19 @@ These functions must be implemented or stubbed by all data stores. - at include includes/fun-elephant-backend-execute-transaction.texinfo + at include includes/fun-elephant-data-store-execute-transaction.texinfo - at include includes/fun-elephant-backend-controller-start-transaction.texinfo - at include includes/fun-elephant-backend-controller-commit-transaction.texinfo - at include includes/fun-elephant-backend-controller-abort-transaction.texinfo + at include includes/fun-elephant-data-store-controller-start-transaction.texinfo + at include includes/fun-elephant-data-store-controller-commit-transaction.texinfo + at include includes/fun-elephant-data-store-controller-abort-transaction.texinfo These are supporting functions and variables for implementing transactions. - at include includes/var-elephant-backend-star-current-transaction-star.texinfo - at include includes/fun-elephant-backend-make-transaction-record.texinfo - at include includes/fun-elephant-backend-transaction-store.texinfo - at include includes/fun-elephant-backend-transaction-object.texinfo + at include includes/var-elephant-data-store-star-current-transaction-star.texinfo + at include includes/fun-elephant-data-store-make-transaction-record.texinfo + at include includes/fun-elephant-data-store-transaction-store.texinfo + at include includes/fun-elephant-data-store-transaction-object.texinfo ;; Designer considerations: ;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction @@ -229,7 +229,7 @@ @section Handling Serialization @cindex Serialization -Backends must initialize @ref{Class elephant:store-controller} with +Data stores must initialize @ref{Class elephant:store-controller} with internal serializer functions. Packages @code{elephant-serializer1} and @code{elephant-serializer2} contains serialize and deserialize methods on buffer-streams as defined in @code{elephant-memutil}. The @@ -240,18 +240,18 @@ NOTE: This should perhaps become entirely the job of the data store to decide how to serialize values and for a specific version, what serializer to use. The elphant main package can define serializers -for use by different backends. +for use by different data stores. @end verbatim - at include includes/fun-elephant-backend-serialize.texinfo - at include includes/fun-elephant-backend-deserialize.texinfo + at include includes/fun-elephant-data-store-serialize.texinfo + at include includes/fun-elephant-data-store-deserialize.texinfo These utility functions are useful if a data store does not have the ability to store variable length binary data. They are based on the @code{cl-base64} library. - at include includes/fun-elephant-backend-serialize-to-base64-string.texinfo - at include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo + at include includes/fun-elephant-data-store-serialize-to-base64-string.texinfo + at include includes/fun-elephant-data-store-deserialize-from-base64-string.texinfo @node DSR Memory Utilities @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/docstrings.lisp 2004/09/19 17:44:43 1.1 +++ /project/elephant/cvsroot/elephant/doc/docstrings.lisp 2007/04/27 03:14:55 1.2 @@ -759,7 +759,9 @@ escaped in symbol names, but if a docstring contains invalid Texinfo markup, you lose." (handler-bind ((warning #'muffle-warning)) - (let ((directory (merge-pathnames (pathname directory)))) + (let ((directory (merge-pathnames (pathname directory))) + (*print-right-margin* 200)) + (declare (special *print-miser-width*)) (ensure-directories-exist directory) (dolist (package packages) (dolist (doc (collect-documentation (find-package package))) --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/21 17:22:35 1.4 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/27 03:14:55 1.5 @@ -1,5 +1,6 @@ @c -*-texinfo-*- + at c *** FINISH *** @node Elephant Architecture @comment node-name, next, previous, up @chapter Elephant Architecture @@ -10,17 +11,13 @@ support easy upgrading, repository migration, shared functionality between data stores and general hygene. -The current architecture is modularized into the following pieces: +The architecture has been carefully modularized: - at verbatim -[Picture describing] Metaclass, Store controller, persistent objects, -data stores, serializer & memutils. Derived features such as class -indexing, migration, query system and root operations can also be -illustrated? - at end verbatim + at image{ArchDiagram,,4.5in,[Architecture Diagram],.png} To get a feeling for what is happening inside elephant, it is probably -most useful to walk through the various major protocols: +best to walk through the various major protocols to see how these +components participate in implementing them. @itemize @item Initialization of a store controller @@ -32,23 +29,247 @@ @section Initializing a store controller -When the main elephant @code{open-store} function is called, it calls - at code{get-controller} which grabs an existing store controller if the -spec is identical, or builds a new controller. Building the -controller requires loading any dependencies via asdf, calling a -backend initialization function (if it is the first instance of that -backend being created), and then calling an initialization function -that returns a @code{store-controller} subclass instance specific to -that backend. +When the main elephant @code{open-store} function is called with a +specification, it calls get-controller which first checks to see if a +controller already exists for that spec. + +If there is no controller, it calls @code{build-controller} to +construct one. If the data store code base is not present, + at code{load-data-store} is called to ensure that any asdf dependencies +are satisfied. The associations for asdf dependencies are statically +configured in @code{*elephant-data-stores*} for each data store type +supported by elephant. + +While being loaded, the data store is responsible for calling + at code{register-data-store-con-init} to register a data store +initialization function for its spec type (i.e. :BDB or :CLSQL). +For example, from bdb-controller.lisp: + + at lisp +(eval-when (:compile-toplevel :load-toplevel) + (register-data-store-con-init :bdb 'bdb-test-and-construct)) + at end lisp + +This mapping between spec types and initialization functions is +accessed by @code{lookup-data-store-con-init} from within + at code{build-controller}. The function returned by + at code{lookup-data-store-con-init} is passed the full specification and +returns a @code{store-controller} subclass instance for the specified +data store. + +The new controller is stored in the @code{*dbconnection-spec*} hash +table, associating the object with its specification. Finally +Elephant calls open-controller to actually establish a connection to +or create the files of the data store. + +Finally, if the default store controller @code{*store-controller*} is +nil, it will be initialized with the new store controller, otherwise +the original value is left in @code{*store-controller*} until that +store controller is closed using @code{close-store}. -Elephant than calls open-controller to actually establish a connection -to or create the files of the data store. +The data store implementor has access to various utilities to aid +initialization. + + at itemize + at item @code{get-user-configuration-parameter} - Access symbol tags + in my-config.sexp to access data store specific user +configuration. You can also add special variables to variables.lisp +and add a tag-variable pair to @code{*user-configurable-parameters*} +in variables.lisp to automatically initialize it when the store +controller is opened. + at item @code{get-con} behavior when store is closed or lost + at item @code{database-version} a store controller implements this + in order to tell Elephant what serializer to use. Currently, + version 0.6.0 databases use serializer1 and all later database + use serializer version 2. This is to ensure that a given version + of the Elephant code can open databases from prior versions in + order to properly upgrade to the new code base. + at item Symbol conversions. To aid in opening legacy databases, a + symbol conversion facility is provided in controller.lisp to + be applied to any symbols extracted from the legacy data store. + (if, for instance, the type name of subclasses changed, such as + sleepycat-btree becoming bdb-btree) + at end itemize + +At this point, all operations referencing the store controller should +be able to proceed. + +At the end of a session, @section Persistent Object Creation + +The only thing that a data store has to do to support new object +creation, other than implement the slot protocol, is implement the +method @code{next-oid} to return the next unique object id for the +persistent object being created. + +Existing objects are created during deserialization of object +references. The serializer subsystem is built-into the core of +elephant and can be used by data stores. The serializer is abstracted +so that multiple serializers can be co-resident and the data store can +choose the appropriate one. The abstraction boundary between the +serializer, the data store, and the core Elephant system is not +perfect, so be aware and refer to existing data store implementations +if in doubt. + +A serializer takes as arguments the store-controller, lisp object and +a @code{buffer-stream} from the memory utility library and returns the +buffer-stream with the binary serialized object. The deserializer +reverses this process. For all lisp objects except persistent +classes, this means reallocating the storage space for the object and +recreating all its contents. Deserializing a standard object results +in a new standard object of the same class with the same slot values. + +Persistent classes are dealt with specially. When a persistent object +is serialized, it's oid and class are stored in the + at code{buffer-stream}. On deserialization it uses the oid to check in +the store-controller's cache for an existing placeholder object. If +the cache misses, then it creates a new placeholder object using the +class and oid as described in @xref{Persistent Classes and Objects}. +The store controller contains a cache instance that is automatically +initialized by the core Elephant object protocol. + +Currently the serializer is selected by the core Elephant code based +on the store controller's database version. See the reference section +for details on implementing the store-controller database version +method. It is a relatively small change to have the data store choose +its own serializer, however we will have to tighten up and document +the contracts between the Elephant core code, serializer and data store. + @section Persistent Slot Protocol - at section Persistent Slot Protocol - at section Persistent Collection Protocol + +The core protocol that the data store needs to support is the slot +access protocol. During object initialization, these functions are +called to initialize the slots of the object. The four functions are: + + at itemize + at item @code{persistent-slot-reader} + at item @code{persistent-slot-writer} + at item @code{persistent-slot-boundp} + at item @code{persistent-slot-makunbound} + at end itemize + +More details can be found in the data store api reference section. In +short, these functions specialize on the specific @code{store-controller} of +the data store and take instances, values and slotnames as appropriate. + +Typically the oid will be extracted from the instance and be used to +update a table or record where the oid and slotname identifies the +value. A slot is typically unbound when no value exists (as opposed to +nil). + + at section Persistent Collection Protocols + +The BTree protocol is the most extensive interface that data stores must +implement. Data store implementations are required to subclass the +abstract classes @code{btree}, @code{indexed-btree}, and @code{index} +and implement their complete APIs. Each class type is constructed +by Elephant using a @code{store-controller} that builds them. These +methods are @code{build-btree}, @code{build-indexed-btree} and + at code{build-index}. + +The @code{get-value} interface is similar to the persistent +slot reader and writer, but instead of using oid and slotname to set +values, it uses the btree oid and a key value as a unique identifier +for a value. + +The BTree protocol almost requires an actual BTree implementation to +be at all efficient. Keys and values need to be accessible via the +cursor API, which means they need to be walked linearly in the sort +order of the keys (described in @xref{Persistent BTrees}). + +An indexed BTree automatically maintains a hash table of the indices +defined on it so that users can access them by mapping or +lookup-by-name. The data store also has access to this interface. + +A BTree index must also maintain a connection to its parent BTree so +that an index value can be used as a primary tree key to retrieve the +primary BTree value as part of the @code{cursor-pnext} and + at code{cursor-pprev} family of methods. + +The contract of @code{remove-kv} is that the storage in the data store +is actually freed for reuse. + +Persistent set implemenation is optional. A default BTree based +implementation is provided by default + @section Implementing Transactions +One of the most important pieces of functionality remaining to discuss +is implementing transactions. In existing data stores, transactions +are merely extensions of the underlying start, commit and abort +methods of the 3rd party library or server being used. The Elephant +user interfaces to these functions in two ways: a call to + at code{execute-transaction} or explicit calls to @code{controller-start-transaction}, + at code{controller-commit-transaction} and @code{controller-abort-transaction}. + + at subsection Implementing Execute Transaction + +The macros @code{with-transaction} and @code{ensure-transaction} wrap +access to the data store's @code{execute-transaction}. This function +has a rich contract. It accepts as arguments the store controller, a +closure that executes the transaction body and a set of keywords. +Keywords required to be supported by the method (or ignored without +loss of semantics) are @code{:parent} and @code{:retries}. + +The semantics of @code{with-transaction} are that a new transaction +will always be requested of the data store. If a transaction exists, + at code{ensure-transaction} will merely call the transaction closure. +If not it will function as a call to @code{with-transaction}. + + at code{execute-transaction} is that it must ensure that the transaction +closure is executed within a dynamic context that insures the ACID +properties of any database operations (@code{pset}, at code{btree} or +persistent slot operations). If there is a non-local exit during this +execution, the transaction should be aborted. If it returns normally, +the transaction is committed. The integer in the @code{:retries} +argument dictates how many times @code{execute-transaction} should +retry the transaction before failing. + +Elephant provides some bookkeeping to the data store to help with +nested transactions by using the @code{*current-transaction*} dynamic +variable. In the dynamic context of the transaction closure, another +call to @code{execute-transaction} may occur with the transaction +argument defaulting to the value of @code{*current-transaction*}. The +data store has to decide how to handle these cases. To support this, +the first call to execute transaction can create a dynamic binding for + at code{*current-transaction*} using the @code{make-transaction-record} +call. This creates a transaction object that records the store +controller that started the transaction and any data store-specific +transaction data. + +The current policy is that the body of a transaction is executed with +the @code{*store-controller*} variable bound to the store-controller +object creating the transaction. This is important for default +arguments and generally helps more than it hurts, so is an +implementation requirement placed on @code{execute-transaction}. + +If two nested calls to @code{with-transaction} are made successively +in a dynamic context, the data store can create true nested +transactions. The first transaction is passed to the @code{:parent} +argument of the second. The second can choose to just continue the +current transaction (the CLSQL data store policy) or to nest the +transaction (the BDB data store policy). + + at subsection Interleaving Multiple Store Transactions + +Finally, some provision is made for the case where two store +controllers have concurrently active transactions in the same thread. +This feature was created to allow for migration, where a read from one +database happens in one transaction, and while active has to writes to +another data store with a valid transaction. + +The trick is that @code{with-transaction} checks to see if the current +transaction object is the same as the @code{store-controller} object +passed to the @code{:store-controller} argument. If not, a fresh +transaction is started. + +Currently no provision is made for more than two levels of multi-store +nesting as we do not implement a full transaction stack (to avoid +walking the stack on each call to handle this rare case). If a third +transaction is started by the store controller that started the first +transaction, it will have no access to the parent transaction which +may be a significant source of problems for the underlying database. + --- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/21 17:22:35 1.9 +++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/27 03:14:55 1.10 @@ -6,8 +6,8 @@ @copying Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. -2006 SQL Backend, Copyright @copyright{} 2006 Robert L. Read. -2007 Rewrite, Copyright @copyright{} 2007 Ian Eslick. +Version 0.5, Copyright @copyright{} 2006 Robert L. Read. +Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @quotation Permission is granted to copy, distribute and/or modify this document @@ -21,7 +21,7 @@ @titlepage @title Elephant User Manual @subtitle Elephant version 0.9 - at author Ian Eslick and Ben Lee + at author By Ian Eslick with Robert Read and Ben Lee @c The following two commands @c start the copyright page. --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/21 17:22:35 1.10 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/27 03:14:55 1.11 @@ -212,7 +212,8 @@ ... => # -(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" "mydb" "myuser" "")))) +(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" + "mydb" "myuser" "")))) ... ASDF loading messages ... --- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/25 02:27:56 1.12 +++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/27 03:14:55 1.13 @@ -8,21 +8,23 @@ @menu * Store Controllers:: Connecting to a data store. -* Persistent Objects:: Defining persistent classes and creating and manipulating persistent indices. -* Persistent Object Indexing:: Convenient indexing of persistent classes. -* Persistent Sets:: Maintaining persistent collections the easy way. +* Persistent Object API:: Defining persistent classes and creating and manipulating persistent indices. +* Persistent Object Indexing API:: Convenient indexing of persistent classes. +* Persistent Set API:: Maintaining persistent collections the easy way. @c * Query Interfaces:: Finding instances. -* BTrees:: BTrees and indices, a low level persistent data structure. -* Cursors:: Traversing BTrees. -* Transactions:: Transactions. -* Migration and Upgrading:: Migration and upgrading. -* Miscellaneous:: Other functions and data store specific functions +* BTree API:: BTrees and indices, a low level persistent data structure. +* Cursor API:: Traversing BTrees. +* Index Cursor API:: Traversing BTree Indicies. +* Transaction API:: Transaction functions. +* Migration and Upgrading API:: Migration and upgrading. +* Miscellaneous API:: Other functions and data store specific functions @end menu - at node Store Controllers + at node Store Controller API @comment node-name, next, previous, up - at section Store Controllers - at cindex Store Controllers + at section Store Controller API + at cindex Store Controller + at cindex API Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the @@ -45,10 +47,11 @@ @include includes/fun-elephant-root-existsp.texinfo @include includes/fun-elephant-map-root.texinfo - at node Persistent Objects + at node Persistent Object API @comment node-name, next, previous, up - at section Persistent Objects + at section Persistent Object API @cindex Persistent Objects + at cindex API @ref{Class elephant:persistent-metaclass} can be used as the :metaclass argument in a defclass form to create a persistent object. @@ -58,15 +61,17 @@ @include includes/macro-elephant-defpclass.texinfo @include includes/fun-elephant-drop-pobject.texinfo - at node Persistent Object Indexing + at node Persistent Object Indexing API @comment node-name, next, previous, up - at section Persistent Object Indexing + at section Persistent Object Indexing API @cindex Persistent Object Indexing + at cindex API + @subsection Indexed Object Accessors @include includes/fun-elephant-map-class.texinfo - at include includes/fun-elephant-map-class-index.texinfo + at include includes/fun-elephant-map-inverted-index.texinfo @include includes/fun-elephant-get-instances-by-class.texinfo @include includes/fun-elephant-get-instance-by-value.texinfo @@ -93,10 +98,10 @@ @include includes/fun-elephant-add-class-derived-index.texinfo @include includes/fun-elephant-remove-class-derived-index.texinfo - at node Persistent Sets + at node Persistent Set API @comment node-name, next, previous, up - at section Persistent Sets - at cindex Persistent Sets + at section Persistent Set API + at cindex Persistent Set API Persistent sets are a simple persistent collection abstraction. They maintain an unordered collection of objects. Unlike the normal @@ -125,10 +130,12 @@ @c @include includes/fun-elephant-get-query-results.texinfo @c @include includes/fun-elephant-map-class-query.texinfo - at node BTrees + at node BTree API @comment node-name, next, previous, up - at section BTrees - at cindex BTrees + at section BTree API + at cindex BTree + at cindex API + Persistent collections inherit from @ref{Class elephant:persistent-collection} and consist of the @ref{Class elephant:btree}, @ref{Class elephant:indexed-btree} and @@ -142,29 +149,40 @@ @include includes/fun-elephant-setf-get-value.texinfo @include includes/fun-elephant-remove-kv.texinfo @include includes/fun-elephant-existsp.texinfo - + at include includes/fun-elephant-drop-btree.texinfo @include includes/fun-elephant-map-btree.texinfo - at include includes/fun-elephant-map-index.texinfo + +These functions are only defined on indexed btrees. + + at include includes/fun-elephant-make-indexed-btree.texinfo @include includes/fun-elephant-add-index.texinfo @include includes/fun-elephant-get-index.texinfo @include includes/fun-elephant-get-primary-key.texinfo @include includes/fun-elephant-remove-index.texinfo - at include includes/fun-elephant-drop-pset.texinfo - at node Cursors +This function is only valid for indexes. + + at include includes/fun-elephant-map-index.texinfo + + at node Cursor API @comment node-name, next, previous, up - at section Cursors + at section Cursor API @cindex Cursors + at cindex API Cursors are objects of type cursor (@pxref{Class elephant:cursor}) -which provide methods for complex traversals of BTrees. +which provide methods for complex traversals of BTrees. @include includes/macro-elephant-with-btree-cursor.texinfo @include includes/fun-elephant-make-cursor.texinfo @include includes/fun-elephant-cursor-close.texinfo - @include includes/fun-elephant-cursor-duplicate.texinfo + at include includes/fun-elephant-cursor-initialized-p.texinfo + +Each of the following methods return multiple values consisting of + at code{(exists? key value)}. + @include includes/fun-elephant-cursor-current.texinfo @include includes/fun-elephant-cursor-first.texinfo @include includes/fun-elephant-cursor-last.texinfo @@ -177,6 +195,25 @@ @include includes/fun-elephant-cursor-delete.texinfo @include includes/fun-elephant-cursor-put.texinfo + at node Index Cursor API + at comment node-name, next, previous, up + at section Cursor API + at cindex Cursors + at cindex Index + at cindex Indices + at cindex API + +Index cursors are made the same way standard cursors are, with a call +to @code{make-cursor}, except with the index as the argument instead +of a standard btree. In addition to the standard cursor operations, +which provide the direct key and value of a @code{btree-index}, the +following class of ``p'' cursors work on an index and allow you to get +the primary value of the @code{indexed-btree} that the + at code{btree-index} belongs to. + +They each return multiple values @code{(exists? key primary-value +primary-key)}. + @include includes/fun-elephant-cursor-pcurrent.texinfo @include includes/fun-elephant-cursor-pfirst.texinfo @include includes/fun-elephant-cursor-plast.texinfo @@ -194,10 +231,11 @@ @include includes/fun-elephant-cursor-prev-nodup.texinfo @include includes/fun-elephant-cursor-pprev-nodup.texinfo - at node Transactions + at node Transaction API @comment node-name, next, previous, up - at section Transactions - at cindex Transactions + at section Transaction API + at cindex Transaction API + at cindex API @include includes/macro-elephant-with-transaction.texinfo @@ -210,10 +248,11 @@ @include includes/fun-elephant-controller-abort-transaction.texinfo @include includes/fun-elephant-controller-commit-transaction.texinfo - at node Migration and Upgrading + at node Migration and Upgrading API @comment node-name, next, previous, up - at section Migration and Upgrading + at section Migration and Upgrading API @cindex Migration and Upgrading + at cindex API Upgrade is a call to Migrate with checks for compatability. The migrate methods are included here in case you wish to develop a more --- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/25 03:18:13 1.8 +++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/27 03:14:55 1.9 @@ -82,7 +82,9 @@ (unless *system (let ((appname (get-application-name)) (url (get-system-url))) - (setf *system* (make-instance 'system-object :name appname :url url)))) + (setf *system* (make-instance 'system-object + :name appname + :url url)))) *system*) (sys-object) @@ -102,7 +104,8 @@ (defclass system-object () ((appname :accessor system-appname :initarg :name) (url :accessor system-url :initarg :url) - (laststate :accessor system-laststate :initarg :state :initform 'idle) + (laststate :accessor system-laststate :initarg :state + :initform 'idle) (state :accessor system-state :initarg :state :transient t) (:metaclass persistent-metaclass)) @@ -163,7 +166,8 @@ @lisp (defclass phash () - ((btree :accessor phash-btree :initarg :btree :initform (make-btree)))) + ((btree :accessor phash-btree :initarg :btree + :initform (make-btree)))) (defun make-persistent-hash (name) (let ((btree (get-from-root name))) @@ -423,7 +427,8 @@ :transient t) (touched :accessor snapshot-set-touched :initform (make-array 20 :element-type 'fixnum - :initial-element 0 :fill-pointer t :adjustable t) + :initial-element 0 :fill-pointer t + :adjustable t) :transient t)) (:documentation "Keeps track of a set of standard objects allowing a single snapshot call to update the store @@ -455,7 +460,8 @@ @lisp (defmethod register-object ((object standard-object) (set snapshot-set)) - "Register a standard object. Not recorded until snapshot is called on db" + "Register a standard object. Not recorded until + the snapshot function is called on db" (aif (lookup-cached-id object set) (values object it) (let ((id (incf (snapshot-set-next-id set)))) @@ -506,8 +512,11 @@ (defmethod snapshot ((set snapshot-set)) "Saves all objects in the set (and any objects reachable from the current set of objects) to the persistent store" - (with-transaction (:store-controller (get-con (snapshot-set-index set))) - (loop for (obj . id) in (get-cache-entries (snapshot-set-cache set)) do + (with-transaction (:store-controller (get-con + (snapshot-set-index set))) + (loop for (obj . id) in + (get-cache-entries (snapshot-set-cache set)) + do (save-snapshot-object id obj set)) (collect-untouched set))) @@ -581,11 +590,12 @@ @lisp (defmethod restore ((set snapshot-set)) - "Restores a snapshot by setting the snapshot-set state to the last snapshot. - If this is used during runtime, the user needs to drop all references - to objects and retrieve again from the snapshot set. Also used to initialize - the set state when a set is created, for example pulled from the root of a - store-controller, unless :lazy-load is specified" + "Restores a snapshot by setting the snapshot-set state to the last +snapshot. If this is used during runtime, the user needs to drop all +references to objects and retrieve again from the snapshot set. Also +used to initialize the set state when a set is created, for example +pulled from the root of a store-controller, unless :lazy-load is +specified" (clear-cache set) (map-btree (lambda (id object) (load-snapshot-object id object set)) @@ -597,7 +607,8 @@ (load-proxy-object id object set)) ((hash-table-p object) (load-proxy-hash id object set)) - (t (error "Unrecognized type ~A for id ~A in set ~A" (type-of object) id set))))) + (t (error "Unrecognized type ~A for id ~A in set ~A" + (type-of object) id set))))) @end lisp If an object has a reference object in a slot, then we simply restore @@ -700,7 +711,8 @@ hash table indirections with a little macro: @lisp -(defmacro def-snapshot-reference-wrapper (accessor-name (source-classname target-classname hashname uid)) +(defmacro def-snapshot-wrapper + (accessor-name (source-classname target-classname hashname uid)) (with-gensysms (obj key ref) `(progn (defmethod ,accessorname :around ((,obj ,source-classname)) @@ -723,7 +735,8 @@ ((pointer-to-island2 :accessor child :initform nil) (pointer-to-island1 :accessor neighbor :initform nil))) -(def-snapshot-reference-wrapper neighbor (island2 island1 *island1-hash* *unique-id*)) +(def-snapshot-wrapper neighbor + (island2 island1 *island1-hash* *unique-id*)) @end lisp Of course this doesn't work for multi-threaded environments, or for @@ -815,7 +828,7 @@ relationships such as ``PartOf'', ``DesireOf'' and ``EffectOf'' between English phrases. -Elephant's persistence capabilities is used to keep full records of +Elephant's persistence capability is used to keep full records of all source material, extracted relationships and search queries so that it is always possible to trace the source of a learned relation and to avoid repeated queries to web search engines. Conceptminer @@ -828,13 +841,14 @@ @itemize @item Bulk storage of post-processed web data: Elephant was used to -store hundreds of thousands of processed web pages as strings and index -the queries and resulting metadata. - at item Derived index: a custom string hash function was used to populate -a class derived index, allowing fast identification of pages from their URL. +store hundreds of thousands of processed web pages as strings, associate +pages with queries and store related metadata. + at item Derived index: a custom string hash function over URLs was used to populate +a class derived index, allowing fast identification of pages from +their URL without requiring expensive eql comparisons. @item Inverted document index: a (not terribly efficient) data structure that efficiently maps words to documents allowing pages to indexed by -the words contained in them. Allows for eash phrase and conjunction searches. +the words contained in them. Allowed for phrase and conjunction searches. @item User association data structure: a data structure based on oids that supports general one-to-many mappings between classes. Had a custom migrate method to support migration of associations. Supplanted by persistent @@ -845,13 +859,13 @@ architecture to cover in-memory lisp operations. @code{PCOMP} (Process Components) is a framework for constructing and managing simple, dataflow-style multi-threaded applications in Common Lisp. The goal -is to simplify the process sufficiently so that the ordinary user does -not hurt themselves easily. To this end, the model provides for safe, -asynchronous communications among a set of components which may be -scheduled together in a single process or communicate across separate -threads (and potentially processes). Components are packaged into a -system inside a Container object which schedules execution and -mediates communications. +is to simplify the process sufficiently so that the ordinary user can +hide from many of the details associated with aborting transactions. +To this end, the model provides for safe, asynchronous communications +among a set of components which may be scheduled together in a single +process or communicate across separate threads (and potentially +processes). Components are packaged into a system inside a Container +object which schedules execution and mediates communications. Communications between components can be in a dataflow style or using messages. Each component has a single port for receiving incoming data items. These items, if access --- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/25 02:27:56 1.18 +++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/27 03:14:55 1.19 @@ -63,19 +63,22 @@ @item @strong{Store controller:} the interface between lisp and a data store. Most operations require or accept a store controller, or a default store controller stored in @code{*store-controller*} to function. + at item @strong{Persistent Sets:} A simple persistent collection is provided +which allows the creation of persistent sets. @item @strong{BTrees:} Elephant provides a persistent key-value abstraction based on the BTree data structure. Values can be written to or read from a BTree and are stored in a sorted order. - at item @strong{Values:} most lisp values, including standard objects, arrays, etc + at item @strong{Stored values:} most lisp values, including standard objects, arrays, etc can be used as either key or value in a persistent BTree. @item @strong{Persistent objects:} An object where most slot values are stored in the data store and are written to or retrieved from disk on slot -accesses. Any value that can be written to an index can be written to -a persistent slot. - at item @strong{Transactions:} a dynamic context for executing operations on persistent -objects or BTrees that ensures that a set of changes is made atomically. - at item @strong{BTree indices:} A BTree index is a BTree that stores -an alternative ordering of the elements in a reference BTree. +accesses. Storing a persistent object stores only a reference, +allowing for object identity. + at item @strong{Object indexing:} The ability to lookup and sort objects by their slot +values rather than by explicit inclusion in a collection. + at item @strong{Transactions:} a dynamic context for executing operations on objects +or collections such that the side effects exhibit the ACID (atomicity, consistency, +isolation and durability) properties of database. @end itemize There are a set of more advanced concepts you will learn about later, @@ -112,7 +115,8 @@ @lisp (open-store '(:CLSQL (:SQLITE "/users/me/db/sqlite.db"))) -(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" "mydb" "myuser" "")))) +(open-store '(:CLSQL (:POSTGRESQL "localhost.localdomain" + "mydb" "myuser" "")))) @end lisp We use Berkeley DB as our example backend. To open a BDB @@ -197,7 +201,7 @@ => # @end lisp -It is an instance of a class "btree"; @pxref{Using BTrees}. +It is an instance of a class "btree"; @pxref{Persistent BTrees}. @node Serialization @comment node-name, next, previous, up @@ -255,7 +259,7 @@ This will affect all aggregate types: objects, conses, hash-tables, et cetera. (You can of course manually re-store the cons.) In this sense elephant does not automatically provide persistent collections. If you -want to persist every access, you have to use BTrees (@pxref{Using BTrees}). +want to persist every access, you have to use BTrees (@pxref{Persistent BTrees}). @item @strong{Serialization and deserialization can be costly}. While serialization is pretty fast, but it is still expensive to store large @@ -504,7 +508,8 @@ @lisp (defpclass person () ((name :accessor person-name :initarg :name)) - ((friends :accessor person-friends :initarg :friends :initform (make-pset)))) + ((friends :accessor person-friends :initarg :friends + :initform (make-pset)))) (defmethod add-friend ((me person) (them person)) (insert-item them (person-friends me))) @@ -522,7 +527,8 @@ @lisp (defpclass person () ((name :accessor person-name :initarg :name)) - ((friends :accessor person-friends-set :initarg :friends :initform (make-pset)))) + ((friends :accessor person-friends-set :initarg :friends + :initform (make-pset)))) (defmethod person-friends ((me person)) (pset-list (person-friends-set me))) @@ -609,7 +615,8 @@ @lisp (map-btree (lambda (k v) (format t "name: ~A utime: ~A~%" k - (subseq (multiple-value-list (decode-universal-time v)) 3 6))) + (subseq (multiple-value-list + (decode-universal-time v)) 3 6))) *friends-birthdays*) "Andrew" "Ben" @@ -675,11 +682,15 @@ @lisp (defun print-friend (friend) - (format t " name: ~A birthdate: ~A~%" (name friend) (birthday friend))) + (format t " name: ~A birthdate: ~A~%" + (name friend) (birthday friend))) -(make-instance 'friend :name "Carlos" :birthday (encode-date '(1 1 1972))) -(make-instance 'friend :name "Adriana" :birthday (encode-date '(24 4 1980))) -(make-instance 'friend :name "Zaid" :birthday (encode-date '(14 8 1976))) +(make-instance 'friend :name "Carlos" + :birthday (encode-date '(1 1 1972))) +(make-instance 'friend :name "Adriana" + :birthday (encode-date '(24 4 1980))) +(make-instance 'friend :name "Zaid" + :birthday (encode-date '(14 8 1976))) (get-instances-by-class 'friends) => (# # #) @@ -782,7 +793,8 @@ name: Carlos birthdate: (1 1 1972) => NIL -(map-class-index #'print-friend 'friend 'name :start "Adam" :end "Devin") +(map-class-index #'print-friend 'friend 'name + :start "Adam" :end "Devin") name: Adriana birthdate: (24 4 1980) name: Carlos birthdate: (1 1 1972) => NIL @@ -817,7 +829,7 @@ specification differs. This same facility is also available for your own use. For more -information @pxref{Secondary Indices}. +information @pxref{BTree Indexing}. @node Using Transactions @@ -1195,12 +1207,12 @@ and iterating of duplicate or unique values. @item @strong{Using the Map Operators} Mapping operators can be very efficient if properly utilized. - at item @strong{Using Multiple Stores} - Multiple store controllers can be open simultaneously. However it -does make the code more complex and you need to be careful about how -you use them to avoid crashes and other unpleasant side effects. - at item @strong{Custom Transaction Architecture} - You can implement your own version of with-transaction using the + at item @strong{Using Multiple Stores} Multiple store controllers can +be open simultaneously. However it does make the code more complex +and you need to be careful about how you use them to avoid crashes +and other unpleasant side effects. + at item @strong{Custom Transaction Architecture} You can implement your +own version of @code{with-transaction} using the underlying controller methods for starting, aborting and committing transactions. You had better know what you are doing, however! @item @strong{Handling Errors and Conditions} --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/26 03:10:13 1.18 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/27 03:14:55 1.19 @@ -10,8 +10,8 @@ * Serialization details:: The devil hides in the details. * Persistent Classes and Objects:: All the dirt on persistent objects. * Class Indices:: In-depth discussion about indexing objects. -* Persistent Sets:: Using the persistent set. -* Persistent BTrees:: Using the native btree. +* Persistent Sets:: Using the persistent set collection. +* Persistent BTrees:: Using the native btree collection. * BTree Cursors:: Low-level access to BTrees. * BTree Indexing:: Alternative ways to reference objects in btrees. * Index Cursors:: Low-level access to BTree indices. @@ -172,9 +172,10 @@ @end lisp This will affect all aggregate types: objects, conses, hash-tables, et -cetera. (You can of course manually re-store the cons.) In this sense -elephant does not automatically provide persistent collections. If you -want to persist every access, you have to use BTrees (@pxref{Using BTrees}). +cetera. (You can of course manually re-store the cons.) In this +sense elephant does not automatically provide persistent collections. +If you want to persist every access, you have to use Persistent Sets +(@pxref{Persistent Sets}) or BTrees (@pxref{Persistent BTrees}). @item @strong{Serialization and deserialization can be costly}. While serialization is pretty fast, but it is still expensive to store large @@ -382,8 +383,10 @@ @lisp (defpclass my-pclass () ((pslot1 :accessor pslot1 :initarg :pslot1 :initform 'one) - (pslot2 :accessor pslot2 :initarg :pslot2 :initform 'two :persistent t) - (tslot1 :accessor tslot1 :initarg :tslot1 :initform 'three :transient t))) + (pslot2 :accessor pslot2 :initarg :pslot2 :initform 'two + :persistent t) + (tslot1 :accessor tslot1 :initarg :tslot1 :initform 'three + :transient t))) @end lisp In the definition above the class @code{my-pclass} is an instance of @@ -706,8 +709,9 @@ Because the instance is guaranteed to be resident, the operation has none of the resident/non-resident conflicts above. -Changing a persistent instance to a non-persistent class is not -allowed and will result in an error. +Change class cannot convert between persistent and non-persistent classes and will +flag an error if you try to do so. @emph{(Note: this could be implemented in the +future if users request it)} @node Class Indices @comment node-name, next, previous, up @@ -860,7 +864,6 @@ database for reuse. The oid used by the btree, however, will not be recovered. - at c *** FINISH *** @node BTree Cursors @comment node-name, next, previous, up @section BTree Cursors @@ -869,27 +872,139 @@ database, you can also traverse the BTree structure one key-value pair at a time. -Creating cursors (inside transactions) -Duplicating cursors +Cursors must be created in the context of an active transaction +(i.e. a @code{with/ensure-transaction} body). A cursor is made +through a call to the @code{make-cursor} method of the BTree you wish +to traverse. + +An existing cursors can also be duplicated within the same transaction +by calling @code{cursor-duplicate} which avoids the overhead of +setting a second cursor to the same location. -Initializing cursors to location -- first, last, set -Traversing -- next, prev, (set) -Side effects -- delete -- put -- Can also use setf get-value while traversing +Cursors can be in two states: initialized and uninitialized. + at ref{Cursor API} for details. -Large cursor operations (locks, resources?) +To initialize a cursor, you have to use one of the initializing +functions to select a key-value pair in the btree. + + at itemize + at item @code{cursor-first} and @code{cursor-last}: initialize the +cursor to the first and last element of the btree, respectively. + at item @code{cursor-set} and @code{cursor-set-range}: Sets the cursor +to the first key-pair values according to the specified key. If the +set fails, the cursor will remain uninitialized. The ranged set will +set it to the first key-value pair where the key is equal to or +greater than the key argument. + at end itemize + +A valid cursor will return multiple values: @code{(exists? key +value)}. The first argument tells whether or not the cursor is +initialized and pointing at a proper value. The second two arguments +are self-explanatory. + + at code{cursor-current} returns the current state of the cursor, nil if +it is uninitialized. + +Once a cursor is properly initialized, it can be incremented or +decremented, a simple constant-time operation on BTrees. + + at code{cursor-next} and @{cursor-prev} move the cursor a single step +forward or back across the sorted key-value pairs. @code{cursor-next} +moves in ascending order, @code{cursor-prev} in descending order. + +Finally cursors can be used for side effects on the current key-value +pair. The function @code{cursor-put} replaces the value (but does not +increment the current value) and @code{cursor-delete} deletes the +key-value pair and become uninitialized. It is a valid operation to +use the @code{(setf get-value)} method while the cursor is active to +change the value at the current cursor. + +If cursors take place within a transaction, what happens when +traversing a very large BTree? This depends on the data store policy +regarding whether a cursor read locks its entire btree (or the subset +that is being iterated over) or allows changes to any pairs its +transaction has not changed. See your data store documenation for +details. - at c *** FINISH *** @node BTree Indexing @comment node-name, next, previous, up @section BTree Indexing +One powerful feature of Elephant is the ability to add indexes to +BTrees. An indexed btree is a subclass of the standard @code{btree} +called @code{indexed-btree}. The indexed btree maintain a set of +indices (instances of @code{btree-index}) which provide alternative +ways of indexing into the values of the main btree. + +Each index is itself a btree, but with the property that its values +are matched to the keys of the main btree. That is if you have a btree +with key-value pairs: - at c *** FINISH *** + at lisp +("henry" . #) +("larry" . #) + at end lisp + +You can define an index that is populated by the age of the person object: + + at lisp +(29 . "Larry") +(45 . "Henry") + at end lisp + +Now when you call @code{(get-value 29 index)} you get back + at code{#}! Note also that these new pairs are +ordered by age, the opposite of the alphabetic ordering of the names +in the first two pairs. If you read through the tutorial, you may +have guessed by now that this is the mechanism used to implement the +class indexing capabilities previously described. + +An index is created by using the @code{add-index} function. This +function takes the @code{indexed-btree} you wish to index, an +index-name for later retrieval, a key-form which dictates how +the index populates it's keys as a function of the main btree's +keys and values. + +A simple, contrived example is shown in the figure below: + + at center @image{IndexedBtrees1,,4.0in,[BTree Index Diagram],.png} + +Here we have a primary, indexed btree with a set of keys and values +represented by symbols. We'll declare the function @code{val} to take +a value symbol and extract it's number. The key-form in the + at code{mod5 * 2} index is: + + at lisp +(lambda (k v) + (if (= 0 (mod (val v) 5)) + (values t (* 2 (val v))) + (values nil nil))) + at end lisp + +When a key-value pair is written to the primary btree, the index is +automatically updated through a call to the key-form. If the key-form +above is called with @code{key1} and @code{value1}, @code{val} will +return 1 which fails the if test. The second values statement, + at code{(values nil nil)} indicates that this pair is not to be indexed. +If I pass @code{key5} and @code{value5} to this same key form, I get +back 10 as the @code{(val 'value5)} is 5 and @code{(= 0 (mod 5 5))} so +the form returns @code{(values t 10)} meaning the index should add an +index entry of 10 (@code{(* 2 5)}) associated with the key value + at code{key5}. + +So, of course, making the call @code{(get-value 10 index-mod5)} will +return @code{value5}. + +The second index in our little example calculates the number of bits +in all odd numbered values. This illustrates an important property of +the @code{btree-index}: it allows duplicate keys. Standard + at code{btree} and @code{indexed-btree} classes are not allowed to have +duplicate elements. The odd index allows us to ask simple questions +like: ``what are all the odd values with ids that fit into 4 bits?''. + +To extract this set, we have to use cursor functions specifically +designed for the index that iterate over duplicate values. + @node Index Cursors @comment node-name, next, previous, up @section Index Cursors @@ -897,19 +1012,29 @@ Index cursors are just like BTree cursors except you can get the main BTree value instead of the index value. There are also a parallel set of operations such as @code{cursor-pnext} instead of - at code{cursor-next} which returns @code{exists}, @code{key}, @code{primary-btree-value} -and @code{index-value = primary-btree-key}. - -Operations that have the same behavior, but return primary btree values and keys are: + at code{cursor-next} which returns @code{exists}, @code{key}, + at code{primary-btree-value} and @code{index-value = primary-btree-key}. - at c need table here +Operations that have the same behavior, but return primary btree +values and keys are: - at itemize - at item cursor-first -> cursor-pfirst, cursor-last -> cursor-plast - at item cursor-current -> cursor-pcurrent - at item cursor-next -> cursor-pnext, cursor-prev -> cursor-pprev - at item cursor-set-range - at end itemize + at table @samp + at item @code{cursor-first} + at code{cursor-pfirst} + at samp{@code{cursor-first}} + at item @code{cursor-last } + at code{cursor-plast} + at item @code{cursor-current } + at code{cursor-pcurrent} + at item @code{cursor-next } + at code{cursor-pnext} + at item @code{cursor-prev} + at code{cursor-pprev} + at item @code{cursor-set} + at code{cursor-pset} + at item @code{cursor-set-range} + at code{cursor-set-prange} + at end table The big difference between btree cursors and index cursors is that indices can have duplicate key values. This means we have to choose @@ -1256,6 +1381,10 @@ @item @strong{@code{:txn-sync}} This is the default behavior and specifies that the transaction log of the current transaction is flushed to disk before the transaction commit routine returns. This provides full ACID compliance. + at item @strong{@code{:transaction}} This argument is for advanced use. It tells + the Berkeley DB transaction subsystem the transaction it should use rather +than to create a new one. The @code{:parent} argument provides a parent transaction +that can result in a true nested transaction. @end itemize @subsection Special Commands @@ -1357,7 +1486,9 @@ @comment node-name, next, previous, up @section Native Lisp Data Store -The native lisp data store is vaporware at this time. +The native lisp data store is unimplemented. It is tentatively +planned for a 1.1 release sometime in the distant future. Yes, +this is a deliberatively vague declaration. @c @node Querying persistent instances @c @comment node-name, next, previous, up From ieslick at common-lisp.net Sat Apr 28 17:18:33 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 13:18:33 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070428171833.52E69710FB@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv32598 Modified Files: ele-bdb.asd Log Message: test conditions fix, new run-regression test interface; remove doc strings to cleanup manual --- /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/04/19 05:24:37 1.26 +++ /project/elephant/cvsroot/elephant/ele-bdb.asd 2007/04/28 17:18:32 1.27 @@ -18,6 +18,11 @@ (in-package :cl-user) +;; Forward def +(defpackage elephant-system + (:use :cl :asdf) + (:export :elephant-c-source :compiler-options :foreign-libraries-to-load-first :get-config-option)) + (defpackage ele-bdb-system (:use :cl :asdf :elephant-system)) @@ -40,6 +45,9 @@ (setf result (concatenate 'string "/cygdrive" result)) (substitute #\/ #\\ result))) +;; Forward def +(defclass elephant-c-source (c-source-file) ()) + (defclass bdb-c-source (elephant-c-source) ()) (defmethod compiler-options ((compiler (eql :gcc)) (c bdb-c-source) &key &allow-other-keys) From ieslick at common-lisp.net Sat Apr 28 17:18:33 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 13:18:33 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070428171833.BAC9471123@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv32598/src/elephant Modified Files: collections.lisp Log Message: test conditions fix, new run-regression test interface; remove doc strings to cleanup manual --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 02:31:15 1.28 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 17:18:33 1.29 @@ -340,8 +340,8 @@ "Move to the previous duplicate element (with the same key.) Returns has-pair key value.")) +;; Default implementation. Plan is to update both backends when BDB 4.6 comes out (defmethod cursor-prev-dup ((cur cursor)) - "Default implementation. Plan is to update both backends when BDB 4.6 comes out" (when (cursor-initialized-p cur) (multiple-value-bind (exists? skey-cur) (cursor-current cur) @@ -362,8 +362,8 @@ "Move to the previous duplicate element (with the same key.) Returns has-tuple / secondary key / value / primary key.")) +;; Default implementation. Plan is to update both backends when BDB 4.6 comes out (defmethod cursor-pprev-dup ((cur cursor)) - "Default implementation. Plan is to update both backends when BDB 4.6 comes out" (when (cursor-initialized-p cur) (multiple-value-bind (exists? skey-cur) (cursor-current cur) From ieslick at common-lisp.net Sat Apr 28 17:18:34 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 13:18:34 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070428171834.3F669710FB@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv32598/tests Modified Files: elephant-tests.lisp testconditions.lisp Log Message: test conditions fix, new run-regression test interface; remove doc strings to cleanup manual --- /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/04/28 02:31:31 1.32 +++ /project/elephant/cvsroot/elephant/tests/elephant-tests.lisp 2007/04/28 17:18:34 1.33 @@ -86,6 +86,9 @@ (defvar *testpg-spec* '(:clsql (:postgresql "localhost.localdomain" "test" "postgres" ""))) +(defvar *testpg-spec2* + '(:clsql (:postgresql "localhost.localdomain" "test2" "postgres" ""))) + (defvar *testsqlite3-spec* `(:clsql (:sqlite3 ,(namestring @@ -116,14 +119,36 @@ ;; -;; GUIDE TO TESTING +;; COMPREHENSIVE TESTING +;; + +(defun run-regression-tests (type) + (let (sc1 sc2 oid) + (case type + (:BDB + (setf sc1 *testbdb-spec* + sc2 *testbdb-spec2* + oid *testbdb-spec-oid*)) + (:SQLITE + (setf sc1 *testsqlite3-spec* + sc2 *testsqlite3-spec2*)) + (:POSTGRES + (setf sc1 *testpg-spec* + sc2 *testpg-spec*)) + (t (error "Unrecognized data store type: ~A" type))) + (let ((*test-spec-primary* sc1) + (*test-spec-secondary* sc2)) + (declare (special *test-spec-secondary* *test-spec-primary*)) + (do-backend-tests sc1)))) + +;; +;; GUIDE TO SUBTESTS ;; -;; 1) Set *default-spec* to the above spec of your choice -;; 2) Call (do-backend-tests) to test the standard API -;; 3) To test migration: (do-migration-tests *default-spec* ) inserting a second +;; 1) Call (do-backend-tests ) to test the standard API +;; 3) To test migration: (do-migration-tests ) inserting a second ;; spec, typically a bdb spec or create another instance of a sql db depending on ;; your configuration -;; 4) A backend is green if it passes do-backend-tests and can succesfully be +;; 4) A data store is green if it passes do-backend-tests and can succesfully be ;; used as spec1 or spec2 argument in the migration test ;; @@ -146,8 +171,7 @@ "For easy interactive running of single tests while debugging" (when spec (with-open-store (spec) - (let ((*auto-commit* nil)) - (do-test testname))))) + (do-test testname)))) (defun do-migration-tests (spec1 spec2 &optional oid-spec) "Interface to do explicit migration tests between backends" --- /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 03:07:39 1.4 +++ /project/elephant/cvsroot/elephant/tests/testconditions.lisp 2007/04/28 17:18:34 1.5 @@ -19,9 +19,9 @@ (progn (format t "~%Second store spec missing: ignoring") (values t t t t)) - (let (*store-controller* - (sc1 *store-controller*) - (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil))) + (let ((sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect nil)) + (*store-controller* *store-controller*) + (sc1 *store-controller*)) (unwind-protect (let ((inst1 (make-instance 'pfoo :slot1 100 :sc sc1)) (inst2 (make-instance 'pfoo :slot1 200 :sc sc2))) @@ -36,10 +36,7 @@ t t t t) (deftest unindexed-class-condition - (let* ((sc (open-store *test-spec-primary* :recover t :deadlock-detect nil)) - (inst (make-instance 'pfoo :slot1 1 :sc sc))) - (unwind-protect - (signals-specific-condition (persistent-class-not-indexed) - (find-class-index 'pfoo :sc sc)) - (close-store sc))) + (let ((inst (make-instance 'pfoo :slot1 1))) + (signals-specific-condition (persistent-class-not-indexed) + (find-class-index 'pfoo))) t) \ No newline at end of file From ieslick at common-lisp.net Sun Apr 29 02:10:54 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 22:10:54 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070429021054.D3D8D20031@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv12328 Modified Files: elephant.asd Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/21 17:22:35 1.45 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/29 02:10:54 1.46 @@ -305,7 +305,7 @@ (:file "serializer2") ;; 0.6.1 db's (:file "unicode2") (:file "migrate") - (:file "query") +;; (:file "query") (:file "data-store-api")) :serial t :depends-on (memutil utils))))) From ieslick at common-lisp.net Sun Apr 29 02:10:55 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 22:10:55 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070429021055.9B1212102F@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv12328/src/elephant Modified Files: classes.lisp classindex.lisp collections.lisp unicode2.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/27 13:32:16 1.33 +++ /project/elephant/cvsroot/elephant/src/elephant/classes.lisp 2007/04/29 02:10:54 1.34 @@ -273,18 +273,21 @@ "Ensures that object can be written as a reference into store sc" (eq (dbcn-spc-pst object) (controller-spec sc))) -(define-condition cross-reference-error () +(define-condition cross-reference-error (error) ((object :accessor cross-reference-error-object :initarg :object) (home-controller :accessor cross-reference-error-home-controller :initarg :home-ctrl) (foreign-controller :accessor cross-reference-error-foreign-controller :initarg :foreign-ctrl)) (:documentation "An error condition raised when an object is being written into a data store other - than its home store")) + than its home store") + (:report (lambda (condition stream) + (format stream "Attempted to write object ~A with home store ~A into store ~A" + (cross-reference-error-object condition) + (cross-reference-error-home-controller condition) + (cross-reference-error-foreign-controller condition))))) (defun signal-cross-reference-error (object sc) (cerror "Proceed to write incorrect reference" 'cross-reference-error - :format-control "Attempted to write object ~A with home store ~A into store ~A" - :format-arguments (list object (get-con object) sc) :object object :home-ctrl (get-con object) :foreign-ctrl sc)) --- /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/28 03:07:38 1.41 +++ /project/elephant/cvsroot/elephant/src/elephant/classindex.lisp 2007/04/29 02:10:54 1.42 @@ -84,18 +84,17 @@ (let ((class (find-class class-name nil))) (when class (indexed class)))) -(define-condition persistent-class-not-indexed (error) - ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj))) - +(define-condition persistent-class-not-indexed () + ((class-obj :initarg :class :initarg nil :reader unindexed-class-obj)) + (:report (lambda (condition stream) + (format stream "Class ~A is not enabled for indexing" + (class-name (unindexed-class-obj condition)))))) + (defun signal-class-not-indexed (class) (cerror "Ignore and continue?" 'persistent-class-not-indexed - :format-control "Class ~A is not enabled for indexing" - :format-arguments (list (class-name class)) :class class)) -;; (define-condition - (defmethod find-class-index ((class persistent-metaclass) &key (sc *store-controller*) (errorp t)) (ensure-finalized class) (if (not (indexed class)) --- /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/28 17:18:33 1.29 +++ /project/elephant/cvsroot/elephant/src/elephant/collections.lisp 2007/04/29 02:10:54 1.30 @@ -74,13 +74,6 @@ (:documentation "Delete all key-value pairs from the btree and render it an invalid object in the data store")) -(defmethod drop-btree ((bt btree)) - (ensure-transaction (:store-controller *store-controller*) - (with-btree-cursor (cur bt) - (loop for (exists? key) = (multiple-value-list (cursor-first cur)) - then (multiple-value-list (cursor-next cur)) - while exists? - do (remove-kv key bt))))) ;; ;; Btrees that support secondary indices @@ -380,7 +373,6 @@ different key.) Returns has-tuple / secondary key / value / primary key.")) - (defmacro with-btree-cursor ((var bt) &body body) "Macro which opens a named cursor on a BTree (primary or not), evaluates the forms, then closes the cursor." @@ -389,6 +381,14 @@ (progn , at body) (cursor-close ,var)))) +(defmethod drop-btree ((bt btree)) + (ensure-transaction (:store-controller *store-controller*) + (with-btree-cursor (cur bt) + (loop for (exists? key) = (multiple-value-list (cursor-first cur)) + then (multiple-value-list (cursor-next cur)) + while exists? + do (remove-kv key bt))))) + ;; ======================================= ;; Generic Mapping Functions ;; ======================================= --- /project/elephant/cvsroot/elephant/src/elephant/unicode2.lisp 2007/04/12 02:47:33 1.8 +++ /project/elephant/cvsroot/elephant/src/elephant/unicode2.lisp 2007/04/29 02:10:55 1.9 @@ -83,13 +83,13 @@ (let ((code (char-code (schar string i)))) (declare (type fixnum code)) (when (> code #xFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ i size)) code)))) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ i size)) code)))) (string (loop for i fixnum from 0 below characters do (let ((code (char-code (char string i)))) (declare (type fixnum code)) (when (> code #xFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ i size)) code))))) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ i size)) code))))) (setf (buffer-stream-size bstream) needed) (succeed)))))) @@ -120,20 +120,20 @@ (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #xFFFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size)) ;; (coerce (ldb (byte 8 8) code) '(signed 8))) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1)) ;; (coerce (ldb (byte 8 0) code) '(signed 8)))))) (ldb (byte 8 0) code))))) (string (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #xFFFF) (fail)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size)) ;; (coerce (ldb (byte 8 8) code) '(signed 8))) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 2) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 2) size 1)) ;; (coerce (ldb (byte 8 0) code) '(signed 8))))))) (ldb (byte 8 0) code)))))) (incf size (* characters 2)) @@ -158,25 +158,25 @@ (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #x10FFFF) (error "Invalid unicode code type")) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 0)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 0)) (ldb (byte 8 24) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 1)) (ldb (byte 8 16) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 2)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 2)) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 3)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 3)) (ldb (byte 8 0) code))))) (string (loop for i fixnum from 0 below characters do (let ((code (char-code (schar string i)))) (when (> code #x10FFFF) (error "Invalid unicode code type")) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 0)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 0)) (ldb (byte 8 24) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 1)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 1)) (ldb (byte 8 16) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 2)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 2)) (ldb (byte 8 8) code)) - (setf (uffi:deref-array buffer 'array-or-pointer-char (+ (* i 4) size 3)) + (setf (uffi:deref-array buffer '(:array :unsigned-char) (+ (* i 4) size 3)) (ldb (byte 8 0) code)))))) (incf size (* characters 4)) t)))) From ieslick at common-lisp.net Sun Apr 29 02:10:56 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 22:10:56 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/memutil Message-ID: <20070429021056.1D4FF26081@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/memutil In directory clnet:/tmp/cvs-serv12328/src/memutil Modified Files: memutil.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/src/memutil/memutil.lisp 2007/03/21 14:29:31 1.27 +++ /project/elephant/cvsroot/elephant/src/memutil/memutil.lisp 2007/04/29 02:10:55 1.28 @@ -471,12 +471,12 @@ (simple-string (loop for i fixnum from 0 below length do - (setf (deref-array dest 'array-or-pointer-char (+ i dest-offset)) + (setf (deref-array dest '(:array :unsigned-char) (+ i dest-offset)) (char-code (schar src (+ i src-offset)))))) (string (loop for i fixnum from 0 below length do - (setf (deref-array dest 'array-or-pointer-char (+ i dest-offset)) + (setf (deref-array dest '(:array :unsigned-char) (+ i dest-offset)) (char-code (char src (+ i src-offset)))))))) From ieslick at common-lisp.net Sun Apr 29 02:11:02 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 22:11:02 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/utils Message-ID: <20070429021102.AACB933083@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/utils In directory clnet:/tmp/cvs-serv12328/src/utils Modified Files: os.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/12 02:47:51 1.3 +++ /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 02:10:56 1.4 @@ -53,7 +53,7 @@ (ccl:run-program program args :wait nil)) #+lispworks (apply #'sys::call-system - (format nil "~a~{ '~a'~}~@[ &~]" prog args) + (format nil "~a~{ '~a'~} &" program args) :current-directory directory :wait nil) ) From ieslick at common-lisp.net Sun Apr 29 02:11:04 2007 From: ieslick at common-lisp.net (ieslick) Date: Sat, 28 Apr 2007 22:11:04 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/tests Message-ID: <20070429021104.5DF0634055@common-lisp.net> Update of /project/elephant/cvsroot/elephant/tests In directory clnet:/tmp/cvs-serv12328/tests Modified Files: testmigration.lisp Log Message: Fixes for lispworks in tests & deadlock detect; fixes for openmcl in pointer manipulation --- /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/03/24 10:51:45 1.20 +++ /project/elephant/cvsroot/elephant/tests/testmigration.lisp 2007/04/29 02:11:02 1.21 @@ -34,7 +34,7 @@ (progn (format t "~%Single store mode: ignoring") t) - (let* ((*store-controller*) + (let* (*store-controller* (sc1 (open-store *test-spec-primary* :recover t :deadlock-detect t)) (sc2 (open-store *test-spec-secondary* :recover t :deadlock-detect t))) (unwind-protect From ieslick at common-lisp.net Sun Apr 29 04:03:29 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 00:03:29 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-clsql Message-ID: <20070429040329.4363048144@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv6425/src/db-clsql Modified Files: sql-collections.lisp Log Message: Fixed fencepost error in cursor-prev-x in clsql data store --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/12 02:47:24 1.13 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:28 1.14 @@ -518,18 +518,19 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) - (declare (optimize (speed 3))) +;; (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (progn - (let ((cur-pk (get-current-key cursor))) + (let ((prior-pk (get-current-key cursor))) (decf (sql-crsr-ck cursor)) - (if (equal cur-pk (get-current-key cursor)) + (format t "prior-key: ~A current-key: ~A%" prior-pk (get-current-key cursor)) + (if (eq prior-pk (get-current-key cursor)) (setf (dp-nmbr cursor) (max 0 (- (dp-nmbr cursor) 1))) - (setf (dp-nmbr cursor) - (sql-get-from-clcn-cnt (cursor-oid cursor) - (get-current-key cursor) - (get-con (cursor-btree cursor)) - )))) + (setf (dp-nmbr cursor) + (1- (sql-get-from-clcn-cnt (cursor-oid cursor) + (get-current-key cursor) + (get-con (cursor-btree cursor)))) + ))) (has-key-value-scnd cursor :returnpk returnpk)) (cursor-last-x cursor :returnpk returnpk))) From ieslick at common-lisp.net Sun Apr 29 04:03:54 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 00:03:54 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/db-clsql Message-ID: <20070429040354.6A2744904E@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/db-clsql In directory clnet:/tmp/cvs-serv6510/src/db-clsql Modified Files: sql-collections.lisp Log Message: Oops, cleanup debug stmts --- /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:28 1.14 +++ /project/elephant/cvsroot/elephant/src/db-clsql/sql-collections.lisp 2007/04/29 04:03:54 1.15 @@ -518,12 +518,11 @@ (cursor-prev-x cursor) ) (defmethod cursor-prev-x ((cursor sql-secondary-cursor) &key (returnpk nil)) -;; (declare (optimize (speed 3))) + (declare (optimize (speed 3))) (if (cursor-initialized-p cursor) (progn (let ((prior-pk (get-current-key cursor))) (decf (sql-crsr-ck cursor)) - (format t "prior-key: ~A current-key: ~A%" prior-pk (get-current-key cursor)) (if (eq prior-pk (get-current-key cursor)) (setf (dp-nmbr cursor) (max 0 (- (dp-nmbr cursor) 1))) (setf (dp-nmbr cursor) From ieslick at common-lisp.net Sun Apr 29 21:54:51 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 17:54:51 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070429215451.A1D8934052@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv3086/doc Modified Files: installation.texinfo Log Message: Fix for lispworks deadlock detect (silently fails instead of asserting an error) --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 14:24:57 1.13 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/29 21:54:51 1.14 @@ -36,8 +36,10 @@ Elephant supports SBCL, Allegro, Lispworks, OpenMCL and CMUCL. Each lisp is supported on each of the platforms it runs on: Mac OS X, Linux and Windows. As of release 0.6.1, both 32-bit and 64-bit systems -should be supported. Elephant has a small developer base and as of -the writing of this manual, there are: +should be supported. + +Due to the small number of developers and the large number of configurations +providing full test coverage is problematic. There are: @enumerate @item Five lisp environments @@ -46,16 +48,14 @@ @item Three data store configurations: Berkeley DB, SQLite3 and Postgresql @end enumerate -This means that the total number of combinations that should be tested -comes to: +which means that the total number of combinations to be tested could be as +much as: @math{lisps * os * radix * dstore = 5 * 3 * 2 * 3 = 90 configurations} -Of course not all of these combinations are valid, but the -implications of these combinatorics is that not every combination will -be tested in any given release. The developers and active user base -currently cover all three data store configurations on the following -platforms: +Not all of these combinations are valid, but the implication is that +not every combination will be tested in any given release. The +developers and user base regularly use the following platforms @itemize @item 32/64-bit SBCL on Linux and Mac OS X @@ -63,13 +63,15 @@ @item 32-bit Allegro on Mac OS X @end itemize -The developers will do their best to accomodate users who are keen to -test other combinations, but practically these configurations will be -the most stable and reliable. Elephant is becoming quite stable in -general, so don't be afriad to try an unemphasized combination - -chances are it is just a little more work to bring it up. In particular, -Elephant can probably work with MySQL or Oracle with just a little work, -but nobody has asked for this yet. +The CLSQL backend is used predominantly under SBCL on Linux and Mac OS +X at the time of writing. The developers will do their best to +accomodate users who are keen to test other combinations, but the +above configurations will be the most stable and reliable. + +Elephant is now quite stable in general, so don't be afraid to try an +unemphasized combination - chances are it is just a little more work +to bring it up. In particular, Elephant can probably work with MySQL +or Oracle with just a little work, but nobody has asked for this yet. @subsection Library dependencies @@ -77,7 +79,7 @@ @enumerate @item asdf -- @uref{http://www.cliki.net/asdf} - at item uffi -- version 1.5.17 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} + at item uffi -- requires version 1.5.18 or later, @uref{http://uffi.b9.com/} or @uref{http://www.cliki.net/UFFI} @item cl-base64 -- @uref{http://www.cliki.net/cl-base64} @item gcc -- Your system needs GCC (or Cygwin) to build the Elephant C-based serializer library. (Precompiled DLL's are available for Windows platforms on the @uref{http://www.common-lisp.net/project/elephant/downloads.html, download page}. @item rt -- The RT regression test sytem is required to run the test suite: @uref{http://www.cliki.net/RT} @@ -86,7 +88,8 @@ Follow the instructions at these URLs to download and setup the libraries. (Note: uffi and cl-base64 are @uref{http://www.cliki.net/ASDF-Install, asdf-installable} for those -of you with asdf-install on your system) +of you with asdf-install on your system). Elephant, however, is not +asdf-installable today. In addition to these libraries, each data store has their own dependencies as discussed in @ref{Berkeley DB} and @@ -97,10 +100,11 @@ @section Configuring Elephant Before you can load the elephant packages into your running lisp, you -need to setup the configuration file. First, copy the reference file -config.sexp from the root directory to my-config.sexp. my-config.sexp -contains a lisp reader-formatted list of key-value pairs that tells -elephant where to find various libraries, how to build libraries, etc. +need to setup the configuration file. Copy the reference file +config.sexp from the root directory to my-config.sexp in the root +directory. my-config.sexp contains a lisp reader-formatted list of +key-value pairs that tells elephant where to find various libraries +and how to build them. For example: @@ -110,8 +114,6 @@ (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) - (:clsql-lib . nil) (:compiler . :gcc)) @end lisp @@ -132,9 +134,9 @@ The config.sexp file contains a set of example configurations to start from, but you will most likely need to modify it for your system. -Elephant has one small C library that it uses for binary serialization -which means that you need to have gcc in your path (@pxref{Elephant on -Windows} for exceptions on the Windows platform). +Elephant has one small C library that it uses for binary +serialization. This means that you need to have gcc in your path +(@pxref{Elephant on Windows} for exceptions on the Windows platform). @node Loading Elephant @comment node-name, next, previous, up @@ -152,12 +154,11 @@ This will load the cl-base64 and uffi libraries. It will also automatically compile and load the C library. The build process no -longer depends on a Makefile. This build process has been verified on -most platforms, but if you have a problem please report it, and any -output you can capture, to the developers at - at email{elephant-devel@@common-lisp.net}. We will update the FAQ at - at uref{http://trac.common-lisp.net/elephant} with common problems users -run into. +longer depends on a Makefile and has been verified on most platforms, +but if you have a problem please report it, and any output you can +capture, to the developers at @email{elephant-devel@@common-lisp.net}. +We will update the FAQ at @uref{http://trac.common-lisp.net/elephant} +with common problems users run into. @subsection Two-Phase Load Process @@ -170,8 +171,7 @@ (NOTE: There are some good reasons and not so good reasons for this process. One reason you cannot load ele-bdb.asd directly as it depends on lisp code defined in elephant.asd. We decided not to fix -this in this release although later releases may avoid the oddity of -the two phase loading) +this in the 0.9 release although later releases may improve on this). @subsection Packages @@ -199,7 +199,8 @@ (:use :common-lisp :elephant)) @end lisp -The symbols that are imported are captured in @ref{User API Reference} +The imported symbols are all that is needed to control Elephant +databases and are documented in detail in @ref{User API Reference} @subsection Opening a Store @@ -224,19 +225,18 @@ The first time you load a specific data store, Elephant will call ASDF to load all the specified data store's dependencies, connect to a database and return the @code{store-controller} subclass instance for -that backend. +that data store. @node Berkeley DB @comment node-name, next, previous, up @section Berkeley DB -The Berkeley DB Data Store started out as a very simple data dictionary in the -Berkeley Unix operating system. There are many ``Xdb'' systems that use the -same API, or a similarly one. A commercial version of the BDB was provided by -Sleepycat Software, and this product has since been purchased by Oracle corporation, -and is currently distributed under a similar licensing arrangement, with commercial -support also available. Please follow the download and installation procedures -defined here: +The Berkeley DB Data Store started out as a very simple data +dictionary in the Berkeley Unix operating system. There are many +``Xdb'' systems that use the same API, or a similarly one. A free for +non-commercial use version of Berkeley DB is provided by Oracle +corporation with commercial licenses available. Please follow the +download and installation procedures defined here: http://www.oracle.com/technology/products/berkeley-db/db/index.html @@ -246,68 +246,71 @@ @comment node-name, next, previous, up @section Setting up Berkeley DB -Beyond ensuring that the file ``my-config.sexp'' points correctly -to your BDB installation, nothing else should be required to configure -the example that uses a local ``testdb'' directory as a dabase (under ``tests'') -in the top-level Elephant directory. +Beyond ensuring that the file ``my-config.sexp'' points to your BDB +installation directories and files, nothing else should be required to +configure the example that uses a local ``testdb'' directory as a +dabase (under ``tests'') in the top-level Elephant directory. On one Fedora based system, the ``my-config.sexp'' file looked like this: @lisp ((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include") - (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") - (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") - (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb.so") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") (:pthread-lib . nil) - (:clsql-lib . "/usr/local/share/common-lisp/") - (:compiler . :gcc)) + (:clsql-lib . "/usr/local/share/common-lisp/") + (:compiler . :gcc)) @end lisp The @ref{Test Suites} give a nice example of using BDB by running the test using the specification: @lisp -'(:BDB "/home/me/db/testdb/")) +'(:BDB "/tests/testdb/") @end lisp -Once you start working on an application, you will want to change this path -to a directory that make sense for you application, and use that as -the specification passed to - at lisp open-store - at end lisp -in your application. +Once you start working on an application, you will want to change the +path to a directory that is appropriate for your application, and use +that as the specification passed to @code{ open-store } on application +startup. @node Upgrading Berkeley DB Databases @comment node-name, next, previous, up @section Upgrading Berkeley DB Databases -At regular intervals, Elephant will require a specific version of Berkeley DB -because it does not parse the header files which tend to change with each release. -The patches are usually minor, but sometimes Elephant also depends on new -features of Berkeley DB. In this case, you have to update both the Berkeley DB -database files as well as any data Elephant has built within it. That creates -some special constraints for upgrading databases. +When there is a new release of Elephant, it will depend on a new +version of Berkeley DB. If so, you must upgrade your BDB databases to +use the new version Elephant. This forced upgrade is a consequence of +Elephant not parsing the BDB header files which tend to change various +important constants with each release. These patches are usually +minor. Upgrading also happens because Elephant tries to leverage new +features of Berkeley DB. + +The rest of this section talks about how to upgrade your existing +Berkeley DB databases, opening them in the new Elephant version and +migrating them to a newly created Elephant database. @subsection Upgrading to 0.9 -This section outlines how to upgrade from Elephant code base version 0.6.0 -using Berkeley DB 4.3. +This section outlines how to upgrade from Elephant version 0.6.0 and +Berkeley DB 4.3. @enumerate @item Install BDB 4.5 (keep 4.3 around for now) - at item Update my-config.sexp to point to the appropriate BDB 4.5 directories - at item Upgrade your database directory to 4.5 + at item Setup my-config.sexp to point to the appropriate BDB 4.5 directories + at item Upgrade your existing database directory to 4.5 @itemize @item Run db43_recover in your 0.6 database @item Optional: run db43_archive -d to remove all logs not part of a checkpoint This will make catastrophic recovery impossible, but reduces the amount of data you have to backup. @item Backup your db files and remaining logs - @item Run db45_checkpoint -1 in your 0.6 database directory + @item Run db45_checkpoint -1 in the database directory @end itemize - at item Migrate 0.6 data to a new 0.6.1 database + at item Upgrade 0.6 data to a fresh 0.9 database @itemize @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))} - @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))} + @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele090/"))} @end itemize @item Test your new application and report any bugs that arise to @email{elephant-devel@@common-lisp.net} @end enumerate @@ -315,14 +318,15 @@ @emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)} @emph{(NOTE: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit - version of your lisp to update to 0.6.1 and then open that database in your 64-bit + version of your lisp to update to 0.9 and then open that database in your 64-bit lisp. There should be no compatibility problems. Best to test your application on a 32-bit lisp if you can, just to be sure.)} @subsection Upgrade from Elephant 0.5 -Follow the upgrade procedures outlined in Elephant release 0.6.0 to migrate your database -from 0.5 to 0.6. Then follow the above procedures for upgrading from an 0.6 database. +Follow the upgrade procedures outlined in the Elephant 0.6.0 INSTALL +file to upgrade your database from 0.5 to 0.6.0. Then follow the above +procedures for upgrading to 0.9. @emph{(NOTE: It may not take much work to make 0.9 upgrade directly from 0.5 However there are so few (none?) 0.5 users that it wasn't deemed worth @@ -333,55 +337,50 @@ @section CL-SQL Although originally designed as an interface to the BerkeleyDB system, -the original Elephant system has been experimenetally extended to -support the use of relational database management systems as the -implementation of the persistent store. This relies on Kevin Rosenberg's -CL-SQL interface to a large number of relational systems. - -Although the BerkeleyDB system is an ideal object store for LISP objects, -one might prefer the licensing of a different system. For example, at -the time of this writing, it is my interpretation that one cannot use -the BerkeleyDB system behind a public website +the original Elephant system has been extended to support the use of +relational database management systems as the implementation of the +persistent store. This relies on Kevin Rosenberg's CL-SQL interface, +which provides access to a large number of relational systems. + +A major motivation of this extension is that one one might prefer the +licensing of a different system. For example, at the time of this +writing, it is our interpretation that one cannot use the BerkeleyDB +system behind a public website http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. Neither the PostGres DBMS nor SQLite 3, nor Elephant itself, imposes -any such restriction. (Elephant is released under the LLGPL -(http://opensource.franz.com/preamble.html .) Older versions were -released under the GPL.) +any such restriction. Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application needs to use the truly relational aspects of those systems, preference for the tools associated with those systems, etc. -The SQL back-end extention of Elephant provides a function for migrating -data seamlessly between repositories. One can quite easily move -data from a BerkeleyDB repository to a PostGres repository, and vice versa. -This offers at least the possiblity than once can develop using one backend, -for example BerkeleyDB, and then later move to Postgres, or vice versa. -One could even operate simultaneously out of multiple repositories, if -there were a good reason to do so. - -At the time of this writing, the basic strategy for the SQL implementation -is quite simple. The same serializer used for the Sleepycat implementation -is employed, the byte-string is base64 encoded, and placed in a single -table which is managed by Elephant. - -All functionality except nested transaction support and cursor-puts -supported by the BerkeleyDB backend is supported by the CL-SQL -back-end. Concurrency and transaction atomicity have not been stress -tested well for the CL-SQL based system. - -Additionally, it is NOT the case that the Elephant system currently provides -transaction support across multiple repositories; it provides the transaction -support provided by the underlying repository to the user in a per-repository -basis. - -The PostGres backend is as currently employed is about 5 times slower than -the BerkeleyDB backend. This could probably change with continued development. - -CL-SQL supports a lot of DBMS systems, but only PostGres and SqlLite 3 have been tested. +Elephant provides functions for migrating data seamlessly between data +stores. One can quite easily move data from a BerkeleyDB repository +to a PostGres repository, and vice versa. This offers at least the +possibility than one can develop using one data store, for example +BerkeleyDB, and then later move to Postgres. One could even operate +simultaneously out of multiple repositories, if there were a good +reason to do so. + +The SQL implementation shares the serializer with the BDB data store, but +base64 encodes the resulting binary stream. This data is placed into +a single table in the SQL data store. + +All functionality except for nested transaction support and +cursor-puts supported by the BerkeleyDB data store is supported by the +CL-SQL data store. CL-SQL transaction integrity under concurrent +operation has not been extensively stress tested. + +Additionally, it is NOT the case that the Elephant system currently +provides transaction support across multiple repositories; it provides +transaction support on a per-repository basis. + +The PostGres backend is currently about 5 times slower than the +BerkeleyDB backend. As of the time of this writing, only PostGres and +SqlLite 3 have been tested as CL-SQL backends. @node CL-SQL Example @comment node-name, next, previous, up @@ -391,20 +390,20 @@ @enumerate @item Install postgres and make sure postmaster is running. -Postgres may be installed on your system; you may be able to use a package manager to -install it, or you can install it quite easily from the PostgresSQL site directly -(http://www.postgresql.org/). +Postgres may be installed on your system; you may be able to use a [104 lines skipped] From ieslick at common-lisp.net Sun Apr 29 21:54:52 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 17:54:52 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/utils Message-ID: <20070429215452.A965834055@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/utils In directory clnet:/tmp/cvs-serv3086/src/utils Modified Files: os.lisp Log Message: Fix for lispworks deadlock detect (silently fails instead of asserting an error) --- /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 02:10:56 1.4 +++ /project/elephant/cvsroot/elephant/src/utils/os.lisp 2007/04/29 21:54:51 1.5 @@ -52,7 +52,7 @@ (in-directory (directory) (ccl:run-program program args :wait nil)) #+lispworks - (apply #'sys::call-system + (funcall #'sys::call-system (format nil "~a~{ '~a'~} &" program args) :current-directory directory :wait nil) From ieslick at common-lisp.net Mon Apr 30 00:38:31 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 20:38:31 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070430003831.5245E3D00E@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv8074 Modified Files: config.sexp ele-clsql.asd elephant.asd Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/config.sexp 2007/04/25 02:27:56 1.11 +++ /project/elephant/cvsroot/elephant/config.sexp 2007/04/30 00:38:29 1.12 @@ -1,48 +1,63 @@ -#+(or sbcl allegro) -((:berkeley-db-include-dir . "/opt/local/include/db45/") +#+(and (or sbcl allegro) (not mswindows) (not windows)) +((:compiler . :gcc) + (:berkeley-db-include-dir . "/opt/local/include/db45/") (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) (:berkeley-db-cachesize . 20971520) - (:clsql-lib . nil) - (:compiler . :gcc)) + (:berkeley-db-map-degree2 . t) + (:clsql-lib-paths . nil) + (:prebuilt-libraries . nil)) + #+openmcl -((:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") +((:compiler . :gcc) + (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :gcc)) + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:prebuilt-libraries . nil) + (:clsql-lib-paths . nil)) #+(and lispworks (not mswindows)) -((:berkeley-db-include-dir . "/opt/local/include/db45/") - (:berkeley-db-lib-dir . "/opt/local/lib/db45/") - (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") - (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :gcc)) +((:compiler . :gcc) + (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") + (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") + (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") + (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:prebuilt-libraries . nil) + (:clsql-lib-paths . nil)) #+(or mswindows windows) -((:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") +((:compiler . :cygwin) + (:berkeley-db-include-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/include/") (:berkeley-db-lib-dir . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/") (:berkeley-db-lib . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll") (:berkeley-db-deadlock . "C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/db_deadlock.exe") - (:pthread-lib . nil) - (:clsql-lib . nil) - (:compiler . :cygwin)) + (:berkeley-db-cachesize . 20971520) + (:berkeley-db-map-degree2 . t) + (:clsql-lib-paths . nil) + (:prebuilt-libraries . t)) ;; Berkeley 4.5 is required, each system will have different settings for ;; these directories, use this as an indication of what each key means ;; -;; Typical pthread settings are: /lib/tls/libpthread.so.0 -;; -;; nil means that the library in question is not loaded -;; -;; NOTE: The latest SBCL (0.9.17+) on linux no longer needs the pthread library, -;; it is statically linked against it now with the new thread support +;; :prebuilt-libraries is true by default for windows machines. It causes +;; the library loader to look in the elephant root directory for the shared +;; libraries. (nil or t) +;; +;; :clsql-lib-paths tell clsql where to look for which ever SQL distribution +;; library files you need it to look for. For example... +;; +;; (:clsql-lib-paths . ("/Users/me/Work/SQlite3/" "/Users/me/Work/Postgresql/")) +;; +;; :pthread-lib is deprecated, for old versions of sbcl prior to 0.9.17 that +;; did not have pthreads compiled in. If you are using an old version, we +;; recommend that you upgrade! +;; Typical pthread settings were: /lib/tls/libpthread.so.0 ;; ;; :compiler options are ;; :gcc (default: for unix platforms with /usr/bin/gcc) @@ -52,16 +67,18 @@ ;; Additional supported parameters include: ;; ;; -;; :berkeley-db-cachesize - an integer indicating the number of bytes -;; for the page cache, default 10MB which is -;; about enough storage for 10k-30k indexed -;; persistent objects -;; -;; :map-using-degree2 - Boolean parameter that indicates whether map -;; operations lock down the btree for the entire -;; transaction or whether they allow other -;; transactions to add/delete/modify values -;; before the map operation is completed. The -;; map operation remains stable and any writes -;; are kept transactional, see user manual as -;; well as berkeley DB docs for more details +;; :berkeley-db-cachesize +;; An integer indicating the number of bytes +;; for the page cache, default 20MB which is +;; about enough storage for 10k-30k indexed +;; persistent objects +;; +;; :berkeley-db-map-degree2 +;; Boolean parameter that indicates whether map +;; operations lock down the btree for the entire +;; transaction or whether they allow other +;; transactions to add/delete/modify values +;; before the map operation is completed. The +;; map operation remains stable and any writes +;; are kept transactional, see user manual as +;; well as berkeley DB docs for more details. --- /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/03/23 16:31:52 1.10 +++ /project/elephant/cvsroot/elephant/ele-clsql.asd 2007/04/30 00:38:29 1.11 @@ -16,29 +16,33 @@ ;;; as governed by the terms of the Lisp Lesser GNU Public License ;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL. -;; (eval-when (:compile-toplevel :load-toplevel :execute) -;; (unless (find-package 'clsql) -;; (asdf:operate 'asdf:load-op 'clsql))) - -;; ;; clsql needs to know how to find libaries to various databases.... -;; (defvar *clsql-foreign-lib-path* "/usr/lib") -;; (defvar *elephant-lib-path* "/usr/local/share/common-lisp/elephant/") -;; (clsql:push-library-path *clsql-foreign-lib-path*) -;; (clsql:push-library-path *elephant-lib-path*) -;; ) - -;;(defparameter *clsql-foreign-lib-path* #p"/usr/lib") -;;(defparameter *elephant-lib-path* #p"/usr/local/share/common-lisp/elephant/") - -;;(defmethod asdf:perform :after ((o asdf:load-op) -; (c (eql (asdf:find-system 'clsql)))) -;; (let ((plp (find-symbol (symbol-name '#:push-library-path) -;; (find-package 'clsql)))) -;; (funcall plp -;; *clsql-foreign-lib-path*) -;; (funcall plp -;; *elephant-lib-path*) -;;)) +(in-package :cl-user) + +;; Forward definition +(defpackage elephant-system + (:use :cl :asdf) + (:export :elephant-c-source :compiler-options :foreign-libraries-to-load-first :get-config-option)) + +(defpackage ele-clsql-system + (:use :cl :asdf :elephant-system)) + +(in-package :ele-clsql-system) + +;; +;; Make sure asdf is loaded prior to evaluating the rest of the file (tacky) +;; +(eval-when (:compile-toplevel :load-toplevel :execute) + (unless (find-package 'clsql) + (asdf:operate 'asdf:load-op 'clsql))) + +(defmethod asdf:perform :after ((o asdf:load-op) + (c (eql (asdf:find-system 'clsql)))) + (let ((paths (get-config-option :clsql-lib-paths (find-system :elephant))) + (plp (find-symbol (symbol-name '#:push-library-path) + (find-package 'clsql)))) + (loop for path in paths do + (format t "Pushing ~A onto clsql::*FOREIGN-LIBRARY-SEARCH-PATHS*~%" path) + (funcall plp (pathname path))))) (defsystem ele-clsql :name "elephant" --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/29 02:10:54 1.46 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:38:29 1.47 @@ -94,6 +94,7 @@ "Run the appropriate compiler for this platform on the source, getting the specific options from 'compiler-options method. Default options can be overridden or augmented by subclass methods" + (unless (get-config-option :prebuilt-libraries c) #+(or mswindows windows) (progn (let* ((pathname (component-pathname c)) @@ -144,7 +145,7 @@ (compiler-options (c-compiler c) c :input-file (namestring (component-pathname c)) :output-file (namestring (first (output-files o c)))))) - (error 'operation-error :component c :operation o))) + (error 'operation-error :component c :operation o)))) #| (defmethod perform ((o compile-op) (c elephant-c-source)) @@ -247,9 +248,21 @@ (error "Could not load ~A into ~A" file module))))) ;; Load the compiled libraries (dolist (file (output-files (make-instance 'compile-op) c)) - (format t "Loading ~A~%" file) - (or (uffi-funcall :load-foreign-library file :module (component-name c)) - (error "Could not load ~A" file)))) + (format t "Attempting to load ~A...~%" (file-namestring file)) + (if (and (probe-file file) + (not (get-config-option :prebuilt-libraries c))) + (progn + (or (uffi-funcall :load-foreign-library file :module (component-name c)) + (error "Could not load ~A" file)) + (format t "Loaded ~A~%" file)) + (let* ((root-dir (asdf:component-pathname (asdf:component-system c))) + (root-library-file (merge-pathnames (file-namestring file) root-path))) + (or (uffi-funcall :load-foreign-library + root-library-file + :module (component-name c)) + (error "Output file ~A not found in elephant root" (namestring root-library-file))) + (format t "Loaded ~A~%" root-library-file))))) + (defmethod operation-done-p ((o load-op) (c elephant-c-source)) nil) @@ -264,7 +277,7 @@ (defsystem elephant :name "elephant" :author "Ben Lee " - :version "0.6.1" + :version "0.9" :maintainer "Robert Read " :licence "LLGPL" :description "Object database for Common Lisp" From ieslick at common-lisp.net Mon Apr 30 00:38:31 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 20:38:31 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070430003831.C78477E005@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv8074/doc Modified Files: installation.texinfo user-guide.texinfo Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/29 21:54:51 1.14 +++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/30 00:38:31 1.15 @@ -246,6 +246,10 @@ @comment node-name, next, previous, up @section Setting up Berkeley DB +We recommend that you download and build a distribution from Oracle. +Some problems have been reported with linking to Debian, Cygwin or +other packages. This is especially true for Windows users. + Beyond ensuring that the file ``my-config.sexp'' points to your BDB installation directories and files, nothing else should be required to configure the example that uses a local ``testdb'' directory as a @@ -428,13 +432,20 @@ @comment node-name, next, previous, up @section Elephant on Windows -The build process on Windows currently only works with GCC under Cygwin. -The process can be a bit tricky, so if it doesn't work out of the box, -we recommend that you download the DLLs from the Elephant website -download page (@url{http://www.common-lisp.net/project/elephant/downloads.html'}). - -Unpack the .zip file into the elephant root directory. Then run the batch file -``copy-dll.bat'' to move them into their proper directory. +The build process on Windows currently only works with GCC under +Cygwin. The process can be a bit tricky, so if it doesn't work out of +the box or you don't want to install cygwin, we recommend that you +download the DLLs from the Elephant website download page +(@url{http://www.common-lisp.net/project/elephant/downloads.html'}). + +Unpack the .zip file into the elephant root directory. Ensure that +your @code{my-config.sexp} file configuration for Windows has + at code{:prebuilt-binaries} set to ``t'' so it will know to look in +the elephant root during the asdf loading process. + +For Berkeley DB users we recommend downloading the Windows binary +distribution of Berkeley DB 4.5 to minimize any potential linking +issues. @node Test Suites @comment node-name, next, previous, up --- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 14:24:57 1.21 +++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/30 00:38:31 1.22 @@ -1555,7 +1555,7 @@ described in config.sexp. They are: @itemize - at item @code{:map-using-degree2} - Improve the efficiency of cursor traversals + at item @code{:berkeley-db-map-degree2} - Improve the efficiency of cursor traversals in the various mapping functions. Defaults to true, meaning a value you just read while mapping may change before the traversal is done. So if you operate only on the current cursor location, you are From ieslick at common-lisp.net Mon Apr 30 00:38:32 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 20:38:32 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/src/elephant Message-ID: <20070430003832.7B897450C2@common-lisp.net> Update of /project/elephant/cvsroot/elephant/src/elephant In directory clnet:/tmp/cvs-serv8074/src/elephant Modified Files: variables.lisp Log Message: Final changes to build process for prebuilt windows files, documenting user configuration options, enabling clsql-lib load paths and some installation manual edits --- /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/25 02:28:02 1.17 +++ /project/elephant/cvsroot/elephant/src/elephant/variables.lisp 2007/04/30 00:38:31 1.18 @@ -42,7 +42,7 @@ ;;;; General support for user configurable parameters (defvar *user-configurable-parameters* - '((:map-using-degree2 *map-using-degree2*) + '((:berkeley-db-map-degree2 *map-using-degree2*) (:berkeley-db-cachesize *berkeley-db-cachesize*))) ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; From ieslick at common-lisp.net Mon Apr 30 00:54:03 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 20:54:03 -0400 (EDT) Subject: [elephant-cvs] CVS elephant Message-ID: <20070430005403.080162B129@common-lisp.net> Update of /project/elephant/cvsroot/elephant In directory clnet:/tmp/cvs-serv11649 Modified Files: elephant.asd Log Message: Typo --- /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:38:29 1.47 +++ /project/elephant/cvsroot/elephant/elephant.asd 2007/04/30 00:54:03 1.48 @@ -256,7 +256,7 @@ (error "Could not load ~A" file)) (format t "Loaded ~A~%" file)) (let* ((root-dir (asdf:component-pathname (asdf:component-system c))) - (root-library-file (merge-pathnames (file-namestring file) root-path))) + (root-library-file (merge-pathnames (file-namestring file) root-dir))) (or (uffi-funcall :load-foreign-library root-library-file :module (component-name c)) From ieslick at common-lisp.net Mon Apr 30 02:33:17 2007 From: ieslick at common-lisp.net (ieslick) Date: Sun, 29 Apr 2007 22:33:17 -0400 (EDT) Subject: [elephant-cvs] CVS elephant/doc Message-ID: <20070430023317.48E731900D@common-lisp.net> Update of /project/elephant/cvsroot/elephant/doc In directory clnet:/tmp/cvs-serv575 Modified Files: elephant-design.texinfo Log Message: Few tweaks to docs to build them; updated docs on website to match latest checkins --- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/27 03:14:55 1.5 +++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/30 02:33:17 1.6 @@ -13,7 +13,7 @@ The architecture has been carefully modularized: - at image{ArchDiagram,,4.5in,[Architecture Diagram],.png} + at image{ArchDiagram,,4.5in,[Architecture Diagram],png} To get a feeling for what is happening inside elephant, it is probably best to walk through the various major protocols to see how these @@ -176,7 +176,7 @@ The BTree protocol almost requires an actual BTree implementation to be at all efficient. Keys and values need to be accessible via the cursor API, which means they need to be walked linearly in the sort -order of the keys (described in @xref{Persistent BTrees}). +order of the keys (described in @ref{Persistent BTrees}). An indexed BTree automatically maintains a hash table of the indices defined on it so that users can access them by mapping or