[elephant-cvs] CVS elephant/doc
ieslick
ieslick at common-lisp.net
Sun Apr 1 14:33:34 UTC 2007
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv29083/doc
Modified Files:
Makefile data-store-reference.texinfo elephant-design.texinfo
elephant.texinfo installation.texinfo reference.texinfo
tutorial.texinfo user-guide.texinfo
Log Message:
Latest documentation changes
--- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/30 23:36:52 1.5
+++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/04/01 14:33:24 1.6
@@ -9,4 +9,4 @@
makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo
pdf: includes-stuff
- texi2dvi --pdf elphant.texinfo
+ texi2dvi --pdf elephant.texinfo
--- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/30 23:36:52 1.4
+++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/01 14:33:24 1.5
@@ -3,9 +3,8 @@
@node Data Store API Reference
@comment node-name, next, previous, up
@chapter Data Store API Reference
- at cindex Data Store API Reference
@cindex Data Store
- at cindex API Reference
+ at cindex API
This reference includes functions that need to be overridden, classes
inherited from or other action taken to implement support for a new
@@ -26,16 +25,16 @@
relevant to them.
@menu
-* Registration:: Register the backend to parse controller specifications.
-* Store Controllers:: Subclassing the store controller.
-* Handling Serialization:: Available facilities for serializing objects.
-* C Utilities:: Writing primitive C types.
-* Slot access:: Support for metaprotocol slot access.
-* Collections:: BTrees and indices.
-* Cursors:: Traversing BTrees.
-* Transactions:: Transaction implementation.
-* Multithreading:: Multithreading considerations.
-* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries
+* Registration: DSR Registration. Register the backend to parse controller specifications.
+* Store Controllers: DSR Store Controllers. Subclassing the store controller.
+* Handling Serialization: DSR Handling Serialization. Available facilities for serializing objects.
+* Memory Utilities: DSR Memory Utilities. Writing primitive C types.
+* Persistent Objects and Slot access: DSR Persistent Objects and Slot Access. Support for metaprotocol slot access.
+* Collections: DSR Collections. BTrees and indices.
+* Cursors: DSR Cursors. Traversing BTrees.
+* Transactions: DSR Transactions. Transaction implementation.
+* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations.
+* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries
@end menu
@node DSR Registration
@@ -96,17 +95,28 @@
@section Slot Access
@cindex Persistent Objects and Slot Access
+Persistence is implement with a metaclass and several required base
+classes.
+
+ at include includes/class-elephant-persistent-metaclass.texinfo
@include includes/class-elephant-persistent.texinfo
+ at include includes/class-elephant-persistent-object.texinfo
+
+Persistent objects can be queries for their home store controller so
+that functions such as map-btree do not need a store-controller
+argument. (NOTE: Should this function be user visible?)
+
@include includes/fun-elephant-backend-get-con.texinfo
- at c @include includes/fun-elephant-backend-oid.texinfo
-All objects require a unique id. During new object creation the
-backend is asked to produce a unique id.
+All objects require a unique object identifier. During new object
+creation the backend is asked to produce a unique id.
@include includes/fun-elephant-backend-next-oid.texinfo
-These functions are called by the metaclass protocol to support
-operations on persistent class slots.
+These functions are called by the metaclass protocol to implement the
+appropriate operations on persistent class slots. Unless protected by
+a transaction, the side effects of these functions should be atomic,
+persistent and visible to other threads on completion.
@include includes/fun-elephant-backend-persistent-slot-writer.texinfo
@include includes/fun-elephant-backend-persistent-slot-reader.texinfo
@@ -121,7 +131,9 @@
To support collections, the data store must subclass the following
classes.
- at include includes/class-elephant-btree.texinfo.texinfo
+ at include includes/class-elephant-persistent-collection.texinfo
+
+ at include includes/class-elephant-btree.texinfo
@include includes/class-elephant-btree-index.texinfo
@include includes/class-elephant-indexed-btree.texinfo
@@ -131,42 +143,48 @@
@include includes/fun-elephant-build-btree.texinfo
@include includes/fun-elephant-build-indexed-btree.texinfo
-And every btree needs accessors, these must be implemented for btree,
-indexed-btree and btree-index.
+Most of the user-visible operations over BTrees must be implemented.
+Class indexing functions such as @code{map-class} and
+ at code{get-instances-by-value} and related functions are all
+implemented using map-btree and map-index.
+
+ at itemize
+ at item @ref{Generic-Function elephant:get-value}
+ at item @ref{Generic-Function (setf elephant:get-value)}
+ at item @ref{Generic-Function elephant:existsp}
+ at item @ref{Generic-Function elephant:remove-kv}
+ at item @ref{Generic-Function elephant:get-index}
+ at item @ref{Generic-Function elephant:remove-index}
+ at item @ref{Generic-Function elephant:map-btree}
+ at item @ref{Generic-Function elephant:map-index}
+ at end itemize
- at include includes/fun-elephant-get-value.texinfo
- at include includes/fun-elephant-setf-get-value.texinfo
- at include includes/fun-elephant-existsp.texinfo
- at include includes/fun-elephant-remove-kv.texinfo
+Mapping over the indices of a btree is important to derived facilities
+such as class indexing and the query subsystem.
@include includes/fun-elephant-map-indices.texinfo
- at include includes/fun-elephant-get-index.texinfo
- at include includes/fun-elephant-remove-index.texinfo
-
-Critical to indexing and queries are the map operators for collections
-
- at include includes/fun-elephant-map-btree.texinfo
- at include includes/fun-elephant-map-index.texinfo
@node DSR Cursors
@comment node-name, next, previous, up
@section Cursors
@cindex Cursors
- at include includes/class-cursor.texinfo
- at c #:cursor-btree
- at c #:cursor-oid
- at c #:cursor-initialized-p
+Data stores must subclass these cursor classes and implement all the
+methods described in @ref{Cursors} except @ref{Macro
+elephant:with-btree-cursor}.
+
+ at include includes/class-elephant-cursor.texinfo
+ at include includes/class-elephant-secondary-cursor.texinfo
@node DSR Transactions
@comment node-name, next, previous, up
@section Transactions
@cindex Transactions
-These functions must be implemented or stubbed in any
-backend.
+These functions must be implemented or stubbed by all data stores.
@include includes/fun-elephant-backend-execute-transaction.texinfo
+
@include includes/fun-elephant-backend-controller-start-transaction.texinfo
@include includes/fun-elephant-backend-controller-commit-transaction.texinfo
@include includes/fun-elephant-backend-controller-abort-transaction.texinfo
@@ -179,33 +197,77 @@
@include includes/fun-elephant-backend-transaction-store.texinfo
@include includes/fun-elephant-backend-transaction-object.texinfo
+;; Designer considerations:
+;; - with-transaction passes *current-transaction* or the user parameter to execute-transaction
+;; in the parent keyword argument. Backends allowing nested transactions can treat the transaction
+;; as a parent, otherwise they can reuse the current transaction by ignoring it (inheriting the dynamic
+;; value of *current-transaction*) or rebinding the dynamic context (whatever makes coding easier).
+;; - ensure-transaction uses *current-transaction* to determine if there is a current transaction
+;; in progress (not null). If so, it jumps to the body directly. Otherwise it executes the body in a
+;; new transaction by calling ...
+;; - execute-transaction contract:
+;; - Backends must dynamically bind *current-transaction* to a meaningful identifier for the
+;; transaction in progress and execute the provided closure in that context
+;; - All non-local exists result in an abort; only regular return values result in a commit
+;; - If a transaction is aborted due to a deadlock or read conflict, execute-transaction should
+;; automatically retry with an appropriate default amount
+;; - execute-transaction can take any number of backend-defined keywords, although designers should
+;; make sure there are no semantic conflicts if there is a name overlap with existing backends
+;; - A typical design approach is to make sure that the most primitive interfaces to the backend
+;; database look at *current-transaction* to determine whether a transaction is active. Users code can also
+;; access this parameter to check whether a transaction is active.
+
@node DSR Multithreading Considerations
@comment node-name, next, previous, up
@section Multithreading Considerations
@cindex Multithreading
- at c utils locks
- at c utils thread-vars
+Generic locking utility functions
+
+Variable behavior in multithreading situations
@node DSR Handling Serialization
@comment node-name, next, previous, up
@section Handling Serialization
@cindex Serialization
- at c #:deserialize #:serialize
- at c #:serialize-symbol-complete
- at c #:deserialize-from-base64-string
- at c #:serialize-to-base64-string
+Backends must initialize @ref{Class elephant:store-controller} with
+internal serializer functions. Packages @code{elephant-serializer1}
+and @code{elephant-serializer2} contains serialize and deserialize
+methods on buffer-streams as defined in @code{elephant-memutil}. The
+elephant functions @code{serialize} and @code{deserialize} dispatch on
+the appropriate slot values of the store-controller.
+
+ at verbatim
+NOTE: This should perhaps become entirely the job of the data store to
+decide how to serialize values and for a specific version, what
+serializer to use. The elphant main package can define serializers
+for use by different backends.
+ at end verbatim
+
+ at include includes/fun-elephant-backend-serialize.texinfo
+ at include includes/fun-elephant-backend-deserialize.texinfo
+
+These utility functions are useful if a data store does not have the
+ability to store variable length binary data. They are based on the
+ at code{cl-base64} library.
+
+ at include includes/fun-elephant-backend-serialize-to-base64-string.texinfo
+ at include includes/fun-elephant-backend-deserialize-from-base64-string.texinfo
- at node DSR Memory utilities
+ at node DSR Memory Utilities
@comment node-name, next, previous, up
@section Memory utilities
@cindex Memory utilities
- at node DSR Foreign libraries
+Details about memory utilities here.
+
+ at node DSR Foreign Libraries
@comment node-name, next, previous, up
@section Foreign libraries
@cindex Foreign libraries
+How foreign libraries are built and used via UFFI. What functions are
+in the .asd files or main lisp code to build & load libraries?
--- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/30 14:34:34 1.2
+++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/04/01 14:33:29 1.3
@@ -1,9 +1,37 @@
+ at c -*-texinfo-*-
@node Elephant Design
@comment node-name, next, previous, up
- at section Elephant Design
+ at chapter Elephant Design
@cindex design
+Elephant's early architecture was tightly coupled to the Berkeley DB
+API. Over time we've moved towards a more modular architecture to
+support easy upgrading, repository migration, shared functionality
+between data stores and general hygene.
+
+The current architecture is modularized into the following pieces:
+
+ at verbatim
+[Picture describing] Metaclass, Store controller, persistent objects,
+data stores, serializer & memutils. Derived features such as class
+indexing, migration, query system and root operations can also be
+illustrated?
+ at end verbatim
+
+To get a feeling for what is happening inside elephant, it is probably
+most useful to walk through the various major protocols:
+
+ at itemize
+ at item Initialization of a store controller
+ at item Creating a persistent object
+ at item Operations on persistent slots
+ at item Operations on persistent collections
+ at item Implementing @code{with-transaction}
+ at end itemize
+
+ at section Initializing a store controller
+
When the main elephant @code{open-store} function is called, it calls
@code{get-controller} which grabs an existing store controller if the
spec is identical, or builds a new controller. Building the
@@ -13,4 +41,14 @@
that returns a @code{store-controller} subclass instance specific to
that backend.
-Elephant than calls open-controller
+Elephant than calls open-controller to actually establish a connection
+to or create the files of the data store.
+
+ at section Persistent Object Creation
+ at section Persistent Slot Protocol
+ at section Persistent Slot Protocol
+ at section Persistent Collection Protocol
+ at section Implementing Transactions
+
+
+
--- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/03/30 14:34:34 1.6
+++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/01 14:33:29 1.7
@@ -38,6 +38,7 @@
@insertcopying
@end ifnottex
+ at ifhtml
@menu
* Table of Contents::
@end menu
@@ -49,7 +50,7 @@
* Tutorial:: A basic ``getting started'' tutorial.
* Installation:: Installation and test-suite procedures.
* User Guide:: In depth discussion of all Elephant facilities and features.
-* Usage scenarios:: Design scenarios for Elephant applications.
+* Usage Scenarios:: Design scenarios for Elephant applications.
* User API Reference:: Function and class documentation of the user API.
* Elephant Design:: An overview of elephant's internal architecture.
* Data Store API Reference:: Function level documentation for data store implementors.
@@ -66,6 +67,8 @@
* Colophon::
@end menu
+ at end ifhtml
+
@node Table of Contents
@unnumbered
@comment node-name, next, previous, up
--- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/03/30 14:34:34 1.5
+++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/01 14:33:29 1.6
@@ -10,9 +10,9 @@
* Configuring Elephant:: Setting up Elephant and the configuration file.
* Loading Elephant:: Loading Elephant and the data store loading protocol.
* Berkeley DB Data Store:: Installing support for the Berkeley DB data store
-* Berkeley DB Examples:: An example of installing and running the Berkeley DB data store.
+* Berkeley DB Example:: An example of installing and running the Berkeley DB data store.
* CL-SQL Data Store:: Install and connecting to the CL-SQL data store
-* CL-SQL Examples:: An example of using the CL-SQL data store.
+* CL-SQL Example:: An example of using the CL-SQL data store.
* Elephant on Windows:: More details about running Elephant on Windows
* Test Suites:: How to run and interpret the output of the regression test suite
* Documentation:: Building documentation from texinfo sources.
@@ -277,10 +277,6 @@
As of Elephant 0.3, Elephant has been tested to work with both Postgres, and
SQLite 3, thanks do Dan Knapp.
- at node Extension Status
- at comment node-name, next, previous, up
- at section Extension Status
-
As far as is known at this writing, all functionality except nested transaction
support and cursor-puts supported by the BerkeleyDB backend is supported by the
CL-SQL back-end. Concurrency and transaction atomicity have not been stress tested
@@ -302,13 +298,9 @@
the multi-repository version somewhat complicates the underlying
persistent object management.
- at node PostGres Examples
- at comment node-name, next, previous, up
- at section Setting up PostGres
-
- at node Setting up PostGres
+ at node CL-SQL Example
@comment node-name, next, previous, up
- at section Setting up PostGres
+ at section CL-SQL Example
To set up a PostGres based back end, you should:
@@ -351,6 +343,10 @@
sequences, and so on needed by the Elephant system will be created in the
schema named ``test'' automatically.
+ at node Elephant on Windows
+ at comment node-name, next, previous, up
+ at section Elephant on Windows
+
@node Test Suites
@comment node-name, next, previous, up
@section Test Suites
--- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/03/30 23:36:52 1.8
+++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/01 14:33:29 1.9
@@ -22,14 +22,20 @@
@section Store Controllers
@cindex Store Controllers
-Store controllers provide the persistent storage for CLOS objects and BTree collections. Any persistent operations must be done in the context of a store controller.
+Store controllers provide the persistent storage for CLOS objects and
+BTree collections. Any persistent operations must be done in the
+context of a store controller. The default store-controller is stored
+in a global variable.
- at include includes/class-elephant-store-controller.texinfo
@include includes/var-elephant-star-store-controller-star.texinfo
+ at c @include includes/class-elephant-store-controller.texinfo
+ at ref{Class elephant:store-controller} is associated with the following
+user methods and macros:
+
+ at include includes/macro-elephant-with-open-store.texinfo
@include includes/fun-elephant-open-store.texinfo
@include includes/fun-elephant-close-store.texinfo
- at include includes/macro-elephant-with-open-store.texinfo
@include includes/fun-elephant-get-from-root.texinfo
@include includes/fun-elephant-add-to-root.texinfo
@@ -42,12 +48,12 @@
@section Persistent Objects
@cindex Persistent Objects
- at include includes/class-elephant-persistent-metaclass.texinfo
- at include includes/class-elephant-persistent.texinfo
- at include includes/class-elephant-persistent-object.texinfo
+ at ref{Class elephant:persistent-metaclass} can be used as the
+:metaclass argument in a defclass form to create a persistent object.
+Slots of the metaclass take the :index and :transient keyword
+arguments and the class accepts the :index keyword argument.
@include includes/macro-elephant-defpclass.texinfo
-
@include includes/fun-elephant-drop-pobject.texinfo
@node Persistent Object Indexing
@@ -94,22 +100,26 @@
interface is provided for reference only, a new system is under
development for the 0.7 release.
- at include includes/fun-elephant-get-query-results.texinfo
- at include includes/fun-elephant-map-class-query.texinfo
+ at c @include includes/fun-elephant-get-query-results.texinfo
+ at c @include includes/fun-elephant-map-class-query.texinfo
@node Collections
@comment node-name, next, previous, up
@section Collections
@cindex Collections
- at include includes/class-elephant-persistent-collection.texinfo
- at include includes/class-elephant-btree.texinfo
- at include includes/class-elephant-indexed-btree.texinfo
- at include includes/class-elephant-btree-index.texinfo
+Persistent collections inherit from @ref{Class
+elephant:persistent-collection} and consist of the @ref{Class
+elephant:btree}, @ref{Class elephant:indexed-btree} and @ref{Class
+elephant:btree-index} classes. The following operations are defined
+on most of these classes. More information can be found in @ref{Using
+BTrees} and @ref{Secondary Indices}.
@include includes/fun-elephant-get-value.texinfo
@include includes/fun-elephant-setf-get-value.texinfo
@include includes/fun-elephant-remove-kv.texinfo
+ at include includes/fun-elephant-existsp.texinfo
+
@include includes/fun-elephant-map-btree.texinfo
@include includes/fun-elephant-map-index.texinfo
@@ -118,17 +128,17 @@
@include includes/fun-elephant-get-primary-key.texinfo
@include includes/fun-elephant-remove-index.texinfo
-
@node Cursors
@comment node-name, next, previous, up
@section Cursors
@cindex Cursors
- at include includes/class-elephant-cursor.texinfo
- at include includes/class-elephant-secondary-cursor.texinfo
+Cursors are objects of type cursor (@pxref{Class elephant:cursor})
+which provide methods for complex traversals of BTrees.
+
+ at include includes/macro-elephant-with-btree-cursor.texinfo
@include includes/fun-elephant-make-cursor.texinfo
@include includes/fun-elephant-cursor-close.texinfo
- at include includes/macro-elephant-with-btree-cursor.texinfo
@include includes/fun-elephant-cursor-current.texinfo
@include includes/fun-elephant-cursor-delete.texinfo
--- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/03/30 14:34:34 1.10
+++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/01 14:33:29 1.11
@@ -15,6 +15,7 @@
* Persistent collections:: Keep track of collections of values.
* Indexing Persistent Classes:: Simple way to keep track of persistent instances.
* Using Transactions:: Providing ACID database properties.
+* Advanced Topics:: Additional Elephant features covered in the User Guide.
@end menu
@node Overview
@@ -704,14 +705,14 @@
@end lisp
The @ref{User Guide} contains a descriptions of the advanced features
-of @ref{Class indices} such as ``derived indicies'' that allow you to
+of @ref{Class Indices} such as ``derived indicies'' that allow you to
order classes according to an arbitrary function, a dynamic API for
adding and removing slots and how to set a policy for resolving
conflicts between the code image and a database where the indexing
specification differs.
This same facility is also available for your own use. For more
-information @pxref{Using Indexed BTrees}.
+information @pxref{Secondary Indices}.
@node Using Transactions
@@ -914,8 +915,71 @@
complicated. The best strategy at the beginning is a conservative
one, break things up into the smallest logical sets of primitive
operations and only wrap higher level functions in transactions when
-they absolutely have to commit together. See @ref{Transaction details}
-for the full details and @pxref{Usage scenarios} for more examples of
+they absolutely have to commit together. See @ref{Transaction Details}
+for the full details and @pxref{Usage Scenarios} for more examples of
how systems can be designed and tuned using transactions.
+ at node Advanced Topics
+ at comment node-name, next, previous, up
+ at section Advanced Topics
+
+The tutorial covers the essential topics and concepts for using
+Elephant. Many people will find that these features are the ones that
+are most often needed and used in ordinary applications.
+
+More sophisticated uses of Elephant may require additional features
+that are covered in the user guide. The following is a list of major
+features in the user guide that were not covered in this tutorial.
+
+ at itemize
+ at item @strong{Class Heirarchies and Queries}
+ There are some subtle issues to take into account when querying
+persistent classes. For example, how do you query a base class of
+type people to get subclass instances such as employee, manager,
+consultant, etc?
+ at item @strong{Derived Class Indices}
+ You can create your own indices for classes that are arbitrary
+lisp functions of the persistent object.
+ at item @strong{Class Definition/Database Conflict Resolution}
+ When you startup lisp, there are potential conflicts between the
+class definition and the indexing records in the database. There are
+some constraints to account for and some facilities to manage
+how slots, class indices and
+ at item @strong{Dynamic Class Index Management}
+ It is possible to add and remove indexes from classes at runtime.
+ at item @strong{BTree Cursors}
+ If you need to do more than iterate over a collection, or you need
+to delete elements of the collection as you iterate cursors are an
+important data structure. They implement a variety of operators for
+moving backward and forward over a btree, including ranged operations
+and iterating of duplicate or unique values.
+ at item @strong{Indexed BTrees}
+ Indexed BTrees are just like BTrees, except it is possible to add
+indexes which are BTrees who's values are primary keys in the parent
+ at code{indexed-btree}. This allows for multiple ordering and groupings
+of the values of a BTree.
+ at item @strong{Using the Map Operators}
+ Mapping operators can be very efficient if properly utilized.
+ at item @strong{Handling Errors and Conditions}
+ There are a variety of errors that can occur in Elephant that need
+to be dealt with by applications.
+ at item @strong{Deadlock Detection in Berkeley DB}
+ Berkeley DB requires an external process to detect deadlock
+conditions among transactions. The :deadlock-detect keyword argument
+to open-store for Berkeley DB specs will launch this process on most
+lisps.
+ at item @strong{Using Multiple Stores}
+ Multiple store controllers can be open simultaneously. However it
+does make the code more complex and you need to be careful about how
+you use them to avoid crashes and other unpleasant side effects.
+ at item @strong{Custom Transaction Architecture}
+ You can implement your own version of with-transaction using the
+underlying controller methods for starting, aborting and committing
+transactions. You had better know what you are doing, however!
+ at item @strong{Using Multiple Threads and Processes}
+ What constraints must be accommodated to use Elephant data stores in
+multiple threads? What capabilities are there to share data stores
+among multiple processes or machines?
+ at end itemize
+Further, @pxref{Usage Scenarios} for information about Elephant design patterns, solutions to common problems and other scenarios with multiple possible solutions.
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/30 14:34:34 1.4
+++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/01 14:33:29 1.5
@@ -8,18 +8,19 @@
@menu
* The Store Controller:: Behind the curtain.
* Serialization details:: The devil hides in the details.
-* Reachability:: Determining liveness in a store.
* Persistent objects:: All the dirt on persistent objects.
-* Class indices:: In-depth discussion about indexing persistent indices.
+* Class Indices:: In-depth discussion about indexing persistent indices.
* Querying persistent instances:: Retrieving instances of classes.
* Using BTrees:: Using the native btree.
* Secondary Indices:: Alternative ways to index collections.
* Using Cursors:: Low-level access to BTrees.
-* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls.
+* Transaction Details:: Develop a deeper understanding of transactions and avoid the pitfalls.
* Multi-repository Operation:: Specifying repositories.
+* Multi-threaded Applications:: What considerations are required for safe multi-threading
+* Multiple Processes and Distributed Applications:: Can elephant be run on multiple CPUs and multiple machines?
* Repository Migration and Upgrade:: How to move objects from one repository to another.
-* Garbage collection:: How to recover storage and OIDs in long-lived repositories.
-* Performance tuning:: How to get the most from Elephant.
+* Garbage Collection:: How to recover storage and OIDs in long-lived repositories.
+* Performance Tuning:: How to get the most from Elephant.
@end menu
@node Persistent objects
@@ -83,9 +84,15 @@
@code{with-open-controller} macro. Opening and closing a controller
is very expensive.
+ at node Serialization details
+ at comment node-name, next, previous, up
+ at section Serialization details
+
+Empty.
+
@node Class Indices
@comment node-name, next, previous, up
- at section Class Indicies
+ at section Class Indices
You can enable/disable class indexing for an entire class. When you disable
indexing all references to instances of that class are lost. If you re-enable
@@ -130,6 +137,10 @@
somewhat user customizable; documentation for this exists in the source
file referenced above.
+ at node Querying persistent instances
+ at comment node-name, next, previous, up
+ at section Querying persistent instances
+
@node Using BTrees
@comment node-name, next, previous, up
@section Using BTrees
@@ -147,10 +158,127 @@
blocks of data is relatively inexpensive after a seek and comparisons
on objects that are stored in memory is cheap.
+ at node Secondary Indices
+ at comment node-name, next, previous, up
+ at section Secondary Indices
+
+Empty.
+
+ at node Using Cursors
+ at comment node-name, next, previous, up
+ at section Using Cursors
+
+Empty.
- at node Repository Migration
+ at node Transaction Details
@comment node-name, next, previous, up
- at section Repository Migration
+ at section Transaction Details
+
+;; Transaction architecture:
+;;
+;; User and designer considerations:
+;; - *current-transaction* is reserved for use by dynamic transaction context. The default global
+;; value must always be null (no transaction). Each backend can set it to a different parameter
+;; within the dynamic context of an execute-transaction.
+;; - Any closures returned from within a transaction cannot bind *current-transaction*
+;; - Only a normal return value will result in the transaction being committed, any non-local exit
+;; results in a transaction abort. If you want to do something more sophisticated, roll your own
+;; using controller-start-transaction, etc.
+;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc)
+;; knowing that the transaction will be aborted
+;;
+
+ at node Multi-threaded Applications
+ at comment node-name, next, previous, up
+ at section Multi-threaded Applications
+
+Sleepycat plays well with threads and processes. The store controller
+is thread-safe by default, that is, can be shared amongst threads.
+This is set by the @code{:thread} key. Transactions may not be shared
+amongst threads except serially. One thing which is NOT thread and
+process safe is recovery, which should be run when no one is else is
+talking to the database environment. Consult the Sleepycat docs for
+more information.
+
+Elephant uses some specials to hold parameters and buffers. If you're
+using a natively threaded lisp, you can initialize these specials to
+thread-local storage by using the @code{run-elephant-thread} function,
+assuming your lisp creates thread-local storage for let-bound
+specials. (This functionality is currently broken)
+
+Persisting ordinary aggregate types (e.g. NOT persistent classes or
+btrees) suffers from something called "merge-conflicts." Since
+updating one value of an aggregate object requires the entire object
+to be written to the database, in heavily threaded situations you may
+overwrite changes another thread or process has committed. This is
+not protected by transactions!
+
+Consider two processes operating on the same cons:
+
+ at code{
+-----start---read--update-car--write--commit----------------
+-start------read--update-cdr-----------------write--commit--
+}
+
+Although the first process successfully committed its transaction, its
+work (writing to the car) will be erased by the second transaction
+(which writes both the car and cdr.)
+
+Persistent classes and persistent collections do not suffer from
+merge-conflicts, since each slot / entry is a separate database entry.
+
+ at node Multi-repository Operation
+ at comment node-name, next, previous, up
+ at section Multi-repository Operation
+
+Elephant now keeps a small hashtables that maps ``database specifications'' into
+actual database connections.
+
+If a database spec is a string, it is assumed to be a BerkeleyDB path.
+If it is a list, it is a assumed to be a CL-SQL connection specification.
+For example:
+ at lisp
+ELE-TESTS> *testdb-path*
+"/home/read/projects/elephant/elephant/tests/testdb/"
+ELE-TESTS> *testpg-path*
+(:postgresql "localhost.localdomain" "test" "postgres" "")
+ELE-TESTS>
+ at end lisp
+
+The tests now have a function @code{do-all-tests-spec} that take a spec and
+based on its type attempt to open the correct kind of store controller and
+perform the tests.
+
+The routine @code{get-controller} takes this specifiation.
+
+The basic strategy is that the ``database specification'' object is stored in
+every persistent object and collection so that the repository can be found.
+
+In this way, objects that reside in different repositories can coexist within
+the LISP object space, allowing data migration.
+
+;; Multiple stores
+
+;; Multiple store considerations:
+;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to
+;; work around: how to avoid writing one store with a transaction created in the context of another.
+;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct;
+;; this requirement may relax in the future
+;; - The following macros accomodate multiple stores by requiring that execute-transaction return a
+;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller
+;; is the store instance it is associated with. A nested or ensured transaction is only indicated
+;; in the call to execute transaction if the store controllers match, otherwise a new transaction
+;; for that store is created
+
+ at node Multiple Processes and Distributed Applications
+ at comment node-name, next, previous, up
+ at section Multiple Processes and Distributed Applications
+
+Can we do this? What do we have to say about this?
+
+ at node Repository Migration and Upgrade
+ at comment node-name, next, previous, up
+ at section Repository Migration and Upgrade
This version of Elephant supports migration between store controllers of
any backend type.
@@ -227,49 +355,17 @@
@code{*inhibit-slot-writes*} in your user method using
@code{with-inhibited-slot-copy} a convenience macro.
-
- at node Threading
+ at node Garbage Collection
@comment node-name, next, previous, up
- at section Threading
-
-Sleepycat plays well with threads and processes. The store controller
-is thread-safe by default, that is, can be shared amongst threads.
-This is set by the @code{:thread} key. Transactions may not be shared
-amongst threads except serially. One thing which is NOT thread and
-process safe is recovery, which should be run when no one is else is
-talking to the database environment. Consult the Sleepycat docs for
-more information.
-
-Elephant uses some specials to hold parameters and buffers. If you're
-using a natively threaded lisp, you can initialize these specials to
-thread-local storage by using the @code{run-elephant-thread} function,
-assuming your lisp creates thread-local storage for let-bound
-specials. (This functionality is currently broken)
-
-Persisting ordinary aggregate types (e.g. NOT persistent classes or
-btrees) suffers from something called "merge-conflicts." Since
-updating one value of an aggregate object requires the entire object
-to be written to the database, in heavily threaded situations you may
-overwrite changes another thread or process has committed. This is
-not protected by transactions!
-
-Consider two processes operating on the same cons:
-
- at code{
------start---read--update-car--write--commit----------------
--start------read--update-cdr-----------------write--commit--
-}
-
-Although the first process successfully committed its transaction, its
-work (writing to the car) will be erased by the second transaction
-(which writes both the car and cdr.)
+ at section Garbage Collection
-Persistent classes and persistent collections do not suffer from
-merge-conflicts, since each slot / entry is a separate database entry.
+GC is not implemented, but migration (@pxref{Repository Migration and
+Upgrade}) will consolidate storage and recover OIDs which emulates GC.
+No online solution is currently supported.
- at node Performance Tips
+ at node Performance Tuning
@comment node-name, next, previous, up
- at section Performance Tips
+ at section Performance
Performance is usually measured in transactions per second. Database
reads are cheap. To get more transactions throughput, consider
@@ -290,8 +386,8 @@
things. It is fast but consing with floats and doubles. YMMV with
other values, though I've tried to make them fast.
-Using @code{*auto-commit*} and not @code{with-transactions} is a great
-way to have a huge number of transactions. You'll find that
+Use @code{with-transactions} to avoid many automatic transactions, for
+example you'll find that this construct
@lisp
(dotimes (i 1000) (add-to-root "key" "value"))
@@ -300,9 +396,8 @@
is much slower than
@lisp
-(let ((*auto-commit* nil))
- (with-transaction ()
- (dotimes (i 1000) (add-to-root "key" "value"))))
+(with-transaction ()
+ (dotimes (i 1000) (add-to-root "key" "value"))))
@end lisp
since there's only 1 transaction in the latter. However storing
@@ -319,32 +414,3 @@
mode is not suitable for use in web servers or other typically
multi-threaded applications.
- at node Multi-repository Operation
- at comment node-name, next, previous, up
- at section Multi-repository Operation
-
-Elephant now keeps a small hashtables that maps ``database specifications'' into
-actual database connections.
-
-If a database spec is a string, it is assumed to be a BerkeleyDB path.
-If it is a list, it is a assumed to be a CL-SQL connection specification.
-For example:
- at lisp
-ELE-TESTS> *testdb-path*
-"/home/read/projects/elephant/elephant/tests/testdb/"
-ELE-TESTS> *testpg-path*
-(:postgresql "localhost.localdomain" "test" "postgres" "")
-ELE-TESTS>
- at end lisp
-
-The tests now have a function @code{do-all-tests-spec} that take a spec and
-based on its type attempt to open the correct kind of store controller and
-perform the tests.
-
-The routine @code{get-controller} takes this specifiation.
-
-The basic strategy is that the ``database specification'' object is stored in
-every persistent object and collection so that the repository can be found.
-
-In this way, objects that reside in different repositories can coexist within
-the LISP object space, allowing data migration.
More information about the Elephant-cvs
mailing list