[elephant-cvs] CVS elephant/doc
ieslick
ieslick at common-lisp.net
Sat Apr 28 02:31:14 UTC 2007
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv16753/doc
Modified Files:
copying.texinfo data-store-reference.texinfo elephant.texinfo
installation.texinfo reference.texinfo scenarios.texinfo
tutorial.texinfo user-guide.texinfo
Log Message:
Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft
--- /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/03/30 14:34:34 1.4
+++ /project/elephant/cvsroot/elephant/doc/copying.texinfo 2007/04/28 02:31:07 1.5
@@ -8,29 +8,30 @@
@section Elephant Licensing
- at b{Elephant}: a persistent metaprotocol and object-oriented database
-for Common Lisp.
+Elephant is a persistent metaprotocol and object-oriented database for
+Common Lisp. Detailed information and distributions can be found at
+ at uref{http://www.common-lisp.net/project/elephant}.
-Homepage: @uref{http://www.common-lisp.net/project/elephant}
+The program is released under the following license:
@quotation
-Elephant users are granted the rights to distribute and use this software
-as governed by the terms of the Lisp Lesser GNU Public License
- at uref{http://opensource.franz.com/preamble.html}, also known as the LLGPL.
+Elephant users are granted the rights to distribute and use this
+software as governed by the terms of the Lisp Lesser GNU Public
+License @uref{http://opensource.franz.com/preamble.html}, also known
+as the LLGPL.
@end quotation
Copyrights include:
@quotation
-Copyright (c) 2004 by Andrew Blumberg and Ben Lee
-
-Copyright (c) 2005-2007 by Robert L. Read
-
-Copyright (c) 2006-2007 by Ian Eslick
+Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. @*
+Version 0.5, Copyright @copyright{} 2006 Robert L. Read. @*
+Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @*
+Portions copyright respective contributors (see @file{CREDITS}).
@end quotation
-Portions of this program (namely the C unicode string sorter) are
-derived from IBM's @b{ICU}: @uref{http://oss.software.ibm.com/icu/,
+Portions of the program (namely the C unicode string sorter) are
+derived from IBM's ICU: @uref{http://oss.software.ibm.com/icu/,
ICU Website} whose copyright and license follows below.
@quotation
@@ -75,6 +76,33 @@
are the property of their respective owners.
@end quotation
+ at section Elephant Manual Copyright and Licensing
+
+ at quotation
+Permission is granted to copy, distribute and/or modify this document
+under the terms of the GNU Free Documentation License.
+ at end quotation
+
+Copyrights include:
+
+ at quotation
+Original Version, Copyright @copyright{} 2004 Ben Lee. @*
+Versions 0.5-0.6, Copyright @copyright{} 2006 Robert L. Read. @*
+Current Version, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @*
+ at end quotation
+
+ at section 3rd Party Libraries
+
+Elephant depends on 3rd party lisp libraries. See their respective
+distributions for detailed copyright and licensing information. The
+following is a brief summary.
+
+ at itemize
+ at item @b{uffi}: By Kevin Rosenberg, no significant restrictions
+ at item @b{cl-base64}: By Kevin Rosenberg, no significant restrictions
+ at item @b{rt}: By Richard Waters, MIT License
+ at end itemize
+
@section Data Store Licensing Considerations
The Berkeley DB data store is based on the Berkeley DB C library, now
@@ -100,5 +128,3 @@
@item MySQL: Dual licensing (similar to BDB), see @uref{http://www.mysql.com/company/legal/licensing/, the MySQL license page}
@end itemize
-
-
--- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/27 03:14:55 1.7
+++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/04/28 02:31:07 1.8
@@ -34,7 +34,7 @@
* Cursors: DSR Cursors. Traversing BTrees.
* Transactions: DSR Transactions. Transaction implementation.
* Multithreading Considerations: DSR Multithreading Considerations. Multithreading considerations.
-* Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries
+ at c * Foreign Libraries: DSR Foreign Libraries. Using UFFI and ASDF to build or link foreign libraries
@end menu
@node DSR Registration
@@ -147,8 +147,8 @@
implemented using map-btree and map-index.
@itemize
- at item @ref{Generic-Function elephant:get-value}
- at item @ref{Generic-Function (setf elephant:get-value)}
+ at item @ref{Generic-Function elephant:get-value} (and @code{(setf get-value)})
+ at c @item @ref{Generic-Function (setf elephant:get-value)}
@item @ref{Generic-Function elephant:existsp}
@item @ref{Generic-Function elephant:remove-kv}
@item @ref{Generic-Function elephant:get-index}
@@ -168,7 +168,7 @@
@cindex Cursors
Data stores must subclass these cursor classes and implement all the
-methods described in @ref{Cursors} except @ref{Macro
+methods described in @ref{DSR Cursors} except @ref{Macro
elephant:with-btree-cursor}.
@include includes/class-elephant-cursor.texinfo
@@ -220,9 +220,23 @@
@section Multithreading Considerations
@cindex Multithreading
-Generic locking utility functions
+This expands slightly on the multithreading discussion in
+ at ref{Multi-threaded Applications}.
-Variable behavior in multithreading situations
+Elephant provides a set of generic locking functions in
+ at code{src/utils/locks.lisp} to help protect any shared structures.
+There are standard locking functions (@code{ele-with-lock}) and then
+a special locking interface called @code{ele-with-fast-lock} which
+on some lisps provides a faster locking option than the standard OS
+locks of the basic interface. (i.e. under Allegro this uses
+ at code{without-interrupts} because Allegro still runs in a single
+OS process on all platforms, this is not true of SBCL).
+
+See the sections on Transaction handling, particularly the dynamic
+behavior of @code{*current-transaction*}. Also read up on the
+store controller section in the User Guide to better understand the
+role of @code{*store-controller*}. At this time there are no other
+global variables to worry about.
@node DSR Handling Serialization
@comment node-name, next, previous, up
@@ -253,19 +267,19 @@
@include includes/fun-elephant-data-store-serialize-to-base64-string.texinfo
@include includes/fun-elephant-data-store-deserialize-from-base64-string.texinfo
- at node DSR Memory Utilities
- at comment node-name, next, previous, up
- at section Memory utilities
- at cindex Memory utilities
-
-Details about memory utilities here.
-
- at node DSR Foreign Libraries
- at comment node-name, next, previous, up
- at section Foreign libraries
- at cindex Foreign libraries
+ at c @node DSR Memory Utilities
+ at c @comment node-name, next, previous, up
+ at c @section Memory utilities
+ at c @cindex Memory utilities
+
+ at c Details about memory utilities here.
+
+ at c @node DSR Foreign Libraries
+ at c @comment node-name, next, previous, up
+ at c @section Foreign libraries
+ at c @cindex Foreign libraries
-How foreign libraries are built and used via UFFI. What functions are
-in the .asd files or main lisp code to build & load libraries?
+ at c How foreign libraries are built and used via UFFI. What functions are
+ at c in the .asd files or main lisp code to build & load libraries?
--- /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/27 03:14:55 1.10
+++ /project/elephant/cvsroot/elephant/doc/elephant.texinfo 2007/04/28 02:31:07 1.11
@@ -2,19 +2,27 @@
@c %**start of header
@setfilename elephant.info
@settitle Elephant User Manual
+ at setchapternewpage odd
@c %**end of header
-
- at copying
-Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg.
-Version 0.5, Copyright @copyright{} 2006 Robert L. Read.
-Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read
+ at copying
@quotation
+Elephant System @*
+Original Version, Copyright @copyright{} 2004 Ben Lee and Andrew Blumberg. @*
+Version 0.5, Copyright @copyright{} 2006 Robert L. Read. @*
+Versions 0.6-0.9, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @*
+Portions copyright respective contributors (see @file{CREDITS}). @*
+
+Elephant Manual @*
+Original Version, Copyright @copyright{} 2004 Ben Lee. @*
+Versions 0.5-0.6, Copyright @copyright{} 2006 Robert L. Read. @*
+Current Version, Copyright @copyright{} 2006-2007 Ian Eslick and Robert L. Read @*
+
Permission is granted to copy, distribute and/or modify this document
-under the terms of the GNU Free Documentation License. Portions of
-Elephant are derived from ICU, IBM's Unicode library. Details about
-copyright, license and warranty information can be found in
- at file{LICENSE} or in the ``Copying'' section of this document.
+under the terms of the GNU Free Documentation License. See the
+Copyright and License chapter for details about copyright, license and
+warranty for this manual and the Elephant system.
+
@end quotation
@end copying
--- /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/27 03:14:55 1.11
+++ /project/elephant/cvsroot/elephant/doc/installation.texinfo 2007/04/28 02:31:07 1.12
@@ -9,9 +9,10 @@
* Requirements:: Supported lisps and required libraries.
* Configuring Elephant:: Setting up Elephant and the configuration file.
* Loading Elephant:: Loading Elephant and the data store loading protocol.
-* Berkeley DB:: Installing support for the Berkeley DB data store
+* Berkeley DB:: Installing support for the Berkeley DB data store.
* Berkeley DB Example:: An example of installing and running the Berkeley DB data store.
-* CL-SQL:: Install and connecting to the CL-SQL data store
+* Upgrading Berkeley DB Database:: How to upgrade to a new version of Berkeley DB.
+* CL-SQL:: Install and connecting to the CL-SQL data store.
* CL-SQL Example:: An example of using the CL-SQL data store.
* Elephant on Windows:: More details about running Elephant on Windows
* Test Suites:: How to run and interpret the output of the regression test suite
@@ -275,6 +276,57 @@
@end lisp
in your application.
+ at node Upgrading Berkeley DB Databases
+ at comment node-name, next, previous, up
+ at section Upgrading Berkeley DB Databases
+
+At regular intervals, Elephant will require a specific version of Berkeley DB
+because it does not parse the header files which tend to change with each release.
+The patches are usually minor, but sometimes Elephant also depends on new
+features of Berkeley DB. In this case, you have to update both the Berkeley DB
+database files as well as any data Elephant has built within it. That creates
+some special constraints for upgrading databases.
+
+ at subsection Upgrading to 0.9
+
+This section outlines how to upgrade from Elephant code base version 0.6.0
+using Berkeley DB 4.3.
+
+ at enumerate
+ at item Install BDB 4.5 (keep 4.3 around for now)
+ at item Update my-config.sexp to point to the appropriate BDB 4.5 directories
+ at item Upgrade your database directory to 4.5
+ @itemize
+ @item Run db43_recover in your 0.6 database
+ @item Optional: run db43_archive -d to remove all logs not part of a checkpoint
+ This will make catastrophic recovery impossible, but reduces the amount of data you
+ have to backup.
+ @item Backup your db files and remaining logs
+ @item Run db45_checkpoint -1 in your 0.6 database directory
+ @end itemize
+ at item Migrate 0.6 data to a new 0.6.1 database
+ @itemize
+ @item Open your old database: @code{(setf sc (open-store '(:BDB "/Users/me/db/ele060/")))}
+ @item Run upgrade: @code{(upgrade sc '(:BDB "/Users/me/db/ele061/"))}
+ @end itemize
+ at item Test your new application and report any bugs that arise to elephant-devel at common-lisp.net
+ at end itemize
+
+ at emph{(NOTE: close-store may fail when closing the old 0.6 database, this is OK.)}
+
+ at emph{(NOTE: 64-bit lisps will not successfully upgrade 32-bit 0.6 databases. Use a 32-bit
+ version of your lisp to update to 0.6.1 and then open that database in your 64-bit
+ lisp. There should be no compatibility problems. Best to test your application on
+ a 32-bit lisp if you can, just to be sure.)}
+
+ at subsection Upgrade from Elephant 0.5
+
+Follow the upgrade procedures outlined in Elephant release 0.6.0 to migrate your database
+from 0.5 to 0.6. Then follow the above procedures for upgrading from an 0.6 database.
+
+ at emph{(NOTE: It may not take much work to make 0.9 upgrade directly from 0.5
+However there are so few (none?) 0.5 users that it wasn't deemed worth
+the work given that there's an upgrade path available.)}
@node CL-SQL
@comment node-name, next, previous, up
--- /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/27 03:14:55 1.13
+++ /project/elephant/cvsroot/elephant/doc/reference.texinfo 2007/04/28 02:31:07 1.14
@@ -140,13 +140,15 @@
Persistent collections inherit from @ref{Class elephant:persistent-collection}
and consist of the @ref{Class elephant:btree}, @ref{Class elephant:indexed-btree} and
@ref{Class elephant:btree-index} classes. The following operations are defined
-on most of these classes. More information can be found in @ref{Using BTrees}
-and @ref{Secondary Indices}.
+on most of these classes. More information can be found in @ref{Persistent BTrees}
+and @ref{BTree Indexing}.
@include includes/fun-elephant-make-btree.texinfo
-
@include includes/fun-elephant-get-value.texinfo
- at include includes/fun-elephant-setf-get-value.texinfo
+
+Values are written to a btree using the @code{setf} method on @code{get-value}.
+ at c @include includes/fun-elephant-setf-get-value.texinfo
+
@include includes/fun-elephant-remove-kv.texinfo
@include includes/fun-elephant-existsp.texinfo
@include includes/fun-elephant-drop-btree.texinfo
@@ -178,7 +180,8 @@
@include includes/fun-elephant-make-cursor.texinfo
@include includes/fun-elephant-cursor-close.texinfo
@include includes/fun-elephant-cursor-duplicate.texinfo
- at include includes/fun-elephant-cursor-initialized-p.texinfo
+ at c fixme, not generated by docstrings
+ at c @include includes/fun-elephant-cursor-initialized-p.texinfo
Each of the following methods return multiple values consisting of
@code{(exists? key value)}.
@@ -197,8 +200,8 @@
@node Index Cursor API
@comment node-name, next, previous, up
- at section Cursor API
- at cindex Cursors
+ at section Index Cursor API
+ at cindex Cursor
@cindex Index
@cindex Indices
@cindex API
@@ -224,12 +227,15 @@
@include includes/fun-elephant-cursor-pget-both.texinfo
@include includes/fun-elephant-cursor-pget-both-range.texinfo
- at include includes/fun-elephant-cursor-next-dup.texinfo
@include includes/fun-elephant-cursor-next-nodup.texinfo
- at include includes/fun-elephant-cursor-pnext-dup.texinfo
+ at include includes/fun-elephant-cursor-next-dup.texinfo
@include includes/fun-elephant-cursor-pnext-nodup.texinfo
+ at include includes/fun-elephant-cursor-pnext-dup.texinfo
+
@include includes/fun-elephant-cursor-prev-nodup.texinfo
+ at include includes/fun-elephant-cursor-prev-dup.texinfo
@include includes/fun-elephant-cursor-pprev-nodup.texinfo
+ at include includes/fun-elephant-cursor-pprev-dup.texinfo
@node Transaction API
@comment node-name, next, previous, up
--- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/27 03:14:55 1.9
+++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/04/28 02:31:07 1.10
@@ -711,14 +711,15 @@
hash table indirections with a little macro:
@lisp
-(defmacro def-snapshot-wrapper
- (accessor-name (source-classname target-classname hashname uid))
+(defmacro def-snapshot-wrapper (accessor-name
+ (source-classname target-classname hashname uid))
(with-gensysms (obj key ref)
`(progn
(defmethod ,accessorname :around ((,obj ,source-classname))
(let ((,key (call-next-method)))
(when ,key (gethash ,key ,hashname))))
- (defmethod (setf ,accessorname) :around (,ref (,obj ,source-classname))
+ (defmethod (setf ,accessorname) :around
+ (,ref (,obj ,source-classname))
(cond ((subtypep (type-of ,ref) ,target-classname)
(let ((,key (find-object ,ref ,hashname)))
(if ,key
--- /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/27 03:14:55 1.19
+++ /project/elephant/cvsroot/elephant/doc/tutorial.texinfo 2007/04/28 02:31:11 1.20
@@ -272,7 +272,7 @@
@item @strong{Merge-conflicts in heavily multi-process/threaded situations}.
This is the common read-modify-write problem in all databases. We will talk
-more about this in the @ref{Transactions} section.
+more about this in the @ref{Using Transactions} section.
@end enumerate
@@ -761,7 +761,9 @@
(get-instances-by-range 'friends 'name "Adam" "Devin")
=> (#<Adriana> #<Carlos>)
-(get-instances-by-range 'friend 'birthday (encode-date '(1 1 1974)) (encode-date '(31 12 1984)))
+(get-instances-by-range 'friend 'birthday
+ (encode-date '(1 1 1974))
+ (encode-date '(31 12 1984)))
=> (#<Zaid> #<Adriana>)
(mapc #'print-friend *)
@@ -1207,12 +1209,12 @@
and iterating of duplicate or unique values.
@item @strong{Using the Map Operators}
Mapping operators can be very efficient if properly utilized.
- at item @strong{Using Multiple Stores} Multiple store controllers can
-be open simultaneously. However it does make the code more complex
-and you need to be careful about how you use them to avoid crashes
-and other unpleasant side effects.
- at item @strong{Custom Transaction Architecture} You can implement your
-own version of @code{with-transaction} using the
+ at item @strong{Using Multiple Stores}
+Multiple store controllers can be open simultaneously. However it
+does make the code more complex and you need to be careful about how
+you use them to avoid crashes and other unpleasant side effects.
+ at item @strong{Custom Transaction Architecture}
+You can implement your own version of @code{with-transaction} using the
underlying controller methods for starting, aborting and committing
transactions. You had better know what you are doing, however!
@item @strong{Handling Errors and Conditions}
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/27 03:14:55 1.19
+++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/04/28 02:31:11 1.20
@@ -188,7 +188,7 @@
@item @strong{Merge-conflicts in heavily multi-process/threaded situations}.
This is the common read-modify-write problem in all databases. We will talk
-more about this in the @ref{Transactions} section.
+more about this in the @ref{Transaction Details} section.
@item @strong{Byte Ordering}.
The primitive elements such as integers are written to disk in
@@ -908,7 +908,7 @@
Once a cursor is properly initialized, it can be incremented or
decremented, a simple constant-time operation on BTrees.
- at code{cursor-next} and @{cursor-prev} move the cursor a single step
+ at code{cursor-next} and @code{cursor-prev} move the cursor a single step
forward or back across the sorted key-value pairs. @code{cursor-next}
moves in ascending order, @code{cursor-prev} in descending order.
@@ -960,10 +960,11 @@
class indexing capabilities previously described.
An index is created by using the @code{add-index} function. This
-function takes the @code{indexed-btree} you wish to index, an
-index-name for later retrieval, a key-form which dictates how
-the index populates it's keys as a function of the main btree's
-keys and values.
+function takes the @code{indexed-btree} you wish to index, an symbolic
+name for the index and a key-form which dictates how the index
+populates it's keys as a function of the main btree's keys and values.
+(It is a function of three arguments: the index itself, the key and
+the value).
A simple, contrived example is shown in the figure below:
@@ -1018,23 +1019,37 @@
Operations that have the same behavior, but return primary btree
values and keys are:
- at table @samp
+ at multitable @columnfractions .3 .1 .35
+ at headitem BTree Cursor Function @tab @tab Index Cursor Function
+
@item @code{cursor-first}
- at code{cursor-pfirst}
- at samp{@code{cursor-first}}
+ at tab =>
+ at tab @code{cursor-pfirst}
+
@item @code{cursor-last }
- at code{cursor-plast}
+ at tab =>
+ at tab @code{cursor-plast}
+
@item @code{cursor-current }
- at code{cursor-pcurrent}
+ at tab =>
+ at tab @code{cursor-pcurrent}
+
@item @code{cursor-next }
- at code{cursor-pnext}
+ at tab =>
+ at tab @code{cursor-pnext}
+
@item @code{cursor-prev}
- at code{cursor-pprev}
+ at tab =>
+ at tab @code{cursor-pprev}
+
@item @code{cursor-set}
- at code{cursor-pset}
+ at tab =>
+ at tab @code{cursor-pset}
+
@item @code{cursor-set-range}
- at code{cursor-set-prange}
- at end table
+ at tab =>
+ at tab @code{cursor-set-prange}
+ at end multitable
The big difference between btree cursors and index cursors is that
indices can have duplicate key values. This means we have to choose
@@ -1042,100 +1057,217 @@
a duplicate segment. There are cursor operations for each:
@itemize
- at item Simple move. Standard btree operations plus @code{cursor-pnext} and @code{cursor-pprev}.
- at item Move to next key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup}.
- at item Move to next duplicate.
+ at item Simple move. Standard btree operations work plus @code{cursor-pnext} and @code{cursor-pprev}
+ at item Move to a different key value. @code{cursor-pnext-nodup} and @code{cursor-pprev-nodup}
+ at item Move to next duplicate key value. @code{cursor-pnext-dup} and @code{cursor-pprev-dup}
@end itemize
- at c FINISH
+After incrementing through a set of duplicate items using a
+ at code{xxx-dup} function, the last next operation returns nil
+indicating there are no more duplicates. The consequence of this is
+that the cursor is now uninitialized (@code{cursor-initialized-p}) and
+needs to be reset by a set or set both call.
+
+
@node Multi-threaded Applications
@comment node-name, next, previous, up
@section Multi-threaded Applications
-Berkeley DB plays well with threads and processes. The store
-controller is thread-safe by default, that is, can be shared amongst
-threads. This is enabled by the @code{:thread} keyword argument which
-defaults to true. Transactions may not be shared amongst threads
-except serially. One thing which is NOT thread and process safe is
-recovery, which should be run when no one is else is talking to the
-database environment.
-
-The following shared regions of Elephant are protected by standard locks:
-- buffer stream pool
-- access to serializer circular buffers
-- writes to connection db
-- where else?
+Elephant is thread-safe by design. Users should not have to think
+about threading except to follow a couple of simple rules.
+
+ at enumerate
+ at item Do not perform transactions across multiple threads
+ at item Do not perform add/remove index operations on indexed-btrees
+in more than one thread.
+ at end enumerate
+
+This and common coding sense should be sufficient! Elephant's
+internal design for thread safety employs a number of policies
+to try to minimize using lisp locks and simplify analysis of
+multi threaded interactions:
+
+ at enumerate
+ at item @strong{Rely on the thread safety of the data store databases}
+ at item @strong{Ensure transaction isolation}
+ at item @strong{Minimize dependency on thread-local special variables}
+ at item @strong{Protect shared resources for a given store controller}.
+ at item @strong{A use policy for shared objects (above)}
+ at end enumerate
+
+ at subsection Shared Resources
+
+Elephant has a few shared resources which are protected by standard locks.
+These are:
+ at itemize
+ at item The store controller connection table
+ at item The instance cache
+ at item The circularity buffer pool for the serializer
+ at item The buffer-stream pool in memutils
+ at end itemize
+
+In some cases, and on some lisp platforms, we try to use a fast lock
+strategy for frequently accessed items (the resource pools and
+instance cache especially).
+
+ at subsection Data Store Thread Safety and Transactions
+
+Both CLSQL and Berkelely DB backends are thread safe. In CLSQL this
+is by ensuring that every thread has it's own handle into the SQL
+libraries or sockets. Berkeley DB is reentrant and handles locking
+internally.
+
+Elephant depends on these guarantees especially for the isolation
+properties of transactions. All operations in the context of a
+given transaction should be isolated and atomic. It is important
+that a transaction not be shared across threads, however.
+
+ at subsection Minimize Dependency on Thread-Local Specials
+
+Elephant uses several global variables as default arguments. Most
+of these were removed leaving only a couple to handle:
+
+ at itemize
+ at item @strong{@code{*store-controller*}}. Store controller objects can
+be shared between threads and if a user resets this variable in a local
+thread to another controller, there is no problem with that either.
+Users of multiple concurrent stores can specify the store controller to
+all elephant API commands that don't get it from a persistent object
+implicitely.
+ at code{*current-transaction*}. This is always set to the proper null
+value globally and should not be reset in local threads. Instead,
+transactions take place in a dynamic context that rebinds this variable
+as a special with the current transaction. This allows for a dynamic
+transaction stack for data stores that can nest transactions or when
+two datastores are both doing transactions concurrently.
+ at end itemize
- at c *** FINISH ***
@node Transaction Details
@comment node-name, next, previous, up
@section Transaction Details
+Transactions are dynamic contexts in which all side effects to
+persistent slots and other persistent objects such as BTrees are
+guaranteed to have the ACID properties: atomicity, consistency,
+isolation and durability. On a normal exit from context, the
+side effects are committed as a group. On a non-local exit,
+the transaction is aborted.
+
+For most users, the tutorial section @ref{Using Transactions} is the
+best introduction to transactions. This section adds to that by
+exposing some of the details of how it is implemented.
+
+To reiterate, there are a few important restrictions to adhere to:
+
+ at itemize
+ at item @code{*current-transaction*} is reserved for use by the transaction system. Users should not override, manipulate or close over this variable.
+ at item The body of a transaction cannot throw, signal or jump without aborting the transaction. Any non-local exit is considered an aborting event. Catch signals inside the transaction and return a value instead.
+ at item The dynamic extent of a transaction body must stay within the same thread
+ at end itemize
+
+ at subsection @code{with-transaction} internals
+
+The @code{with-transaction} macro wraps the body expression with an
+anonymous lambda expression. This closure is passed to a call to the
+ at code{execute-transaction} generic function which is specialized to
+the current data store.
+
+The only bookkeeping done by the macro is ensuring that the
+ at code{:parent} argument is checked for the current dynamic transaction
+context. If it is not owned by the default or provided store
+controller, then it is not passed to @code{execute-transaction}. This
+maintains a continuous dynamic stack transactions through the
+with/ensure transaction macros, but allows for a single leaf
+transaction to another store controller.
+
+Be very careful about mixing transactions between store controller.
+This facility was only added to ensure that migrate worked correctly.
+
+The macro processes keywords arguments @code{:store-controller}
+(defaults to @code{*store-controller*}), @code{:parent} (defaults to
+ at code{*current-transaction*}) and @code{:retries} and passes the
+remaining keywords to the call to @code{execute-transaction} allowing
+the user to pass data store specific transaction keywords to their
+preferred data store. The consumed keywords are analyzed and then
+passed on to @code{execute-transaction}.
+
+Any non-standard keywords for a given data store will be ignored by
+other data store implementation of @code{execute-transaction} so
+portable programs should not use keywords that change the semantics of
+the transaction.
+
+ at code{ensure-transaction} only calls @code{execute-transaction} if
+it needs to create a fresh transaction. If the transaction in
+ at code{*current-transaction*} exists and belongs to the store controller
+passed to @code{ensure-transaction} then it merely calls the transaction
+closure, relying on the environment that created the transaction to
+handle any exit procedures and determining whether to abort or commit.
+
+ at code{*current-transaction*} contains transaction records during the
+dynamic execution of a transaction. These records capture any data
+store specific bookkeeping as well as the store-controller that the
+transaction is associated with.
+
+ at subsection @code{execute-transaction} internals
+
+See the @ref{Elephant Architecture} section for details on how
+execute-transaction works. It will provide some deeper insight
+into the transaction system.
+
+ at subsection Building your own transactional framework
+
+Data stores are required to implement three primitive transaction
+methods: @code{controller-start-transaction},
+ at code{controller-abort-transaction} and
+ at code{controller-commit-transaction}. These are wrappers for the data
+store's primitive transaction mechanism. If you use these, it is up
+to you to make sure that you properly manage nested transactions,
+maintain the state of @code{*current-transaction*} handle any
+automated retries you might want, and handle detecting
+
+If you use these, you are on your own - it is easy to make mistakes with
+transactions and create very complex bugs that are hard to track down.
+Most users are much better off sticking with the two transaction macros
+and the underlying @code{execute-transaction} method.
+
+ at subsection Analyzing Dynamic Transaction Behavior
+
You can trace @code{elephant::execute-transaction} to see the sequence
-of calls to @code{execute-transaction} that occur dynamically and
-detect where transactions are and are not happening. We may add some
-transaction diagnosis and tracing tools in the future, such as
-throwing a condition when @code{with-transaction} forms are nested
-dynamically.
-
-;; Transaction architecture:
-;;
-;; User and designer considerations:
-;; - *current-transaction* is reserved for use by dynamic transaction context. The default global
-;; value must always be null (no transaction). Each data storeend can set it to a different parameter
-;; within the dynamic context of an execute-transaction.
-;; - Any closures returned from within a transaction cannot bind *current-transaction*
-;; - Only a normal return value will result in the transaction being committed, any non-local exit
-;; results in a transaction abort. If you want to do something more sophisticated, roll your own
-;; using controller-start-transaction, etc.
-;; - The body of a with or ensure transaction can take any action (throw, signal, error, etc)
-;; knowing that the transaction will be aborted
-;;
+of calls that occur dynamically and detect where and how many
+transactions are and are not happening.
+
+ at c IT WOULD BE GOOD TO EXPAND ON THIS REGARDING HOW TO SOLVE COMMON PROBLEMS
- at c *** FINISH ***
@node Multi-repository Operation
@comment node-name, next, previous, up
@section Multi-repository Operation
-Elephant now keeps a small hashtables that maps ``database specifications'' into
-actual database connections.
+Elephant maintains a small hashtable that maps ``database
+specifications'' into actual @code{store-controller} objects.
+
+The basic strategy is that the ``database specification'' object is
+stored in every persistent object and collection so that the
+repository can be found. In this way, objects that reside in
+different repositories can coexist within the LISP object space,
+allowing data migration or multiple user stores.
+
+All persistent instances store their oid and a store-controller
+reference in internal slots. Slot access and other protocols use
+this to provide access. This executes an auto-transaction or joins
+a surrounding transaction if the @code{transaction-record} in
+ at code{*current-transaction*} matches the store.
+
+When operating with multiple stores and nested transactions there are
+some subtle issues to work around: how to avoid writing one store with
+a transaction created in the context of another. A nested or ensured
+transaction is only indicated in the call to
+ at code{execute-transaction} if the store controllers match, otherwise a
+new transaction for that store is created.
+
+ at c A PICTURE OF THE DYNAMIC CONTEXT WOULD BE USEFUL HERE
-If a database spec is a string, it is assumed to be a BerkeleyDB path.
-If it is a list, it is a assumed to be a CLSQL connection specification.
-For example:
- at lisp
-ELE-TESTS> *testdb-path*
-"/home/read/projects/elephant/elephant/tests/testdb/"
-ELE-TESTS> *testpg-path*
-(:postgresql "localhost.localdomain" "test" "postgres" "")
-ELE-TESTS>
- at end lisp
-The tests now have a function @code{do-all-tests-spec} that take a spec and
-based on its type attempt to open the correct kind of store controller and
-perform the tests.
-
-The routine @code{get-controller} takes this specifiation.
-
-The basic strategy is that the ``database specification'' object is stored in
-every persistent object and collection so that the repository can be found.
-
-In this way, objects that reside in different repositories can coexist within
-the LISP object space, allowing data migration.
-
-;; Multiple stores
-
-;; Multiple store considerations:
-;; - When operating with multiple stores, nested transactions and BDB there are some subtle issues to
-;; work around: how to avoid writing one store with a transaction created in the context of another.
-;; - For many leaf functions: *store-controller* and *current-transaction* have to both be correct;
-;; this requirement may relax in the future
-;; - The following macros accomodate multiple stores by requiring that execute-transaction return a
-;; pair of (store-controller . txn-obj) where txn-obj is owned by the backend and the store-controller
-;; is the store instance it is associated with. A nested or ensured transaction is only indicated
-;; in the call to execute transaction if the store controllers match, otherwise a new transaction
-;; for that store is created
@node Multiple Processes and Distributed Applications
@comment node-name, next, previous, up
More information about the Elephant-cvs
mailing list