[elephant-cvs] CVS elephant/doc
ieslick
ieslick at common-lisp.net
Sat Mar 24 13:55:15 UTC 2007
Update of /project/elephant/cvsroot/elephant/doc
In directory clnet:/tmp/cvs-serv32267
Modified Files:
Makefile make-ref.lisp
Added Files:
data-store-reference.texinfo elephant-design.texinfo
lisp-data-store.texinfo scenarios.texinfo style.css
user-guide.texinfo
Removed Files:
notes.texinfo
Log Message:
First batch of edits for new user manual
--- /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 12:16:02 1.3
+++ /project/elephant/cvsroot/elephant/doc/Makefile 2007/03/24 13:55:15 1.4
@@ -1,8 +1,10 @@
+all: docs
+includes-stuff:
+ cd includes; sbcl < ../make-ref.lisp
docs: includes-stuff
- makeinfo -v --html --force elephant.texinfo
+ makeinfo -v --html --css-include=style.css --force elephant.texinfo
+ makeinfo -v --html --css-include=style.css --force --no-split elephant.texinfo
-includes-stuff:
- cd includes; sbcl < ../make-ref.lisp
--- /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 12:16:02 1.3
+++ /project/elephant/cvsroot/elephant/doc/make-ref.lisp 2007/03/24 13:55:15 1.4
@@ -1,10 +1,21 @@
(require 'asdf)
(asdf:operate 'asdf:load-op 'elephant-tests)
-(sb-posix:chdir "/Users/eslick/Work/fsrc/elephant-cvs/doc/includes/")
-(load "/Users/eslick/Work/fsrc/elephant-cvs/doc/docstrings.lisp")
+(defparameter include-dir-path
+ (namestring
+ (merge-pathnames
+ #p"doc/includes/"
+ (asdf:component-pathname (asdf:find-system 'elephant-tests)))))
+
+(defparameter docstrings-path
+ (namestring
+ (merge-pathnames
+ #p"doc/docstrings.lisp"
+ (asdf:component-pathname (asdf:find-system 'elephant-tests)))))
+
+(sb-posix:chdir include-dir-path)
+(load docstrings-path)
(defun make-docs ()
-;; (when (check-complete)
(when t
(elephant:open-store elephant-tests::*testbdb-spec*)
(make-instance 'elephant::persistent-collection)
--- /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/data-store-reference.texinfo 2007/03/24 13:55:15 1.1
@c -*-texinfo-*-
@node Data Store API Reference
@comment node-name, next, previous, up
@chapter Data Store API Reference
@cindex Data Store API Reference
@cindex Data Store
@cindex API Reference
These are the functions that need to be overridden to implement
support for a data store backend. Included are the exported elephant
functions that need methods defined on them. Some functions here are
utilities from the main elephant package that support store
implementations. Migration, class indices and query interfaces are
implemented on top of the store API and require no special support by
implementors.
@menu
* Registration:: Register the backend to parse controller specifications
* Store Controllers:: Subclassing the store controller.
* Slot access:: Support for metaprotocol slot access.
* Collections:: BTrees and indices.
* Cursors:: Traversing BTrees.
* Transactions:: Transaction implementation.
* Multithreading:: Multithreading considerations.
* Serialization:: Facilities for serializing objects.
* C Utilities:: Writing primitive C types.
* Foreign libraries:: Using UFFI and ASDF to build or link foreign libraries
@end menu
@node Registration
@comment node-name, next, previous, up
@section Registration
@cindex Registration
@include includes/fun-elephant-register-backend-con-init.texinfo
@include includes/fun-elephant-lookup-backend-con-init.texinfo
@node Store Controllers
@comment node-name, next, previous, up
@section Store Controllers
@cindex Store Controllers
Subclass store-controller and implement store and close controller
which are called by open-store and close-store respectively.
@include includes/fun-elephant-store-controller.texinfo
@include includes/fun-elephant-backend-open-controller.texinfo
@include includes/fun-elephant-backend-close-controller.texinfo
The slots for these accessors must be initialized.
@include includes/fun-elephant-backend-database-version.texinfo
@include includes/fun-elephant-backend-controller-serialize.texinfo
@include includes/fun-elephant-backend-controller-deserialize.texinfo
@include includes/fun-elephant-backend-root.texinfo
@include includes/fun-elephant-backend-class-root.texinfo
These functions are important utilities for implementing
store-controllers.
@include includes/fun-elephant-backend-oid.texinfo
@include includes/fun-elephant-backend-get-con.texinfo
@include includes/fun-elephant-backend-next-oid.texinfo
@include includes/fun-elephant-backend-connection-is-indeed-open.texinfo
@node Slot Access
@comment node-name, next, previous, up
@section Slot Access
@cindex Slot Access
These functions are called by the metaclass protocol to support
operations on persistent class slots.
@include includes/fun-elephant-backend-persistent-slot-writer.texinfo
@include includes/fun-elephant-backend-persistent-slot-reader.texinfo
@include includes/fun-elephant-backend-persistent-slot-boundp.texinfo
@include includes/fun-elephant-backend-persistent-slot-makunbound.texinfo
@node Collections
@comment node-name, next, previous, up
@section Collections
@cindex Collections
@c #:btree #:btree-index #:indexed-btree
@c #:build-indexed-btree #:build-btree #:existsp
@c #:map-indices
@node Cursors
@comment node-name, next, previous, up
@section Cursors
@cindex Cursors
@c #:cursor
@c #:cursor-btree
@c #:cursor-oid
@c #:cursor-initialized-p
@node Transactions
@comment node-name, next, previous, up
@section Transactions
@cindex Transactions
@c #:*current-transaction*
@c #:make-transaction-record
@c #:transaction-store
@c #:transaction-object
@c #:execute-transaction
@c #:controller-start-transaction
@c #:controller-commit-transaction
@c #:controller-abort-transaction
@node Multithreading
@comment node-name, next, previous, up
@section Multithreading
@cindex Multithreading
@node Serialization
@comment node-name, next, previous, up
@section Serialization
@cindex Serialization
@c #:deserialize #:serialize
@c #:serialize-symbol-complete
@c #:deserialize-from-base64-string
@c #:serialize-to-base64-string
@node Memory utilities
@comment node-name, next, previous, up
@section Memory utilities
@cindex Memory utilities
@node Foreign libraries
@comment node-name, next, previous, up
@section Foreign libraries
@cindex Foreign libraries
--- /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/elephant-design.texinfo 2007/03/24 13:55:15 1.1
Debugger entered--Lisp error: (void-variable Design)
eval(Design)
eval-last-sexp-1(nil)
eval-last-sexp(nil)
call-interactively(eval-last-sexp)
--- /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/lisp-data-store.texinfo 2007/03/24 13:55:15 1.1
@c -*-texinfo-*-
@node Lisp Data Store
@comment node-name, next, previous, up
@chapter Lisp Data Store
@cindex Lisp Data Store
@cindex Data Store
@cindex API Reference
--- /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/scenarios.texinfo 2007/03/24 13:55:15 1.1
@c -*-texinfo-*-
@node Usage Scenarios
@comment node-name, next, previous, up
@chapter Usage Scenarios
@cindex Usage Scenarios
Sorry, haven't written this section yet.
Simple file replacement and indexing
- Keep track of ordinary objects, ignore metaprotocol
Persist system objects
- Intermingle persistent objects and regular objects
- Look up objects using class indices
Full database system
- storage, rich data models, references, queries, etc
Multithreaded web applications
- DB + multithreading
Object-oriented data storage, large graph traversals
--- /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/style.css 2007/03/24 13:55:15 1.1
--- /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/user-guide.texinfo 2007/03/24 13:55:15 1.1
@c -*-texinfo-*-
@node User Guide
@comment node-name, next, previous, up
@chapter User Guide
@cindex User Guide
@menu
* The Store Controller:: Behind the curtain.
* Serialization details:: The devil hides in the details.
* Reachability:: Determining liveness in a store.
* Persistent objects:: All the dirt on persistent objects.
* Class indices:: In-depth discussion about indexing persistent indices.
* Querying persistent instances:: Retrieving instances of classes.
* Using BTrees:: Using the native btree.
* Secondary Indices:: Alternative ways to index collections.
* Using Cursors:: Low-level access to BTrees.
* Transaction details:: Develop a deeper understanding of transactions and avoid the pitfalls.
* Repository Migration and Upgrade:: How to move objects from one repository to another.
* Garbage collection:: How to recover storage and OIDs in long-lived repositories.
* Performance tuning:: How to get the most from Elephant.
@end menu
@node Persistent objects
@comment node-name, next, previous, up
@section Persistent Objects
Finally, if you for some reason make an instance with a specified OID
which already exists in the database, @code{initargs} take precedence
over values in the database, which take precedences over
@code{initforms}.
Also currently there is a bug where
@code{initforms} are always evaluated, so beware.
(What is the current model here?)
@node The Store Controller
@comment node-name, next, previous, up
@section The Store Controller
What is @code{open-store} doing? It creates a @code{store-controller}
object, and sets the special @code{*store-controller*} to point to it.
The store controller holds the handles to the database environment and
tables, and some other bookkeeping. If for some reason you need to
run recovery on the database (see sleepycat docs) you can specify that
with the @code{:recover} and @code{:recover-fatal} keys.
To create one by hand one can do,
@lisp
* (setq *store-controller* (make-instance 'store-controller :path "testdb"))
=> #<STORE-CONTROLLER @{49252F75@}>
* (open-controller *store-controller*)
=> #<STORE-CONTROLLER @{49252F75@}>
@end lisp
but
@lisp
* (open-store "testdb"))
@end lisp
is the preferred mechanism.
This opens the environment and database. The @code{persistent-*} objects
reference the @code{*store-controller*} special. (This is in part because
slot accessors can't take additional arguments.) If for some reason
you want to operate on 2 store controllers, you'll have to do that by
flipping the @code{*store-controller*} special.
@code{close-store} closes the store controller. Alternatively
@code{close-controller} can be called on a handle. Don't forget to do
this or else you may need to run recovery later. There is a
@code{with-open-controller} macro. Opening and closing a controller
is very expensive.
@node{Using BTrees}
@comment node-name, next, previous, up
@section Using BTrees
The btree class are to hash-tables as persistent-objects are to
ordinary objects. BTrees have a hash-table-like interface, but store
their keys and values directly as rows in a Sleepycat BTree. Btrees
may be persisted simply by their OID. Hence they have all the nice
properties of persistent objects: identity, fast serialization /
deserialization, no merge conflicts.....
(defvar *friends-birthdays* (make-btree))
=> *FRIENDS-BIRTHDAYS*
(add-to-root "friends-birthdays" *friends-birthdays*)
=> #<BTREE {4951CF6D}>
(setf (get-value "Andrew" *friends-birthdays*)
(encode-universal-time 0 0 0 22 12 1976))
=> 2429071200
(setf (get-value "Ben" *friends-birthdays*)
(encode-universal-time 0 0 0 14 4 1976))
=> 2407298400
(get-value "Andrew" *friends-birthdays*)
=> 2429071200
=> T
(decode-universal-time *)
=> 0
0
0
22
12
1976
2
NIL
6
Because of serialization semantics, BTrees hash on a value, not
identity. This is probably ok for strings, numbers, and persistent
things, but may be strange for other values.
@node Repository Migration
@comment node-name, next, previous, up
@section Repository Migration
This version of Elephant supports migration between store controllers of
any backend type.
The tests @code{migrate1} - @code{migrate5} are demonstrations of this
capability.
There is a single generic function @code{migrate} that is used to copy
different object types to a target repository. It is assumed that typically
migrate will be called on two repositories and all live objects (those
reachable in the root or class-root) will be copied to the target repository
via recursive calls to migrate for specific objects.
When persistent instances are copied, their internal pointer will be updated
to point to the new repository so after migration the lisp image should be
merely updated to refer to the target repository in the *store-controller*
variable or whatever variable the application is using to store the primary
controller instance.
There are some limitations to the current migration implementation:
@enumerate
@item Migrate currently will not handle circular list objects
@item Migrate does not support arrays with nested persistent objects
@item Indexed classes only have their class index copied if you use the
top level migration. Objects will be copied without slot data if you
try to migrate an object outside of a store-to-store migration due to
the class object belonging to one store or another
@item Migrate assumes that after migration, indexed classes belong to the
target store.
@item In general, migration is a one-time activity and afterwards (or after
a validation test) the source store should be closed. Any failures
in migration should then be easy to catch.
@item Each call to migration will be good about keeping track of already
copied objects to avoid duplication. Duplication shouldn't screw
up the semantics, just add storage overhead but is to be avoided.
However this information is not saved between calls and there's no
other way to do comparisons between objects across stores (different
oid namespaces) so user beware of the pitfalls of partial migrations...
@item Migrate keeps a memory-resident hash of all objects; this means
you cannot currently migrate a store that has more data than your
main memory. (This could be fixed by keeping the oid table in
the target store and deleting it on completion)
@item Migration does not maintain OID equivalence so any datastructures which
index into those will have to have a way to reconstruct themselves (better
to keep the object references themselves rather than oids in general)
but they can overload the migrate method to accomplish this cleanly
@end enumerate
Users can customize migration if they create unusual datastructures that
are not automatically supported by the existing @code{migrate} methods.
For example, a datastructure that stores only object OIDs instead of
serialized object references will need to overload migrate to ensure
that all referenced objects are in fact copied (otherwise the OIDs will
just be treated as fixnums potentially leaving dangling references.
To customize migration overload a version of migrate to specialize on
your specific persistent class type.
@lisp
(defmethod migrate ((dst store-controller) (src my-class)))
@end lisp
In the body of this method you can call @code{(call-next-method)}
to get a destination repository object with all the slots copied over
to the target repository which you can then overwrite. To avoid the
default persistent slot copying, bind the dynamic variable
@code{*inhibit-slot-writes*} in your user method using
@code(with-inhibited-slot-copy () ...)} a convenience macro.
@node Threading
@comment node-name, next, previous, up
@section Threading
Sleepycat plays well with threads and processes. The store controller
is thread-safe by default, that is, can be shared amongst threads.
This is set by the @code{:thread} key. Transactions may not be shared
amongst threads except serially. One thing which is NOT thread and
process safe is recovery, which should be run when no one is else is
talking to the database environment. Consult the Sleepycat docs for
more information.
Elephant uses some specials to hold parameters and buffers. If you're
using a natively threaded lisp, you can initialize these specials to
thread-local storage by using the @code{run-elephant-thread} function,
assuming your lisp creates thread-local storage for let-bound
specials. (This functionality is currently broken)
[72 lines skipped]
More information about the Elephant-cvs
mailing list