[elephant-cvs] CVS update: elephant/doc/Makefile elephant/doc/sql-backend.texinfo elephant/doc/elephant.texinfo elephant/doc/make-ref.lisp elephant/doc/reference.texinfo

[Robert L. Read]

Update of /project/elephant/cvsroot/elephant/doc
In directory common-lisp.net:/tmp/cvs-serv30677/doc

Modified Files:
	elephant.texinfo make-ref.lisp reference.texinfo 
Added Files:
	Makefile sql-backend.texinfo 
Log Message:
This is the big merger from the SQL-BACK-END branch.

Date: Wed Nov 23 18:51:35 2005
Author: rread

Index: elephant/doc/Makefile
diff -u /dev/null elephant/doc/Makefile:1.2
--- /dev/null	Wed Nov 23 18:51:35 2005
+++ elephant/doc/Makefile	Wed Nov 23 18:51:34 2005
@@ -0,0 +1,8 @@
+
+
+
+docs: includes-stuff
+	makeinfo -v --html --force elephant.texinfo
+
+includes-stuff: 
+	cd includes; lisp < ../make-ref.lisp


Index: elephant/doc/sql-backend.texinfo
diff -u /dev/null elephant/doc/sql-backend.texinfo:1.2
--- /dev/null	Wed Nov 23 18:51:35 2005
+++ elephant/doc/sql-backend.texinfo	Wed Nov 23 18:51:34 2005
@@ -0,0 +1,327 @@
+ at c -*-texinfo-*-
+
+ at node SQL back-end
+ at comment node-name, next, previous, up
+ at chapter SQL back-end
+ at cindex SQL back-end
+
+ at menu
+* SQL-Introduction:: The design and status of the SQL back-end extention.
+* Extention Status:: The current status of the SQL back-end extention.
+* Back-compatibility:: Issues if you have already been using Elephant
+* Multi-repository Operation:: Specifying repositories
+* Setting up PostGres:: An example
+* Repository Migration:: How to move objects from one repository to another
+ at end menu
+
+ at node SQL-Introduction
+ at comment node-name, next, previous, up
+ at section SQL-Introduction
+
+Although originally designed as an interface to the BerkeleyDB system,
+the original Elephant system has been experimenetally extended to 
+support the use of relational database management systems as the
+implementation of the persistent store.  This relies on Kevin Rosenberg's
+CL-SQL interface to relational systems.
+
+Although the BerkeleyDB system is an ideal object store for LISP objects,
+one might prefer the licensing of a different system.  For example, at 
+the time of this writing, it is my interpretation that one cannot use
+the BerkeleyDB system behind a public website 
+http://www.sleepycat.com/download/licensinginfo.shtml#redistribute
+unless one releases the entire web application as open source.
+
+Neither the PostGres DBMS nor SQLite 3 has any such restriction.  Elephant itself is released
+under the GPL.  It is somewhat debatable if the GPL allows one to construct
+to construct a non-open-source web application but the preponderance of 
+opinion appears to be that it does.  Thefore using Elephant and the other
+GPLed software that it depends upon allows one to host a a non open-source
+web application.  This might be a reason to use Elephant on PostGres of SQLite rather
+than Elephant on BerkeleyDB.
+
+Other reasons to use a relational database system might include:
+familiarity with those systems, the fact that some part of your application
+needs to use the truly relational aspects of those systems, preference for
+the tools associated with those systems, etc.
+
+The SQL back-end extention of Elephant provides a function for migrating
+data seamlessly between repositories.  That is, one can quite easily move
+data from a BerkeleyDB repository to a PostGres repository, and vice versa.
+In fact, one of the most important aspects of the extention is that it 
+makes Elephant a multi-repository system, rather than a single repository
+system, as addition to allowing different implementation strategies for 
+those repositories.  This offers at least the possiblity than once 
+can develop using one backend, for example BerkeleyDB, and then later
+move to MySQL.
+
+At the time of this writing, the basic strategy for the SQL implementation
+is quite simple.  The same serializer used for the Sleepycat implementation
+is employed, the byte-string is base64 encoded, and placed in a single 
+table which is managed by Elephant.
+
+As of Elephant 0.3, Elephant has been tested to work with both Postgres, and 
+SQLite 3, thanks do Dan Knapp.
+
+ at node Extention Status
+ at comment node-name, next, previous, up
+ at section Extention Status
+
+As far as is known at this writing, all functionality except nested transaction
+support and cursor-put's that is supported by the BerkeleyDB backend is supported by the CL-SQL
+based back-end.  Concurrency and transaction atomicity has not been tested well
+for the CL-SQL based system.
+
+Additionally, it is NOT the case that the Elephant system currently provides
+transaction support across multiple repositories; it provides the transaction
+support provided by the underlying repository to the user in a per-repository
+basis.
+
+The PostGres backend is as currently employed is about 5 times slower than 
+the BerkeleyDB backend.  This could probably change with continued development.
+
+CL-SQL supports a lot of DBMS systems, but only PostGres has been tested.
+
+The SQL back-end extention has only been tested under SBCL 0.8.18.
+
+The SQL back-end is as easy to use as the BerkeleyDB back-end.  However,
+the multi-repository version somewhat complicates the underlying 
+persistent object management.  At the time of this writing, the 
+community has not decided if this extention will be a part of 
+Elephant proper or a separate branch; if it is not made a part of 
+Elephant proper, a user might prefer the simpler (and better maintained?)
+system if they only want to use the BerkeleyDB back-end.
+
+
+ at node Back-compatibility
+ at comment node-name, next, previous, up
+ at section Back-compatibility
+
+The CL-SQL based extention is very back-compatible with any existing Elephant
+application, except for two items.
+
+First, the routines ``build-btree'' and ``build-index-btree'' should be used
+in place of the previous approach to direct calls to make-instance.  This is
+necessary, because the underlying class of the object depends on what repository
+it is stored in.  These routines, like make-instance on persistent objects directly,
+allow you to specify the store controller at creation time.  However, build-btree
+and build-index-btree will use the global *store-controller* if no keyword
+argument is provided.
+
+Secondly, in addition to executing:
+ at lisp
+(asdf:operate 'asdf:load-op :elephant)
+ at end lisp
+to load elephant, one must at least one of:
+ at lisp
+(asdf:operate 'asdf:load-op :ele-clsql)
+(asdf:operate 'asdf:load-op :ele-bdb)
+ at end lisp
+
+To use SQLLite3, you must execute:
+ at lisp
+(asdf:operate 'asdf:load-op :ele-sqlite3)
+ at end lisp
+
+depending on whether or not you wish to use the clsql backend or the BerkeleyDB 
+backend, or both.
+
+You will have to have the CL-SQL package installed.  Following the 
+documentation for CL-SQL under the section ``How CLSQL finds and loads foreign
+libraries'' you may need to do something like:
+ at lisp
+(clsql:push-library-path "/usr/lib/")
+ at end lisp
+
+before doing
+ at lisp
+(asdf:oos 'asdf:load-op :clsql-postgresql-socket)
+ at end lisp
+
+in order for clsql to find the PostGres library libpq.so, for example.
+
+Without modifcation, Elephant uses this as it's lib path:
+ at lisp
+/usr/local/share/common-lisp/elephant-0.3/
+ at end lisp
+
+So you could put a symbolic link to libpq.so there, where libmemutil.so and 
+libsleepycat.so will also reside.
+
+Versions of CL-SQL older than 3.2.3 might requie something different.
+
+ at node Multi-repository Operation
+ at comment node-name, next, previous, up
+ at section Multi-repository Operation
+
+Elephant now keeps a small hashtables that maps ``database specifications'' into
+actual database connections.
+
+If a database spec is a string, it is assumed to be a BerkeleyDB path.
+If it is a list, it is a assumed to be a CL-SQL connection specification.
+For example:
+ at lisp
+ELE-TESTS> *testdb-path*
+"/home/read/projects/elephant/elephant/tests/testdb/"
+ELE-TESTS> *testpg-path*
+(:postgresql "localhost.localdomain" "test" "postgres" "")
+ELE-TESTS> 
+ at end lisp
+
+
+The tests now have a function @code{do-all-tests-spec} that take a spec and 
+based on its type attempt to open the correct kind of store controller and 
+perform the tests.
+
+The routine @code{get-controller} takes this specifiation.
+
+The basic strategy is that the ``database specification'' object is stored in
+every persistent object and collection so that the repository can be found.
+
+In this way, objects that reside in different repositories can coexist within
+the LISP object space, allowing data migration.
+
+
+
+
+ at node Setting up PostGres
+ at comment node-name, next, previous, up
+ at section Setting up PostGres
+
+
+To set up a PostGres based back end, you should:
+
+ at enumerate
+ at item Install postgres and make sure postmaster is running.
+
+ at item Create a database called ``test'' and set its permissions
+to be reached by whatever connection specification you intend to use.  The
+tests use: 
+
+ at lisp
+(defvar *testpg-path*
+'(:postgreql "localhost.localdomain" "test" "postgres" ""))
+ at end lisp
+
+meaning that connections must be allowed to the database test, user ``postgres'',
+no password, connected from the same machine ``localhost.localdomain''.
+(This would be changed to something more secure in a real application.)
+Typically you edit the file : pg_hba.conf to enable various kinds of connections
+in postgres.
+
+ at item  Be sure to enable socket connection to postgres when you invoke the postmaster.
+
+ at item  Test that you can connect to the database with these credentials by running:
+
+ at code{ psql -h 127.0.0.1 -U postgres test}
+
+Before you attempt to connect with Elephant.
+ at end enumerate
+
+
+meaning that connections must be allowed to the database test, user ``postgres'',
+no password, connected from the same machine ``localhost.localdomain''.
+(This would be changed to something more secure in a real application.)
+
+Furthermore, you must grant practically all creation/read/write privileges 
+to the user postgres on this schema, so that it can construct the tables 
+it needs.
+
+Upon first opening a CL-SQL based store controller, the tables, indexes,
+sequences, and so on needed by the Elephant system will be created in the 
+schema named ``test'' automatically.
+
+To run the tests, execute:
+
+ at lisp
+(asdf:operate 'asdf:load-op :elephant)
+(asdf:operate 'asdf:load-op :ele-clsql)
+(asdf:oos 'asdf:load-op :clsql-postgresql-socket)
+(in-package "ELEPHANT-TESTS")
+(do-all-tests-spec *testpg-path*)
+ at end lisp
+
+This should produce a small number of errors (about 7) for those test having 
+to do with migration and the BerkeleyDB system specifically.
+
+If you execute:
+
+ at lisp
+(asdf:operate 'asdf:load-op :ele-bdb)
+ at end lisp
+
+Then connection to the BerkeleyDB system will be enabled, and you should 
+be able to execute both
+
+ at lisp
+(do-all-tests-spec *testpg-path*)
+(do-all-tests-spec *testdb-path*)
+ at end lisp
+
+with no errors in either case.
+
+At present the system has only been tested under PostGres. Some code
+parametrization would be required to work with other databases.
+
+Setting up SQLite3 is even easier.  Install SQLite3 (I had to use
+the source rather than the binary install, in order to get the dynamic
+libraries constructed.)
+
+An example use of SQLLite3 would be:
+ at lisp
+(asdf:operate 'asdf:load-op :elephant)
+(asdf:operate 'asdf:load-op :ele-clsql)
+(asdf:operate 'asdf:load-op :ele-sqlite3)
+(in-package "ELEPHANT-TESTS")
+(setq *test-path-primary* '(:sqlite3 "testdb"))
+(do-all-tests-spec *test-path-primary*)
+ at end lisp
+
+The file RUNTESTS.lisp, although possibly not exactly what you want,
+contains useful example code.
+
+You can of course migrate between the three currently supported repository
+strategies in any combination: BDB, Postgresql, and SQLite3.
+
+In all probability, other relational datbases would be very easy to 
+support but have not yet been tested.  The basic pattern of 
+the ``path'' specifiers is (cons clsqal-database-type-symbol (normal-clsql-connection-specifier)).
+
+ at node Repository Migration
+ at comment node-name, next, previous, up
+ at section Repository Migration
+
+
+This version of Elephant supports migration betwen store controllers,
+whether of the same implementation strategy or not.
+
+The tests @code{migrate1} - @code{migrate5} are demonstrations of this techinque.
+
+The functions for performing these migrations are:
+
+ at code{migraten-pobj}
+
+The name of this function is meant to imply that it is 
+destructive of the object in question, mutating it to 
+point at the new repository.
+
+Which requies that you provide a copy-function to copy whatever 
+slots you want from the persistent object as deeply or as shallowly 
+as you desire.
+
+Data collections (btree's) can be move with the function:
+
+ at code{migrate}
+
+A simple object that does not inherit from ``persistent'' but is 
+attached to a key (on the root) can be copied with the routine
+
+ at code{copy-from-key}
+
+It is hoped that these routines would allow, with some labor,
+a user to use one repository, and later decide to start using 
+a different implementation strategy, and easily migrate the 
+objects to the the new repository.  The old repository could
+then be abandoned, or multiple repositories could be used
+at the same time.
+
+ 


Index: elephant/doc/elephant.texinfo
diff -u elephant/doc/elephant.texinfo:1.1 elephant/doc/elephant.texinfo:1.2
--- elephant/doc/elephant.texinfo:1.1	Sun Sep 19 19:44:43 2004
+++ elephant/doc/elephant.texinfo	Wed Nov 23 18:51:34 2005
@@ -43,6 +43,7 @@
 * Introduction:: Introducing Elephant!
 * Tutorial::     A leisurely walk-through.
 * Reference::    API documentation.
+* SQL back-end:: CL-SQL based implementation
 * Design Notes:: Internals.
 * Copying::      Your rights and freedoms.
 * Concept Index::
@@ -56,6 +57,7 @@
 @include tutorial.texinfo
 @include reference.texinfo
 @include notes.texinfo
+ at include sql-backend.texinfo
 @include copying.texinfo
 
 @node Concept Index


Index: elephant/doc/make-ref.lisp
diff -u elephant/doc/make-ref.lisp:1.1 elephant/doc/make-ref.lisp:1.2
--- elephant/doc/make-ref.lisp:1.1	Sun Sep 19 19:44:43 2004
+++ elephant/doc/make-ref.lisp	Wed Nov 23 18:51:34 2005
@@ -1,7 +1,10 @@
 (require 'asdf)
 (require 'elephant)
-(load "docstrings.lisp")
+(load "../docstrings.lisp")
 
 (defun make-docs ()
-  (when (check-complete)
-    (sb-texinfo:generate-includes #p"includes" (find-package :ele))))
\ No newline at end of file
+;;  (when (check-complete)
+  (when t
+    (sb-texinfo:generate-includes #p"includes" (find-package :ele))))
+
+(make-docs)


Index: elephant/doc/reference.texinfo
diff -u elephant/doc/reference.texinfo:1.1 elephant/doc/reference.texinfo:1.2
--- elephant/doc/reference.texinfo:1.1	Sun Sep 19 19:44:42 2004
+++ elephant/doc/reference.texinfo	Wed Nov 23 18:51:34 2005
@@ -43,7 +43,7 @@
 
 @include includes/var-elephant-star-auto-commit-star.texinfo
 @include includes/var-elephant-star-current-transaction-star.texinfo
- at include includes/fun-elephant-start-transaction.texinfo
+ at include includes/fun-elephant-start-ele-transaction.texinfo
 @include includes/fun-elephant-commit-transaction.texinfo
 @include includes/fun-elephant-abort-transaction.texinfo