<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
Dear Elephant Devel team,<BR>
<BR>
Here is my good-enough-to-be-released extension of elephant to use CL-SQL as a back-end (with PostGres).<BR>
<BR>
Below is the documentation that I wrote (It's in the texinfo, but I thought you might like to read it directly, so I pasted it here.)<BR>
Please find attached a tar file containing all of the new files, and additionally a file containing the cvs diff of the files that<BR>
already existed in the repository.<BR>
<BR>
If your interested in this, you should probably read the documentation below. The code change is so great that <BR>
I doubt you can see much from the diff; the best way to really analyze would be to create a separate branch in CVS and <BR>
allow me to commit my stuff there, so that one can easily checkout the whole branch.<BR>
<BR>
In a nutshell, I have extended Elephant to utilized CL-SQL (on top of PostGres) as a back-end in addition to <BR>
BerkeleyDB. Some people may think this pointless; others may be pleased with the more permissive licensing of <BR>
PostGres, or see this as a step to supporting, for example, SQLite. Addtionally, this version is a "mutli-repository" version<BR>
in that many repositories can be open at the same time, and data can be migrated between them, without regard to <BR>
the implementation strategy that underlies the repository.<BR>
<BR>
I commend the Elephant developers on the good set of tests they had; I have expanded them to allow testing<BR>
on multiple repository types and to test migration.<BR>
<BR>
I know the current owners of Elephant are looking for a new owner. That, and the fact that some people might<BR>
like Elephant the way it is and hate the apparent complexity that I have added, or might really like what I have done,<BR>
creates a complicated set of questions we have to answer:<BR>
<BR>
1) Who will own Elephant?<BR>
2) Should this be the next version of Elephant, or should it be a fork (that is, a completely different project, maybe<BR>
"Bignose" or something)?<BR>
<BR>
I don't know how many people use Elephant or are on this list. I enjoyed it until I hit the BerkeleyDB licensing<BR>
restriction. I definitely plan to use this work that I have done in a website moving forward, and will be maintaining<BR>
it, one way or the other.<BR>
<BR>
I have not yet offered to own Elephant, since I am not an expert LISP coder, have never managed a large open-source<BR>
project before, don't know who is using it, don't plan to pay for any lisp system, and don't know how much time is <BR>
involved in maintaining such a package, and am not an expert on BerkeleyDB. However, if nobody else has volunteered<BR>
and Ben can't do it, I suppose that I must be better than nothing at all.<BR>
<BR>
Please express some sort of opinion. I spent 5 solid weeks on this; which in hindsight was probably a three-week <BR>
waste of time compared to just implementing my own serializer, which would have served my purpose.<BR>
However, now that it is presentable, I certainly hope that someone in addition to myself will benefit from it.<BR>
<BR>
<BR>
<BR>
<BR>
<B><FONT SIZE="5">5 SQL back-end</FONT></B>
<UL>
<LI>SQL-Introduction: The design and status of the SQL back-end extention.
<LI>Extention Status: The current status of the SQL back-end extention.
<LI>Back-compatibility: Issues if you have already been using Elephant
<LI>Multi-repository Operation: Specifying repositories
<LI>Setting up PostGres: An example
<LI>Repository Migration: How to move objects from one repository to another
<LI><B><FONT SIZE="4">5.1 SQL-Introduction</FONT></B>
</UL>
Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to support the use of relational database management systems as the implementation of the persistent store. This relies on Kevin Rosenberg's CL-SQL interface to relational systems. <BR>
<BR>
Although the BerkeleyDB system is an ideal object store for LISP objects, one might prefer the licensing of a different system. For example, at the time of this writing, it is my interpretation that one cannot use the BerkeleyDB system behind a public website http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. <BR>
<BR>
The PostGres DBMS has no such restriction. Elephant itself is released under the GPL. It is somewhat debatable if the GPL allows one to construct to construct a non-open-source web application but the preponderance of opinion appears to be that it does. Thefore using Elephant and the other GPLed software that it depends upon allows one to host a a non open-source web application. This might be a reason to use Elephant on PostGres rather than Elephant on BerkeleyDB. <BR>
<BR>
Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application needs to use the truly relational aspects of those systems, preference for the tools associated with those systems, etc. <BR>
<BR>
The SQL back-end extention of Elephant provides a function for migrating data seamlessly between repositories. That is, one can quite easily move data from a BerkeleyDB repository to a PostGres repository, and vice versa. In fact, one of the most important aspects of the extention is that it makes Elephant a multi-repository system, rather than a single repository system, as addition to allowing different implementation strategies for those repositories. This offers at least the possiblity than once can develop using one backend, for example BerkeleyDB, and then later move to MySQL. <BR>
<BR>
At the time of this writing, the basic strategy for the SQL implementation is quite simple. The same serializer used for the Sleepycat implementation is employed, the byte-string is base64 encoded, and placed in a single table which is managed by Elephant. <BR>
<BR>
<B><FONT SIZE="4">5.1 SQL-Introduction</FONT></B><BR>
Although originally designed as an interface to the BerkeleyDB system, the original Elephant system has been experimenetally extended to support the use of relational database management systems as the implementation of the persistent store. This relies on Kevin Rosenberg's CL-SQL interface to relational systems. <BR>
<BR>
Although the BerkeleyDB system is an ideal object store for LISP objects, one might prefer the licensing of a different system. For example, at the time of this writing, it is my interpretation that one cannot use the BerkeleyDB system behind a public website http://www.sleepycat.com/download/licensinginfo.shtml#redistribute unless one releases the entire web application as open source. <BR>
<BR>
The PostGres DBMS has no such restriction. Elephant itself is released under the GPL. It is somewhat debatable if the GPL allows one to construct to construct a non-open-source web application but the preponderance of opinion appears to be that it does. Thefore using Elephant and the other GPLed software that it depends upon allows one to host a a non open-source web application. This might be a reason to use Elephant on PostGres rather than Elephant on BerkeleyDB. <BR>
<BR>
Other reasons to use a relational database system might include: familiarity with those systems, the fact that some part of your application needs to use the truly relational aspects of those systems, preference for the tools associated with those systems, etc. <BR>
<BR>
The SQL back-end extention of Elephant provides a function for migrating data seamlessly between repositories. That is, one can quite easily move data from a BerkeleyDB repository to a PostGres repository, and vice versa. In fact, one of the most important aspects of the extention is that it makes Elephant a multi-repository system, rather than a single repository system, as addition to allowing different implementation strategies for those repositories. This offers at least the possiblity than once can develop using one backend, for example BerkeleyDB, and then later move to MySQL. <BR>
<BR>
At the time of this writing, the basic strategy for the SQL implementation is quite simple. The same serializer used for the Sleepycat implementation is employed, the byte-string is base64 encoded, and placed in a single table which is managed by Elephant. <BR>
<BR>
<B><FONT SIZE="4">5.3 Back-compatibility</FONT></B><BR>
The CL-SQL based extention is very back-compatible with any existing Elephant application, except for two items. <BR>
<BR>
First, the routines “build-btree” and “build-index-btree” should be used in place of the previous approach to direct calls to make-instance. This is necessary, because the underlying class of the object depends on what repository it is stored in. These routines, like make-instance on persistent objects directly, allow you to specify the store controller at creation time. However, build-btree and build-index-btree will use the global *store-controller* if no keyword argument is provided. <BR>
<BR>
Secondly, in addition to executing:<BR>
<BR>
<PRE>
(asdf:operate 'asdf:load-op :elephant)
</PRE>
<BR>
to load elephant, one must execute either or both of:<BR>
<BR>
<PRE>
(asdf:operate 'asdf:load-op :ele-clsql)
(asdf:operate 'asdf:load-op :ele-bdb)
</PRE>
<BR>
depending on whether or not you wish to use the clsql backend or the BerkeleyDB backend, or both. <BR>
<BR>
<B><FONT SIZE="4">5.4 Multi-repository Operation</FONT></B><BR>
Elephant now keeps a small hashtables that maps “database specifications” into actual database connections. <BR>
<BR>
If a database spec is a string, it is assumed to be a BerkeleyDB path. If it is a list, it is a assumed to be a CL-SQL connection specification. <BR>
<BR>
The tests now have a function “do-all-tests-spec” that take a spec and based on its type attempt to open the correct kind of store controller and perform the tests. <BR>
<BR>
The routine “get-controller” takes this specifiation. <BR>
<BR>
The basic strategy is that the “database specification” object is stored in every persistent object and collection so that the repository can be found. <BR>
<BR>
In this way, objects that reside in different repositories can coexist within the LISP object space, allowing data migration. <BR>
<BR>
<B><FONT SIZE="4">5.5 Setting up PostGres</FONT></B><BR>
To set up a PostGres based back end, you should: <BR>
<BR>
<OL TYPE=1>
<LI TYPE=1 VALUE=1>Install postgres and make sure postmaster is running.
<LI TYPE=1 VALUE=2>Create a database called “test” and set its permissions to be reached by whatever connection specification you intend to use. The tests use: <BR>
<PRE>
(defvar *testpg-path*
'("localhost.localdomain" "test" "postgres" ""))
</PRE>
<BR>
meaning that connections must be allowed to the database test, user “postgres”, no password, connected from the same machine “localhost.localdomain”. (This would be changed to something more secure in a real application.) Typically you edit the file : pg_hba.conf to enable various kinds of connections in postgres. <BR>
<BR>
<LI TYPE=1 VALUE=3>Be sure to enable socket connection to postgres when you invoke the postmaster.
<LI TYPE=1 VALUE=4>Test that you can connect to the database with these credentials by running: <BR>
<BR>
psql -h 127.0.0.1 -U postgres test <BR>
<BR>
Before you attempt to connect with Elephant. <BR>
<BR>
</OL>
<BR>
meaning that connections must be allowed to the database test, user “postgres”, no password, connected from the same machine “localhost.localdomain”. (This would be changed to something more secure in a real application.) <BR>
<BR>
Furthermore, you must grant practically all creation/read/write privileges to the user postgres on this schema, so that it can construct the tables it needs. <BR>
<BR>
Upon first opening a CL-SQL based store controller, the tables, indexes, sequences, and so on needed by the Elephant system will be created in the schema named “test” automatically. <BR>
<BR>
To run the tests, execute:<BR>
<BR>
<PRE>
(asdf:operate 'asdf:load-op :elephant)
(asdf:operate 'asdf:load-op :ele-clsql)
(asdf:oos 'asdf:load-op :clsql-postgresql-socket)
(in-package "ELEPHANT-TESTS")
(do-all-tests-spec *testpg-path*)
</PRE>
<BR>
This should produce a small number of errors (about 7) for those test having to do with migration and the BerkeleyDB system specifically. <BR>
<BR>
If you execute:<BR>
<BR>
<PRE>
(asdf:operate 'asdf:load-op :ele-bdb)
</PRE>
<BR>
Then connection to the BerkeleyDB system will be enabled, and you should be able to execute both<BR>
<BR>
<PRE>
(do-all-tests-spec *testpg-path*)
(do-all-tests-spec *testdb-path*)
</PRE>
<BR>
with no errors in either case. <BR>
<BR>
At present the system has only been tested under PostGres. Some code parametrization would be required to work with other databases. <BR>
<BR>
<B><FONT SIZE="4">5.6 Repository Migration</FONT></B><BR>
This version of Elephant supports migration betwen store controllers, whether of the same implementation strategy or not. <BR>
<BR>
The tests <TT>migrate1</TT> - <TT>migrate5</TT> are demonstrations of this techinque. <BR>
<BR>
The functions for performing these migrations are: <BR>
<BR>
<TT>migraten-pobj</TT> <BR>
<BR>
The name of this function is meant to imply that it is destructive of the object in question, mutating it to point at the new repository. <BR>
<BR>
Which requies that you provide a copy-function to copy whatever slots you want from the persistent object as deeply or as shallowly as you desire. <BR>
<BR>
Data collections (btree's) can be move with the function: <BR>
<BR>
<TT>migrate</TT> <BR>
<BR>
A simple object that does not inherit from “persistent” but is attached to a key (on the root) can be copied with the routine <BR>
<BR>
<TT>copy-from-key</TT> <BR>
<BR>
It is hoped that these routines would allow, with some labor, a user to use one repository, and later decide to start using a different implementation strategy, and easily migrate the objects to the the new repository. The old repository could then be abandoned, or multiple repositories could be used at the same time.<BR>
<BR>
<BR>
<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
----<BR>
Robert L. Read, PhD read &T robertlread.net<BR>
Consider visiting Progressive Engineering: http://robertlread.net/pe<BR>
In Austin: 912-8593 "Think globally, Act locally." -- RBF<BR>
<BR>
<BR>
<BR>
<BR>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>