[elephant-cvs] CVS elephant/doc/oldfiles
ieslick
ieslick at common-lisp.net
Sat Apr 28 02:31:15 UTC 2007
Update of /project/elephant/cvsroot/elephant/doc/oldfiles
In directory clnet:/tmp/cvs-serv16753/doc/oldfiles
Added Files:
INSTALL NEWS NOTES TODO TUTORIAL
Log Message:
Cleaning up root directory files; map-index performance enhancement, index api cleanup, ensure transaction fix, alpha quality documentation draft
--- /project/elephant/cvsroot/elephant/doc/oldfiles/INSTALL 2007/04/28 02:31:14 NONE
+++ /project/elephant/cvsroot/elephant/doc/oldfiles/INSTALL 2007/04/28 02:31:14 1.1
------------
Requirements
------------
Supported Lisps:
CMUCL 19a Linux
SBCL 0.9.17/1.0+ Linux / Mac OSX
Allegro CL 7.0/8.0 Linux / Mac OSX
OpenMCL 0.14.2
LispWorks (port in-progress)
Lisp libraries:
ASDF - http://www.cliki.net/asdf
UFFI 1.5.4+ - http://uffi.b9.com/
A Backend Database:
1) Oracle Berkeley DB 4.5 - http://www.oracle.com/database/berkeley-db.html
2) CLSQL - http://clsql.b9.com/ with an appropriate SQL installation.
Tested with SQlite3 and Postgresql so far
A C compiler, probably gcc or Visual Studio. Presumably you have this if you installed
------------------
Short Instructions
------------------
The new build system should work out of the box on most Un*x
platforms that have asdf, uffi and either clsql or Berkeley DB
installed in the usual places.
Try: (asdf:operate 'asdf:load-op :elephant)
Then: (open-store '(<backend> <spec>))
Where <backend> = { :bdb | :clsql }
<spec> = { "fresh directory for BDB files" | '(:sqlite3 "db path") | '(:postgresql "db path")
This should load all files, including compiling libraries, on most
systems. For Win32, see the instructions below.
(We'll improve the build process for Win32 if there is demand)
-----------------
Long Instructions
-----------------
For SBCL, CMUCL, Allegro 8.0+, MCL and CLISP:
0) Unpack Elephant. I put mine in the directory
/usr/local/share/common-lisp/elephant-0.6.x/
1) Install ASDF.
Ensure that you have a recent version of ASDF installed as
the load process now depends upon it.
2) Install UFFI
3) Install a backend: Either Berkeley DB 4.5, PostGresql, or SQLite 3.
-------
SQL
-------
For relational database systems, refering the formal documentation
other the heading "SQL-BACK-END".
-------------
Berkeley 4.5:
-------------
(Note: 0.6.0 required BDB 4.3; to upgrade 0.6.0 to 0.6.1, upgrade BDB to 4.5,
modify my-config.sexp appropriately then run 0.6.1+; your underlying Berekely DB
files will automatically upgrade when the DB is opened. To use 0.6.1, you will
have to manually migrate your 0.6.0 database to a fresh database created in 0.6.1)
Under Un*x, you may actually already have this installed, though
it may be compiled with funny options, so if things don't work
you may want to try to start from scratch. FreeBSD has a port
for this, as I'm sure do other BSDs (including DarwinPorts/Fink.)
Take note of where libdb.so and db.h are installed, usually:
/usr/local/BerkeleyDB.4.5/lib/libdb.so and
/usr/local/BerkeleyDB.4.5/include/db.h, or
/usr/local/lib/db45/libdb.so and
/usr/local/include/db45/db.h.)
a) Site specific configuration
config.sexp
Which contains an alist providing string paths pointing to the root
of the Berkeley DB distribution :berkeley-db-root, the library to load
:berkeley-db-lib and the pthreads library if you're running linux :pthread-lib.
For Win32 (directions courtesy of Bill Clementson):
---------------------------------------------------
Create an MSVC dll project and add src/db-bdb/libberkeley-db.c,
src/db-bdb/libberkeley-db.def and the Berkeley DB libdb43.lib files
to the project (should be in the build_win32/release folder)
Add the Berkeley DB dbinc include files directory and the
build_win32/release directory (where the Berkeley DB install
instructions builds the Berkeley DB objects by default) to
the build directories for the project
Build the Elephant DLL file
Since you've statically included libdb43.lib inside
libberkeley-db.c, it may or may not be necessary to load
libdb43.dll into Lisp (see below.)
4) Compile and load Elephant:
The new backend load process should work automatically on Un*x
systems but if there are probolems with loading foreign libraries,
then you can test your C tools setup with 'make' in the elephant
root directory. This will build the common memutils library
in src/memutil/libmemutil.so/dylib that all backends require.
There is a new two-phase load process. The first requires that
you use asdf to load the main elephant front-end:
(asdf:operate 'asdf:load-op :elephant)
This will load and compile Elephant. This will also automatically
load UFFI.
When you call (open-store <spec>) inside lisp it will automatically
load the remaining dependencies for the specified backend via ASDF.
To test the load process explicitly the following asdf files are
provided:
if you are using Berkeley DB, type:
(asdf:operate 'asdf:load-op :ele-bdb)
if you are using CL-SQL, type:
(asdf:operate 'asdf:load-op :ele-clsql)
if you are using SQLite3, type:
(asdf:operate 'asdf:load-op :ele-sqlite3)
5) Make the documentation:
Execute:
make
In the doc directory should be build the HTML version of the texinfo files.
-------
Testing
-------
Elephant uses RT for regression testing, available at:
http://www.cliki.net/RT
Once RT is installed
(asdf:operate 'asdf:load-op :elephant-tests)
(in-package :ele-tests)
(setf *default-spec* <backend>)
Where <backend> = { *testsqlite3-spec* | *testpg-spec* | *testbdb-spec* }
(do-backend-tests)
This will test the standalone API for your backend. Currently all tests are
passing on 0.6.0. There will be a set of migration tests that will be 'ignored'
but the final message should indicate no failing tests.
This should take less than 5 minutes on decent hardware.
The tests are not idempotent, so if you run the tests a second time,
they are likely to fail. To avoid this, for example if you are
debugging tests, just run the script delscript.sh (or do the
equivalent on Win32) in the elephant/tests directory.
Elephant allows migration between repositories. To test this:
(do-migration-tests *default-spec* <backend>)
where <backend> is a different *testXXXXX-spec* variable to test migration
to that backend.
This should take less than 2 minutes on decent hardware.
A backend is considered "green" if it can pass both the backend tests and the
migration tests.
--- /project/elephant/cvsroot/elephant/doc/oldfiles/NEWS 2007/04/28 02:31:14 NONE
+++ /project/elephant/cvsroot/elephant/doc/oldfiles/NEWS 2007/04/28 02:31:14 1.1
April, 2006 - Elephant 0.6.0 released by
Robert Read and Ian Eslick. Supports class slot
indexing and benefits from a clean refactoring
of backends and a host of other small changes.
This is a solid BETA release.
November 30, 2005 - Elephant 0.3.0 released by
the new maintainer, Robert L. Read, providing
support for relational database backends, repository
migration, and multi-repository operation.
As of this release, the documentation provides a
lot of information about installation and getting
things working; I wouldn't at all claim that it
is complete, smooth, or well organized. The more
notes I get about the use of Elephant, the more
inclined I will be to invest time in improving
the documentation.
October 7, 2004 -
Elephant 0.2.1 released. Thanks to Bill Clementson,
Elephant should compile on Win32 now. Also, a few minor
fixups.
September 19, 2004 -
Elephant 0.2 released. This is an BETA release.
New features:
- Secondary indices and cursors
- PPC Darwin OpenMCL / SBCL
- Doc strings and improved documentation
- An RT-based test suite
- many bugfixes
This release has been tested on CMUCL 19a, SBCL 0.8.14 and
Allegro 6.2 on x86 Linux and FreeBSD, and OpenMCL 0.14.2-p1
and SBCL 0.8.14 on PPC Darwin.
September 2, 2004 -
The bad news: there was a bug in 0.1 which made OID
generation inside of manual transactions deadlock.
The good news: this is fixed, and I've added OpenMCL
support. So I'm releasing 0.1-p1.
August 30, 2004 -
Elephant 0.1 was released August 30th, 2004. This is an
ALPHA quality release, so claims about correctness,
performance and safety should be taken with a grain of salt.
This release has been tested on CMUCL 19a, SBCL 0.8.13 and
Allegro 6.2 on x86 Linux and FreeBSD. OpenMCL and Lispworks
versions will come soon. As a proof of concept I've
compiled and run CL-IRC
http:://www.common-lisp.net/project/cl-irc
making all objects and slots persistent, except for the
socket-streams. It runs, and saves everything except for
the socket-streams.
--- /project/elephant/cvsroot/elephant/doc/oldfiles/NOTES 2007/04/28 02:31:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/oldfiles/NOTES 2007/04/28 02:31:15 1.1
-------
GENERAL
-------
this has been optimized for use with CMUCL / SBCL / Allegro.
OpenMCL has been minimally supported. Lispworks is a target as well
but less so as the developers don't have access to it.
Theoretically one can port this to any lisp with a decent
FFI and MOP. However since those are two of the less
standardized bits of Lisp, in practice this might be
difficult.
>From top to bottom, here are the implementation layers:
ELEPHANT package
persistent meta-object, persistent collections
controller
serializer
memutils package
UFFI / implementation specific stuff
libsleepycat.so
Sleepycat 4.2/3
While I loath specials, since you can't change the signature
of slot accessors, in order to pass parameters to the
database / serializer, specials are needed. Also specials
will probably play nice with threaded lisps.
-----------------------
CLASSES AND METACLASSES
-----------------------
Persistent classes which the user defines are declared and
instrumented by using the persistent-metaclass. Ideally
creating persistent versions of class, slot-defintion, et al
would be enough, but in reality various implementations do
things in different ways.
CMUCL / SBCL: their's a bit of work to make class slot
allocation and reader / writer / slot-boundp work right.
Allegro: is using slot-boundp instead of
slot-boundp-using-class inside of shared-initialize, which
necessitates some work.
CMUCL doesn't do non-standard allocation types correctly, so
we've created our own slot definition keyword :transient.
In the future this will change.
Andrew will add some notes here in the future.
-----------
COLLECTIONS
-----------
While we support serializing and persisting a wide class of
Lisp data types, there are problems with persisting
aggregate types (conses, lists, arrays, objects,
hash-tables...)
1) not automatic: there's no way for elephant to know when
you've changed a value in an aggregate object, so you have
to manually restore it back into the slot to get it saved.
example 1: you put a cons into the database. you change
it's car. this is not saved unless you resave the cons into
the database.
example 2: slot-1 of obj A (saved in the database) contains
a cons. you change the car of the cons. this is not
reflected into the database unless you resave A.
2) merge-conflicts: changing one value and saving an
aggregate will write out the whole aggregate, possibly
blowing away changes other threads have made behind your
back. this is not protected by transactions!
3) consing, non-lazy and expensive (de)serialization: you
have to serialize/deserialize the entire aggregate every
time you save it or restore it. This is pretty fast all
things considered, but it's probably better to use
persistent collections.
4) you have to store the entire collection in memory,
whereas one of the points of the database to store large
collections of objects.....
For these and other reasons, we provide a hash-table-like
interface to Berkeley BTrees. These have many advantages
over ordinary hash-tables from the point of view of
persistence.
There is a separate table for BTrees. This is because we
use a hand coded C function for sorting, which understands a
little of the serialized data. It can handle numbers (up to
64-bit bignums -- they are approximated by floats) and
strings (case-insensitive for 8-bit, code-point-order for
16-bit Unicode.) It should be fast but we don't want a
performance penalty on objects.
Secondary indices are mostly handled on the lisp side,
because of our weird table layout (see below) and to avoid
crossing FFI boundaries. Some unscientific microbenchmarks
indicated that there was no performance benefit on CMUCL /
SBCL, and only minor benefit (asymptotically nil) on
OpenMCL. They have a separate table. Actually two handles
are opened on this table: one which is plain, and one which
is associated to the primary btree table by a no-op indexing
function. Since we maintain the secondary keys ourselves,
the associated handle is good for gets / cursor traversals.
We use the unassociated handle for updates.
----------
CONTROLLER
----------
The controller is accessed through the special
*store-controller*. The controller keeps track of
1) the environment handle
2) the DB handle(s)
3) the instance cache
4) the root object
The environment handle and DB handle currently aren't really
exposed. Eventually they should be, so that tuning flags
can be set on them.
OIDs are generated by a bit of C code, which isn't great,
nor that safe (to get acceptable performance i use
DB_TXN_NOSYNC.) Waiting for Sleepycat 4.3.
The instance cache is implemented as a values-weak
hash-table. This is a hash-table where the values can be
collected, and when they are, the entire key-value entry is
deleted. There are implementations of this on the various
platforms. The instance cache is there to make
deserialization of persistant objects faster. Since we
[249 lines skipped]
--- /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/oldfiles/TODO 2007/04/28 02:31:15 1.1
[498 lines skipped]
--- /project/elephant/cvsroot/elephant/doc/oldfiles/TUTORIAL 2007/04/28 02:31:15 NONE
+++ /project/elephant/cvsroot/elephant/doc/oldfiles/TUTORIAL 2007/04/28 02:31:15 1.1
[923 lines skipped]
More information about the Elephant-cvs
mailing list