[elephant-devel] Database evolution (i.e. schema migration)

[Red Daly]

In the other database systems I have worked with, it is common to have
some means of migrating the schema of a database and the data along
with it.  As far as I know, in Elephant there is no standard way to
define database versions and upgrades/downgrades to accompany them.
Elephant could do with such a system.

In my use case, the main problem comes when a new class or slot is
introduced, a slot is renamed, or slot properties change.  I will make
such a change in a development environment and then run a function
that performs any necessary migration.  I must then remember to run
the migration function on the production database.

For example, when a new slot is introduced it is unbound in existing
instances.  I will often set the slot value to nil in the migration
function.  Other times I will need to query the web for the correct
value for existing objects' new slots.

Defining a schema evolution system seems tricky now because the
database schema is derived somewhat implicitly from defclass forms.
Compare this scenario to Ruby on Rails where a series of migrations
moves the database from a blank slate to a the latest and greatest
with a series of migration scripts.  The actual 'model' for a
persistent Rails class is derived from the database itself, and the
rails equivalent of defclass does not even list slots.  I am not
suggesting that we do things the Rails way, but I must say that the
migration scheme is rather simple and powerful.

The desire is to define a migration framework that satisfies the
following criteria:

* the database should keep track of the version of its schema.  This
need not be linear (see the difference between Ruby on Rails
migrations in 2.0 and 2.1)
* a developer should be able to introduce a new version of the
database and know exactly how the schema changes
* version upgrades should call hooks that migrate the database in
light of the schema changes

Perhaps the most straightforward implementation would be to introduce
a new means of defining persistent classes and their indices more
explicitly, instead of letting defclass do all the work.  Instead of
adding ":index t" to a slot definition, you would instead define a
migration and call (add-index-for-slot (find-class 'user)
'telephone-number).

Is there a means of achieving this already?  If not, it would be a
great feature to add and I would appreciate any comments.

Best,
Red