[elephant-devel] Class definition vs. store schema conflicts

[Ian Eslick]

My fellow Elephants,

I've recently added support for a full schema evolution infrastructure  
on my local development branch.  Every persistent class now has a  
schema object associated with it and each store that has one or more  
instances of that class has a corresponding database-specific schema  
that includes, among other things, a unique schema id for that class  
version and that store.  The schema in the database is sufficient to  
reproduce a basic defclass form (slot names and types, not accessors,  
initargs, initforms).

The prior version of class indexing had a sophisticated mechanism for  
synchronizing between the in-memory class definition and the in-store  
index list.  This doesn't seem to be used, and is too complicated to  
be useful so I have nixed it.  For the new schema-based notion of  
synchronization, I have made some simplifying assumptions:

MASTER SCHEMA:
    The in-memory class definition is always the master schema for all  
open stores that contain instances
    of that class.

A class redefinition or connecting to a store with a stale schema may  
mean that the master and the store schemas are now different.  This  
means we need to upgrade the store schema and potentially store  
instances to the new master schema.

IN-PLACE EVOLUTION:
    If only the indexed slots differ, then we simply add/delete  
indices to accommodate.
    This means that we have to keep track of the hierarchy so we can  
remove subclasses,
    or merge a set of subclass schemas if we move the index to a base  
class.  In-place
    is fine because no data becomes irrecoverable and all changes are  
at the class level.

FULL SCHEMA EVOLUTION:
    We have add/deleted slots or changed the type of a slot  
(persistent->transient)
    - We compute a diff function that adds/deletes the slot storage  
from the store
    - Any in-memory instances are upgraded
    - Store instances are upgraded:
      1) A scan function is provided to upgrade all instances in the  
store and delete the old schema versions
      2) Instances of prior schema versions still in the store can be  
lazily upgraded on load

SOME DESIGN CHOICES:

- Do we delete data (dropped slots, indices) by default during a  
schema evolution, or do we keep it around just in case?

- What kind of warning/error conditions do we want to provide and when?
   - When we load a new class def, connect to a store, and need to  
make schema changes on the store?
   - Class redefinition with one or more open stores
   - etc...

- How do we enable users to specify an upgrade function to move from  
schema to schema; do we provide a way to specify a schema version and  
an upgrade function and allow non-specified versions to just upgrade  
automatically?

- What happens when two lisp images are attached to the same store and  
one updates its class definition?  (Maybe you just get what you  
deserve?)

- What happens if a store has a schema for a class for which there is  
no in-memory class object?

- Do we want the database schema to store initforms, initargs, and  
accessor/reader/writer names?

- Given class schemas we could probably add persistent class slots...

Thanks,
Ian