[elephant-devel] Add indexed persistent class slots to elephant?

[Ian Eslick]

Robert,

Thanks for the comments.  You make some good points that I'll have to 
think about a bit.  I make a few comments below.

> author; in some ways I do not even consider myself qualified to 
> answer.  I will happily accept
> and your code, as long as the test are green and are you are expanding 
> the possibilities for
> the users, rather than constraining them.  And documentation would be 
> nice.

Ok, I'll stick myself on CVS so I can push things over incrementally and 
back things out more easily in case of damage (plus keep up to date on 
change you and others are making).  Did I hear that someone was working 
on GC for persistent objects?

>     As to your technical questions, about the approaches to 
> implementing the indexing, in my
> experience indexing is one of the most fluid aspects of an object 
> space; it is a tunable issue
> that tends to change as usage pattern changes even more than the 
> underlying data structure.
> So from my point of view I would much prefer macros or simple 
> functions that let you easily
> add, and discard, indexes.  Of course, your observation that one does 
> typically index a slot
> is quite correct, and it would be very convenient if such an index 
> could be added and dropped
> very easily; if one could also implement boolean operators based on 
> those indexes, so much
> the better.  As you state, one should be able to somehow introspect 
> the class and discover
> the existing indexing structure; I'm not sure of the best way to do that.

I think the big convenience feature is that object slot-writes will 
automatically update the respective slot index.  This isn't something 
you can do with an API layers on top.  In fact, doing this sort of 
indexing isn't easy at all.  As I understand it to update a slot index 
which is defined as a primary index I have to do a separate call to 
(setf get-value) to write the new value as key and object as value.  As 
a secondary index I have to do a write to the primary btree to update 
the secondary index which has the effect of calling all the secondary 
index functions on that object.  If I do separate btrees for each slot 
then each application needs to write it's own :after method on accessors 
to handle index updating.  This makes a simple add/delete of indexes 
less convenient and 'built-in' to the persistent language.

The changes to the MOP itself are actually fairly trivial.  One change 
parallels the existing transient slot definition semantics and the other 
is a single function that wraps the :around method on (setf 
slot-value-by-class).  Everything else is or should be separable. 

I will go ahead and make an API that uses primary btrees and just have 
the MOP extension use that.  If you redefine the class to remove the 
indexed keyword, the index will be dropped and if you add it then the 
new slot btree will be created and brought up to date.  That is the 
cleanest way to support add/drop of indexed slots using the metaclass 
approach.

You can also choose not to use the :indexed slot keyword and use the 
separate API instead.  You could also wrap your persistent object's 
accessors to automatically keep the index up to date so there will be a 
relatively simple way to bypass any dependence on the MOP.  Someone 
could then write a macro for defining indexed persistent classes that 
does the wrapping automatically.

I have to think about a common way to keep track of slot btrees so the 
two access methods play nicely.  I'll send some documentation on the 
whole things when I've got it working.

>     Although I am a "Smug Lisp Weanie" and fairly well educated, I 
> have to admit that all
> of this direct hacking of CLOS and MOP makes my head spin a little.  
> That is the stuff that
> creates the most problems in trying to work with multiple platforms.  
> We seem now to
> work with ACL and SBCL, and Andrew Blumberg is helping me solve and 
> OpenMCL problem
> right now.  As a maintainer, I'm a little loathe to recommend even 
> more complexity in that part.

This stuff always makes my head hurt as well.  I try to think of it like 
a good mental workout!  I'll try to make sure that the changes are as 
orthogonal as possible so there aren't any additional issues across 
versions.

>     However, I always believe interfaces are more important than 
> implementations.  The real
> question is: are you conveniently (elegantly?) expanding the API to 
> make elephant more useful?
> Whether this is done with a simple function or metaclass keywords 
> might actually be a side
> issue.  The code that will utilize the slot-based indexes is more 
> important than the code that
> will create them.
>     If I have a right to insist on anything, I will insist that the 
> indexes be easily droppable.

Please insist!  Any constraints provided by others means a smaller 
design space for me to get lost in!  Let me know if my proposal above 
satisfies your request.

Regards,
Ian