[elephant-devel] Re: Derived Indicies

[Ian Eslick]

Another thought on this topic is that with a sufficiently efficient  
query system, some of the indices you describe should become  
unnecessary.  If you have messages indexed by time than you can simply  
walk the index and filter by user until you have a web page worth of  
messages.  A scan, even of an index, may not be fast enough so you  
could do an intersection of user-to values with an ordered list of  
recent messages.  This should perform similarly to a SQL engine which  
many ORM systems use for queries like this.

Nothing prevents you from making custom indices when you have to speed  
things up, of course!

Ian

On Mar 23, 2008, at 11:12 AM, Alex Mizrahi wrote:
>      IE>  :index t is not necessary - in fact it is ignored.  :slot- 
> deps
> are
>      also not required, but the derived index is updated on any slot  
> write
>      if that slot is not transient, set-valued or an association.   
> We can
>      add those last three slot types into the mix if necessary, but  
> I'm
>      trying to avoid too much complex computation taking place  
> during slot
>      writes (self-deadlock, etc) for the time being.
>
>
> seems to be fine..
> actually we are using derived indices in quite special way -- to get  
> index
> that is ordered in special way, to do efficient lookups of some kind.
> suppose we have a messaging system with two persistent classes --  
> user and
> message:
>
> (defpclass user ()
>     ((username :index t)
>     ...)
>
> (defpclass message ()
>    ((from-user :accessor from-user :index t)
>     (to-user :accessor to-user :index t)
>     (text :accessor text-of :index t)
>     (modification-time :accessor modification-time-of :index t)))
>
> suppose we'd like to get inbox and outbox views for user, i.e. list  
> of 10
> latest messages to user or from user.
> with considerable amounts of users and messages it is not efficient  
> to get
> these latest messages from any of normal indices, as it requires  
> scanning
> potentially large number of messages.
>
> it's possible to make efficient lookups via derived indices --  
> messages
> ordered first by user, then my modification time. iirc cons sorting in
> elephant has desired characteristics, but unfortunately postmodern  
> backend
> does not support complex type sorting.
>
> but we happily can reduce problem to sorting strings, e.g.  
> "13_31321433"
> where 13 is oid of from-user  and 31321433 is modification time  
> (universal
> time). (actually we have user-id field instead of oid to preserve  
> identity
> across multiple stores etc).
>
> thus, via two derived indices we can efficiently list messages for  
> inbox and
> outbox. (via cursor operations, that's not very easy, but works fine).
>
> as i understant you're going to make elephant more high level, and  
> possibly
> such low level index operations can be replaced with some high level
> concept.
> are you planning something like these dual indices?
>
> if there will be no special functionality for these, it would make  
> sense to
> make derived indices somewhat more flexible. for example, i suspect  
> some
> uses might require "foreign slot dependency".
> like if user definition above had group field, and we'd like to build
> messages-by-group index. when user-group changes, all messages of  
> this user
> should update derived index.
> probably there's no need in this stuff right now, but it would be  
> nice if it
> will be possible to add it in future, so advanced indices can be built
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel