[elephant-devel] Querying Advice

[Daniel Salama]

> Were you storing persistent-metaclass objects or simply normal  
> objects?
> Normal objects are huge relative to persistent objects as currently  
> all
> the slot names are also serialized (potentially in 32-bit unicode  
> if on
> SBCL).  I have some improvements planned to reduce the storage and
> processing overhead for strings, but it will take some evaluation to
> make sure performance doesn't suffer over much.  Some of this will  
> make
> it into the forthcoming 0.6.1 release I'm trying to finish a bunch of
> work for prior to my twin daughters are born.
>
> Also SBCL strings in the 0.6.0 release are stored in their internal
> 32-bit form so we can memcpy them back and forth between elephant and
> the internal format.  The new serializer will store them in the  
> minimal
> format and then convert back to the 32-bit internal form for unicode
> enabled SBCL.

We were storing persistent objects. However, instead of storing them  
all into the root, we were storing them into a normal collection  
class (I think it was even a BTree), such that we would have a  
collection of "people". Maybe this was the wrong approach. We were  
coming from the RDBMS world so what we ended up doing is storing the  
root, different collections of objects, where each collection would  
"kind-of" resemble the different tables we have in MySQL.

> DCM is, as I understand it, a form of prevalence with explicit
> checkpointing using Elephant as the backend.

Cool. Will look further into it.

>> With regards to: "...If so, you have the full power of lisp at your
>> command in dealing with the querying...", I agree with you. However,
>> where I'm trying to get at is how "easy" would it be to generate  
>> these
>> type of dynamic queries in a generic way. Of course, we could always
>> hard code all the cases for each of our different searchable screens,
>> but the thought of that simply just makes me vomit :)
> Associate a constraint on the display with the user action for
> selections and an ordering function for specific columns.  If you want
> to sort by multiple columns, then compose your functions:
>
> Selection constraint fn:
> #'(lambda (obj) (= (slot-accessor obj) value)
>
> Sorting fn:
> #'(lambda (obj1 obj2) (string< (name obj1) (name obj2)))
> #'(lambda (obj1 obj2) (< (age obj1) (age obj2)))
>
> Composed sorting function:
> (defun compose-sorts (sort1 sort2)
>     (lambda (o1 o2)
>         (or (funcall sort1 o1 o2)
>              (funcall sort2 o1 o2))))
>
> Same thing with constraints:
> (defun compose-constraint (c1 c2)
>     (lambda (obj)
>         (and (funcall c1 obj)
>                 (funcall c2 obj))))
>
> Or to be a little more fancy:
> (defun compose-constraints (clist)
>     (lambda (obj)
>         (every #'(lambda (c) (funcall c obj)) clist)))
>
> (defun compose-sorts (slist)
>     (lambda (o1 o2)
>         (some #'(lambda (sort) (funcall sort o1 o2)))))
>
> This all comes down to:
>
> (defun select-results (objects)
>     (select-if (compose-constraints (make-name-constraints "Bob")
> (make-age-constraint 18 120))
>                   objects))
>
> (defun order-results (objects)
>     (sort (copy-list objects)
>             (compose-sorts #'by-occupation #'by-name #'by-date-of- 
> birth)))
>
> This way you can have a set of widgets with static constraints and  
> then
> you can compose functions over the constraints based on the dynamic
> query.  There may be more elegant ways to do this but I think this is
> what Robert was driving at.
>
> To do this on elephant simply requires that you (get-instances-by- 
> class
> 'person) to get all instances of person and then run the lisp query  
> over
> it.  If you access person alot, these persistent references will be
> cached in memory.  Or they're all in the cache including slot  
> values in
> DCM - in which case these queries will run really fast!

I think I understand this. Will try to play with these ideas further  
and see where we go.

> Every time you finish or 'commit' a user transaction (form submit),  
> just
> write the object data back to the Elephant persistent store before
> acking to the user.

Is this done automatic by the way DCM is configured or is it a manual  
commit you do in DCM/Elephant to persistent store?

>
>
> Ian

Thanks,
Daniel