[elephant-devel] difference between get-instances-by-class and cursors

[Ian Eslick]

Actually there are two separate issues here:

1) There was a report of space being used even when overwriting keys.  I
was unable to confirm this on repeated attempts to write a range of keys
to a btree.  The total DB size did grow the first two times I did it but
not thereafter - not sure what's happening but it isn't a major problem. 

The 4.3 version of BDB does not reclaim deleted space so your DB will
never shrink unless you migrate.  Migration will reclaim namespace,
compact the storage and garbage collect - although it's an expensive
process it can be done online but you'll have to disconnect and
reconnect after you're done.  In lisp, of course, you can write
infrastructure to do that behind the scenes so the image itself never
need go down.  But I digress...

2) As for Aycan's issue, the problem is that you're trying to maintain
your own btree and indices for the user class.  Each indexed class has
its own, private 'indexed-btree' that maintains a record of each
instance of the class (oid->instance).  This is so we can easily handle
duplicate key indices for mapping from key values back to instances
(keyval->oid).  These btrees are tracked by the (controller-class-root
*store-controller*) top-level btree.  I did this so as not to require
reserved keywords in the root namespace.  Thus there are two roots from
which reachability is defined (in the GC sense).

If you remove a user from your own btree, you've only removed the
reference to the instance from that btree, you haven't removed the
class-level reference to it.  Dropping instances from the class can be
done by calling (drop-instance <list of objects>).

Aycan,

So you don't have to bother with managing your own index of users, there
is an implicit index reachable via the class index I described above. 
When you put in :index into a slot definition the btree and indices are
created automatically (look ma, no hands!)  It should look something
like this:

(defclass user
  ((timestamp :accessor user-timestamp :initform (get-universal-time)
:index t)
   (name :accessor user-name :initarg :name)))

(make-instance 'user :name "Aycan")

(get-instances-by-range 'users 'timestamp (get-universal-time) (-
(get-universal-time) 32000))
=> (#<USER "Aycan" ...>)

The error you are seeing is due to passing in your index and not the
slot name.  The example log system Robert and I hacked up should provide
a good guide as to how to do things like this.  I do timestamp based
range queries on indexed objects all the time now in my own system.

It's late here so I hope this answers the question to your satisfaction
and sorry if I rambled a little!  But please let me know if you have
further questions.  I should be able to finish my tasks for 0.6.0 in the
next week or two.

Cheers,
Ian

Robert L. Read wrote:
> Dear Aycan,
>
>     I will try to answer this more carefully tomorrow morning.
>    
>     For now just let me say that I have heard there is a bug about
> keys not being actually
> removed; I will try to think about this when I have time tomorrow morning.
>
>     I am not sure what the problem is with the timestamps; but I have
> done things very similar
> to what you are trying, so I assume we will be able to get it to work.
>
>
>
> On Sat, 2006-04-08 at 11:26 +0300, Aycan iRiCAN wrote:
>> Hi,
>>
>> Firstly, I want to thank you again for this great project.
>>
>> Let me ask some questions. What is the difference between using
>> get-instances-by-class and using cursors?
>>
>> When I use get-instances-by-class I get every instance (including
>> deleted keys) of the class. But when I use cursors, I only get
>> existing keys.
>>
>> HEDEE> (get-instances-by-class 'user)
>> (#<USER "test" 3353473039> #<USER "aycan" 3353182543>)
>> HEDEE> (remove-kv 5 *hedee-users*)
>> T
>> HEDEE> (get-instances-by-class 'user)
>> (#<USER "test" 3353473039> #<USER "aycan" 3353182543>)
>> HEDEE> (with-transaction ()
>> 	 (with-btree-cursor (c *hedee-users*)
>> 	   (loop
>> 	    for (m k v) = (multiple-value-list
>> 		           (cursor-first c))
>> 	    then (multiple-value-list (cursor-next c))
>> 	    while m      
>> 	    collect v into users
>> 	    finally (return (nreverse users)))))
>> (#<USER "aycan" 3353182543>)
>>
>> Is it possible to completely remove a key from a btree?
>>
>> Say I have a class named user (used above):
>>
>> (defclass user ()
>>   ((id :accessor id :initarg :id :type integer
>>        :initform (next-id)
>>        :index t)
>>    (name :accessor name :initarg :name
>> 	 :initform nil :index t)
>>    (epitaph :accessor epitaph :initarg :epitaph
>> 	    :initform nil)
>>    (image-type :accessor image-type :initarg :image-type :type string
>> 	       :initform nil)
>>    (timestamp :accessor timestamp :initarg :timestamp :type integer :index t
>> 	      :initform (get-universal-time)))
>>   (:metaclass persistent-metaclass))
>>
>> and I created secondary indices via:
>>
>> (defun add-indices ()
>>   (with-transaction ()
>>     (add-index *hedee-users*
>> 	       :index-name 'id :key-form '(lambda (s key value)
>> 					   (declare (ignore s key))
>> 					   (values t (id value))))
>>     (add-index *hedee-users*
>> 	       :index-name 'name :key-form '(lambda (s key value)
>> 					     (declare (ignore s key))
>> 					     (values t (name value))))
>>     (add-index *hedee-users*
>> 	       :index-name 'timestamp :key-form '(lambda (s key value)
>> 						  (declare (ignore s key))
>> 						  (values t (timestamp value))))))
>>
>> I tried to get users with a timestamp around some delta (320000 in
>> below) via:
>>
>> (get-instances-by-range 'user
>>                         (get-index *hedee-users* 'timestamp) 
>>                         (get-universal-time) 
>>                         (- (get-universal-time) 320000))
>>
>> But I'm getting:
>>
>> The value 3353473355 is not of type FIXNUM.
>>
>> How can I query by range using a timestamp?
>>
>> Best Regards,
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net <mailto:elephant-devel at common-lisp.net>
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>     
> ------------------------------------------------------------------------
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel