[elephant-devel] Class indexes dropped after database restart?

Red Daly reddaly at gmail.com
Thu Jan 7 05:33:49 UTC 2010 

I'm just checking in to see if anyone has been able to reproduce this
bug.  It's a pretty nasty one and my solution does not scale
well--requires a lot of time when reopening a database to construct
the indexes.

-Red

On Tue, Dec 29, 2009 at 9:51 PM, Ian Eslick <eslick at media.mit.edu> wrote:
> I'll give it a try with that configuration...
>
> Thanks,
> Ian
>
> On Dec 29, 2009, at 9:42 PM, Red Daly wrote:
>
>> On Tue, Dec 29, 2009 at 5:53 PM, Ian Eslick <eslick at media.mit.edu> wrote:
>>> Hmmm....I can't reproduce this locally with my copy of latest.
>>>
>>> Your test code passes on a fresh DB as well.  It's possible you're catching an error having to do with opening non 1.0 databases.
>>>
>>> A few quick sanity checks:
>>> 1) Did you clean out the test repository before running?   (tests/delscript.sh)
>>> 2) Also, are you using elephant-1.0 or the defunct elephant-dev darcs repository?
>>
>>
>> I ran the test in a different, non-elephant directory with a fresh
>> darcs repository pulled from the elephant-1.0 tree.  Here's a copy of
>> the my-config.sexp:
>>
>> ((:compiler . :gcc)
>> (:berkeley-db-version . "4.7")
>> (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.7/include/")
>> (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.7/lib/")
>> (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.7/lib/libdb-4.7.so")
>> (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.7/bin/db_deadlock")
>> (:berkeley-db-cachesize . 20971520)
>> (:berkeley-db-max-locks . 2000)
>> (:berkeley-db-max-objects . 2000)
>> (:berkeley-db-max-transactions . 1000)
>> (:berkeley-db-map-degree2 . t)
>> (:berkeley-db-mvcc . nil)
>> (:clsql-lib-paths . nil)
>> (:prebuilt-libraries . nil))
>>
>> Did you try to reproduce this using Linux/SBCL?  I don't know why it
>> would make a difference.
>>
>>>
>>> Thank you,
>>> Ian
>>
>> Best,
>> Red
>>
>>>
>>> On Dec 29, 2009, at 12:52 PM, Red Daly wrote:
>>>
>>>> This bug is easy to reproduce with the latest version of elephant.
>>>> Here is some code I used to reproduce the bug:
>>>>
>>>> (require :elephant)
>>>>
>>>> (defpackage :ele-bug
>>>>    (:use :ele :cl))
>>>>
>>>> (in-package :ele-bug)
>>>>
>>>> (defclass a ()
>>>>  ((x :initarg :x :index t :initform 0 :accessor x))
>>>>  (:metaclass ele:persistent-metaclass))
>>>>
>>>> (defun create-instances (n &optional (class 'a))
>>>>  (dotimes (i n)
>>>>    (make-instance class :x i)))
>>>>
>>>> (defun stats ()
>>>>  (dolist (class-sym '(a b))
>>>>    (format t "~A: ~A instances; ~A with value 10~%"
>>>>           class-sym
>>>>           (length (get-instances-by-class class-sym))
>>>>           (length (get-instances-by-value class-sym 'x 10)))))
>>>>
>>>> (defun open-close-test ()
>>>>  (declare (optimize (debug 3)))
>>>>  (macrolet ((is (form)
>>>>              `(assert ,form)))
>>>>    (flet ((open-it ()
>>>>            (ele:open-store '(:bdb "db2/")))
>>>>          (test-class-indexes (n-class n-10)
>>>>            (is (= n-class (length (get-instances-by-class 'a))))
>>>>            (is (= n-10    (length (get-instances-by-value 'a 'x 10))))))
>>>>      (with-open-store ('(:bdb "db2/"))
>>>>       (create-instances 100 'a)
>>>>       (test-class-indexes 100 1))
>>>>      (with-open-store ('(:bdb "db2/"))
>>>>       (test-class-indexes 100 1)))))
>>>>
>>>> (open-close-test)
>>>>
>>>> I was able to fix the problem with a minor patch, but I doubt the
>>>> correctness of my solution.  Here is the diff:
>>>>
>>>> diff -rN old-elephant-1.0/src/elephant/controller.lisp
>>>> new-elephant-1.0/src/elephant/controller.lisp
>>>> 711c711,712
>>>> <                       :key-form 'schema-classname-keyform))
>>>> ---
>>>>>                      :key-form 'schema-classname-keyform
>>>>>                      :populate t))
>>>> 716c717,718
>>>> <                       :key-form 'instance-cidx-keyform))))
>>>> ---
>>>>>                      :key-form 'instance-cidx-keyform
>>>>>                      :populate t))))
>>>>
>>>>
>>>> I essentially just added a :populate t argument to the open-controller
>>>> method.  This seems to indicate that the 'BY-NAME secondary indices
>>>> that map a class indices to OIDs, and class names to class indices, do
>>>> not persist.  Is this expected behavior?  I'm not familiar enough with
>>>> the internals to tell.
>>>>
>>>> Thanks,
>>>> Red
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Sep 23, 2009 at 3:27 PM, Red Daly <reddaly at gmail.com> wrote:
>>>>> This is the 1.0 repository.  I will see if I can reproduce the problem in a
>>>>> test case.
>>>>> Red
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sep 23, 2009, at 9:50 AM, Ian Eslick <eslick at media.mit.edu> wrote:
>>>>>
>>>>> Which darcs repository?  There is the old 0.91 release and the new
>>>>> elephant-1.0 branch...
>>>>> Ian
>>>>> On Sep 22, 2009, at 8:41 PM, Red Daly wrote:
>>>>>
>>>>> I should also note that I am using the version of elephant from darcs.  And
>>>>> I discovered that when the program went down it was due to a segfault (I
>>>>> believe in emacs).  However, all instances had been created under many
>>>>> different transactions that did not abort.
>>>>>
>>>>> Here is some more fiddling around:
>>>>>
>>>>> CL-USER> (lisp-implementation-version)
>>>>> "1.0.29.54.rc1"
>>>>> BOOKS> (ele::get-db-schemas *ele-db* 'material)
>>>>> NIL
>>>>> BOOKS> (ele::controller-instance-class-index *ele-db*)
>>>>> #<BDB-BTREE-INDEX
>>>>> oid:13701>
>>>>> BOOKS> (ele:map-btree #'(lambda (key
>>>>> val)
>>>>>                           (format t "~S => ~S~%" key
>>>>> val))
>>>>>                       (ele::controller-instance-class-index *ele-db*))
>>>>> 307 =>
>>>>> 307
>>>>> NIL
>>>>>
>>>>>
>>>>> Red
>>>>>
>>>>> On Tue, Sep 22, 2009 at 8:08 PM, Red Daly <reddaly at gmail.com> wrote:
>>>>>>
>>>>>> I noticed there was a disparity between get-instances-by-class and
>>>>>> map-inverted-index.  I will let the code speak for itself:
>>>>>>
>>>>>> BOOKS> (length (ele:map-inverted-index #'(lambda (isbn obj) obj) 'material
>>>>>> 'isbn :collect
>>>>>> t))
>>>>>>
>>>>>> 1945
>>>>>> BOOKS> (length (ele:get-instances-by-class 'material))
>>>>>> 0
>>>>>> BOOKS> (defparameter f (make-instance 'material :title "Fake Book"))
>>>>>>
>>>>>> F
>>>>>> BOOKS> (length (ele:get-instances-by-class 'material))
>>>>>>
>>>>>> 1
>>>>>> BOOKS> (length (ele:map-inverted-index #'(lambda (isbn obj) obj) 'material
>>>>>> 'isbn :collect
>>>>>> t))
>>>>>> 1946
>>>>>>
>>>>>> There is a similar problem with the university class.
>>>>>>
>>>>>> BOOKS> (length (ele:get-instances-by-class 'university))
>>>>>> 0
>>>>>> BOOKS> *stanford-university*
>>>>>> #<UNIVERSITY
>>>>>> oid:204>
>>>>>> BOOKS> (official-name *stanford-university*)
>>>>>> "Stanford University"
>>>>>>
>>>>>> All of the instances in the database were created in a prior session.
>>>>>> Then I shut down the DB and restarted Lisp.  Now MAP-CLASS and
>>>>>> GET-INSTANCES-BY-CLASS return nil for all classes.  It seems like new
>>>>>> instances get added to the "class index," but older instances are not
>>>>>> found.  However, these classes persist in the inverted slot indexes.  Can
>>>>>> anybody diagnose what's going on?
>>>>>>
>>>>>> here are the definitions of the classes:
>>>>>>
>>>>>> (defclass department ()
>>>>>>   ((university :initarg :university :initform nil :accessor
>>>>>> department-university
>>>>>>                :index t)
>>>>>>    (official-name :initarg :official-name :initform nil :accessor
>>>>>> official-name)
>>>>>>    (common-name :initarg :common-name :initform nil :accessor common-name)
>>>>>>    (abbreviation :initarg :abbreviation :initform nil :accessor
>>>>>> abbreviation
>>>>>>                  :index t))
>>>>>>   (:metaclass ele:persistent-metaclass)
>>>>>>   (:documentation "A department has a bunch of course listings."))
>>>>>>
>>>>>> (defclass material ()
>>>>>>   ((title :initarg :title :initform nil :accessor material-title
>>>>>>           :index t)
>>>>>>    (authors :initarg :authors :initform nil :accessor material-authors) ;;
>>>>>> a list of
>>>>>> authors
>>>>>>    (edition :initarg :edition :initform nil :accessor material-edition)
>>>>>>    (isbn :initarg :isbn :initform nil :accessor material-isbn
>>>>>>          :index t)
>>>>>>    (asin :initarg :asin :initform nil :accessor material-asin
>>>>>>          :index t
>>>>>>          :documentation "An Amazon.com identifier that uniquely identifies
>>>>>> the material")
>>>>>>    (image-uri :initarg :image-uri :initform nil :accessor
>>>>>> material-image-uri))
>>>>>>   (:metaclass ele:persistent-metaclass))
>>>>>>
>>>>>> Thanks,
>>>>>> Red
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>>
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
>

More information about the elephant-devel mailing list