[elephant-devel] Class indexes dropped after database restart?
Ian Eslick
eslick at media.mit.edu
Sun Jan 10 20:31:34 UTC 2010
Hi Red,
I'm sorry this took so long to get to. I had to bring up a new Linux VM to try to reproduce your test among other things. Unfortunately I'm unable to reproduce this locally on the latest Ubuntu Linux w/ sbcl 1.0.32, BDB 4.7, and the latest elephant-1.0 tree.
I have some theories about what it could be, but if you still get this on the exact above configuration with a fresh image and database I'm at a bit of a loss.
Ian
On Jan 7, 2010, at 12:33 AM, Red Daly wrote:
> I'm just checking in to see if anyone has been able to reproduce this
> bug. It's a pretty nasty one and my solution does not scale
> well--requires a lot of time when reopening a database to construct
> the indexes.
>
> -Red
>
> On Tue, Dec 29, 2009 at 9:51 PM, Ian Eslick <eslick at media.mit.edu> wrote:
>> I'll give it a try with that configuration...
>>
>> Thanks,
>> Ian
>>
>> On Dec 29, 2009, at 9:42 PM, Red Daly wrote:
>>
>>> On Tue, Dec 29, 2009 at 5:53 PM, Ian Eslick <eslick at media.mit.edu> wrote:
>>>> Hmmm....I can't reproduce this locally with my copy of latest.
>>>>
>>>> Your test code passes on a fresh DB as well. It's possible you're catching an error having to do with opening non 1.0 databases.
>>>>
>>>> A few quick sanity checks:
>>>> 1) Did you clean out the test repository before running? (tests/delscript.sh)
>>>> 2) Also, are you using elephant-1.0 or the defunct elephant-dev darcs repository?
>>>
>>>
>>> I ran the test in a different, non-elephant directory with a fresh
>>> darcs repository pulled from the elephant-1.0 tree. Here's a copy of
>>> the my-config.sexp:
>>>
>>> ((:compiler . :gcc)
>>> (:berkeley-db-version . "4.7")
>>> (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.7/include/")
>>> (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.7/lib/")
>>> (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.7/lib/libdb-4.7.so")
>>> (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.7/bin/db_deadlock")
>>> (:berkeley-db-cachesize . 20971520)
>>> (:berkeley-db-max-locks . 2000)
>>> (:berkeley-db-max-objects . 2000)
>>> (:berkeley-db-max-transactions . 1000)
>>> (:berkeley-db-map-degree2 . t)
>>> (:berkeley-db-mvcc . nil)
>>> (:clsql-lib-paths . nil)
>>> (:prebuilt-libraries . nil))
>>>
>>> Did you try to reproduce this using Linux/SBCL? I don't know why it
>>> would make a difference.
>>>
>>>>
>>>> Thank you,
>>>> Ian
>>>
>>> Best,
>>> Red
>>>
>>>>
>>>> On Dec 29, 2009, at 12:52 PM, Red Daly wrote:
>>>>
>>>>> This bug is easy to reproduce with the latest version of elephant.
>>>>> Here is some code I used to reproduce the bug:
>>>>>
>>>>> (require :elephant)
>>>>>
>>>>> (defpackage :ele-bug
>>>>> (:use :ele :cl))
>>>>>
>>>>> (in-package :ele-bug)
>>>>>
>>>>> (defclass a ()
>>>>> ((x :initarg :x :index t :initform 0 :accessor x))
>>>>> (:metaclass ele:persistent-metaclass))
>>>>>
>>>>> (defun create-instances (n &optional (class 'a))
>>>>> (dotimes (i n)
>>>>> (make-instance class :x i)))
>>>>>
>>>>> (defun stats ()
>>>>> (dolist (class-sym '(a b))
>>>>> (format t "~A: ~A instances; ~A with value 10~%"
>>>>> class-sym
>>>>> (length (get-instances-by-class class-sym))
>>>>> (length (get-instances-by-value class-sym 'x 10)))))
>>>>>
>>>>> (defun open-close-test ()
>>>>> (declare (optimize (debug 3)))
>>>>> (macrolet ((is (form)
>>>>> `(assert ,form)))
>>>>> (flet ((open-it ()
>>>>> (ele:open-store '(:bdb "db2/")))
>>>>> (test-class-indexes (n-class n-10)
>>>>> (is (= n-class (length (get-instances-by-class 'a))))
>>>>> (is (= n-10 (length (get-instances-by-value 'a 'x 10))))))
>>>>> (with-open-store ('(:bdb "db2/"))
>>>>> (create-instances 100 'a)
>>>>> (test-class-indexes 100 1))
>>>>> (with-open-store ('(:bdb "db2/"))
>>>>> (test-class-indexes 100 1)))))
>>>>>
>>>>> (open-close-test)
>>>>>
>>>>> I was able to fix the problem with a minor patch, but I doubt the
>>>>> correctness of my solution. Here is the diff:
>>>>>
>>>>> diff -rN old-elephant-1.0/src/elephant/controller.lisp
>>>>> new-elephant-1.0/src/elephant/controller.lisp
>>>>> 711c711,712
>>>>> < :key-form 'schema-classname-keyform))
>>>>> ---
>>>>>> :key-form 'schema-classname-keyform
>>>>>> :populate t))
>>>>> 716c717,718
>>>>> < :key-form 'instance-cidx-keyform))))
>>>>> ---
>>>>>> :key-form 'instance-cidx-keyform
>>>>>> :populate t))))
>>>>>
>>>>>
>>>>> I essentially just added a :populate t argument to the open-controller
>>>>> method. This seems to indicate that the 'BY-NAME secondary indices
>>>>> that map a class indices to OIDs, and class names to class indices, do
>>>>> not persist. Is this expected behavior? I'm not familiar enough with
>>>>> the internals to tell.
>>>>>
>>>>> Thanks,
>>>>> Red
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 23, 2009 at 3:27 PM, Red Daly <reddaly at gmail.com> wrote:
>>>>>> This is the 1.0 repository. I will see if I can reproduce the problem in a
>>>>>> test case.
>>>>>> Red
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sep 23, 2009, at 9:50 AM, Ian Eslick <eslick at media.mit.edu> wrote:
>>>>>>
>>>>>> Which darcs repository? There is the old 0.91 release and the new
>>>>>> elephant-1.0 branch...
>>>>>> Ian
>>>>>> On Sep 22, 2009, at 8:41 PM, Red Daly wrote:
>>>>>>
>>>>>> I should also note that I am using the version of elephant from darcs. And
>>>>>> I discovered that when the program went down it was due to a segfault (I
>>>>>> believe in emacs). However, all instances had been created under many
>>>>>> different transactions that did not abort.
>>>>>>
>>>>>> Here is some more fiddling around:
>>>>>>
>>>>>> CL-USER> (lisp-implementation-version)
>>>>>> "1.0.29.54.rc1"
>>>>>> BOOKS> (ele::get-db-schemas *ele-db* 'material)
>>>>>> NIL
>>>>>> BOOKS> (ele::controller-instance-class-index *ele-db*)
>>>>>> #<BDB-BTREE-INDEX
>>>>>> oid:13701>
>>>>>> BOOKS> (ele:map-btree #'(lambda (key
>>>>>> val)
>>>>>> (format t "~S => ~S~%" key
>>>>>> val))
>>>>>> (ele::controller-instance-class-index *ele-db*))
>>>>>> 307 =>
>>>>>> 307
>>>>>> NIL
>>>>>>
>>>>>>
>>>>>> Red
>>>>>>
>>>>>> On Tue, Sep 22, 2009 at 8:08 PM, Red Daly <reddaly at gmail.com> wrote:
>>>>>>>
>>>>>>> I noticed there was a disparity between get-instances-by-class and
>>>>>>> map-inverted-index. I will let the code speak for itself:
>>>>>>>
>>>>>>> BOOKS> (length (ele:map-inverted-index #'(lambda (isbn obj) obj) 'material
>>>>>>> 'isbn :collect
>>>>>>> t))
>>>>>>>
>>>>>>> 1945
>>>>>>> BOOKS> (length (ele:get-instances-by-class 'material))
>>>>>>> 0
>>>>>>> BOOKS> (defparameter f (make-instance 'material :title "Fake Book"))
>>>>>>>
>>>>>>> F
>>>>>>> BOOKS> (length (ele:get-instances-by-class 'material))
>>>>>>>
>>>>>>> 1
>>>>>>> BOOKS> (length (ele:map-inverted-index #'(lambda (isbn obj) obj) 'material
>>>>>>> 'isbn :collect
>>>>>>> t))
>>>>>>> 1946
>>>>>>>
>>>>>>> There is a similar problem with the university class.
>>>>>>>
>>>>>>> BOOKS> (length (ele:get-instances-by-class 'university))
>>>>>>> 0
>>>>>>> BOOKS> *stanford-university*
>>>>>>> #<UNIVERSITY
>>>>>>> oid:204>
>>>>>>> BOOKS> (official-name *stanford-university*)
>>>>>>> "Stanford University"
>>>>>>>
>>>>>>> All of the instances in the database were created in a prior session.
>>>>>>> Then I shut down the DB and restarted Lisp. Now MAP-CLASS and
>>>>>>> GET-INSTANCES-BY-CLASS return nil for all classes. It seems like new
>>>>>>> instances get added to the "class index," but older instances are not
>>>>>>> found. However, these classes persist in the inverted slot indexes. Can
>>>>>>> anybody diagnose what's going on?
>>>>>>>
>>>>>>> here are the definitions of the classes:
>>>>>>>
>>>>>>> (defclass department ()
>>>>>>> ((university :initarg :university :initform nil :accessor
>>>>>>> department-university
>>>>>>> :index t)
>>>>>>> (official-name :initarg :official-name :initform nil :accessor
>>>>>>> official-name)
>>>>>>> (common-name :initarg :common-name :initform nil :accessor common-name)
>>>>>>> (abbreviation :initarg :abbreviation :initform nil :accessor
>>>>>>> abbreviation
>>>>>>> :index t))
>>>>>>> (:metaclass ele:persistent-metaclass)
>>>>>>> (:documentation "A department has a bunch of course listings."))
>>>>>>>
>>>>>>> (defclass material ()
>>>>>>> ((title :initarg :title :initform nil :accessor material-title
>>>>>>> :index t)
>>>>>>> (authors :initarg :authors :initform nil :accessor material-authors) ;;
>>>>>>> a list of
>>>>>>> authors
>>>>>>> (edition :initarg :edition :initform nil :accessor material-edition)
>>>>>>> (isbn :initarg :isbn :initform nil :accessor material-isbn
>>>>>>> :index t)
>>>>>>> (asin :initarg :asin :initform nil :accessor material-asin
>>>>>>> :index t
>>>>>>> :documentation "An Amazon.com identifier that uniquely identifies
>>>>>>> the material")
>>>>>>> (image-uri :initarg :image-uri :initform nil :accessor
>>>>>>> material-image-uri))
>>>>>>> (:metaclass ele:persistent-metaclass))
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Red
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> elephant-devel site list
>>>>>> elephant-devel at common-lisp.net
>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>>>
>>>>>> _______________________________________________
>>>>>> elephant-devel site list
>>>>>> elephant-devel at common-lisp.net
>>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>>
>>>>> _______________________________________________
>>>>> elephant-devel site list
>>>>> elephant-devel at common-lisp.net
>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>
>>>>
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>>
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
More information about the elephant-devel
mailing list