[elephant-devel] DB migration issue

[quan hu]

Hi Ian, Yarek,

    Thanks for providing the information.
     I downloaded the elephant-unstable and test it with BDB4.7.
     The failure symptom is the same: target store controller get locked.

     After I make following changes in migrate.lisp; The test passed.

      (defmethod copy-btree-contents ((sc store-controller) dst src)
            (let (to-be-migrated)
                  (map-btree (lambda (key value)
                                     (let ((newval (migrate sc value))
                                            (newkey (migrate sc key)))
                                           (push (list newkey newval)
to-be-migrated)))
                                           ;;;;    (setf (get-value newkey
dst) newval)))  ;;
                                           src)
                  (loop for (k v) in to-be-migrated
                    do
                    (setf (get-value k dst) v))))

     Above debug change just avoids the nested transaction of different
store controller for the specific test case.
     However, it is not the solution, as the nested transaction still could
appear in other data pattern.

    What confuses me is that the nested transaction of the same store
controller seem not bring any block.
     Following test passed:
        (with-transaction (:store-controller *migrate-dst*)
             (add-to-root 'foo 'foo :sc *migrate-dst*)
             (with-transaction (:store-controller *migrate-dst*)
                (add-to-root 'bar 'bar :sc *migrate-dst*)))

    But, the nested transaction of different store controller always results
in lock.

        (with-transaction (:store-controller *migrate-dst*)
             (add-to-root 'foo 'foo :sc *migrate-dst*)
             (with-transaction (:store-controller **migrate-src**)
;; trigger database lock
                 (add-to-root 'bar 'bar :sc *migrate-dst*)))

Thanks
Quan



2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>

> Quan,
>
> Unstable is here:
>
>  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
>
> Yarek
>
>
> On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu> wrote:
>
>> That would be elephant-unstable
>>
>> Sent from my iPhone
>>
>> On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu> wrote:
>>
>> > Hi Quan,
>> >
>> > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some changes
>> > to migration, but I can't recall if they were in the latest darcs or
>> > not - I think the latest darcs doesn't play well with BDB 4.5 so
>> > please make sure you can reproduce under the above configuration if
>> > you can.
>> >
>> > Thank you,
>> > Ian
>> >
>> > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
>> >
>> >> Hello,
>> >>
>> >>     I run into a problem when doing the garbage collection via data
>> >> migration.
>> >>     The environment is elephant 0.9.1 and BDB 4.5.  I also tried
>> >> the latest elephant in darc and get the same result.
>> >>
>> >>     1.  Test case to reproduce the problem.
>> >>
>> >>           (defpclass user-profile()
>> >>             ((id :initform nil :index t)
>> >>             (sms-inbox  :initform (make-pset))))
>> >>
>> >>            ;; Create a new db for test purpose
>> >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/src/")))
>> >>           (setf *test-obj* (make-instance 'user-profile :sc
>> >> *migrate-src*))
>> >>            (insert-item 'foo (slot-value *test-obj* 'sms-inbox))
>> >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/dst/")))
>> >>
>> >>          TEST> (migrate *migrate-dst* *migrate-src*)
>> >>              Migrating class indexes for: USER-PROFILE
>> >>           ; => Input get blocked, not response anymore
>> >>
>> >>       2.  Using "db_stat -C A" get following output:
>> >>             1    Lock requests not available due to conflicts, for
>> >> which we waited
>> >>
>> >>       3.  I do some debug work and found the problem may be caused
>> >> by nested transaction of different store controller.
>> >>            I can reproduce the issue with following code segment
>> >>
>> >>          TEST> (ensure-transaction (:store-controller *migrate-dst*)
>> >>                            (add-to-root 'foo 'foo :sc *migrate-dst*)
>> >>                            (ensure-transaction (:store-controller
>> >> *migrate-src*)
>> >>                                 (add-to-root 'bar 'bar :sc *migrate-
>> >> dst*)))
>> >>
>> >>             This kind of nested transaction scenario can appear in
>> >> the migrate process, because it uses source/destination store
>> >> controller at the same time.
>> >>
>> >>            I do not understand why different store controller's
>> >> transaction can result in the lock conflict and want to know how to
>> >> fix it.
>> >>
>> >>            Is there anyone meet the similar problem before?
>> >>
>> >> Thanks
>> >> Quan
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> elephant-devel site list
>> >> elephant-devel at common-lisp.net
>> >> http://common-lisp.net/mailman/listinfo/elephant-devel
>> >
>> >
>> > _______________________________________________
>> > elephant-devel site list
>> > elephant-devel at common-lisp.net
>> > http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20081230/c325a402/attachment.html>