[elephant-devel] DB migration issue
quan hu
ihuquan at gmail.com
Wed Dec 31 13:37:54 UTC 2008
Hi Ian,
I downloaded the patches and have more testing today. The lock issue is
gone!
Thanks a lot.
1. But, another issue is found. migrate pathname fails. I test it under
SBCL 1.0.21
e.g.
> (migrate *store-controller* #p"/tmp) => Error: Cannot migrate
function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-ENOUGH>)
It is caused by that (defmethod migrate ((dst store-controller) (src
structure-object) is called for this case, as pathname is taken as subtype
of structure-object.
> (subtypep 'pathname 'structure-object) => t t
Maybe, one more migrate method is needed for pathname case.
2. I also want to know if upgrading database file from 0.9.1 to
unstable release is supported.
When opening the DB file generated in 0.9.1 in unstable release, I get
error message
" The slot DB-BDB::INDICES is unbound in the object
#<BDB-INDEXED-BTREE oid:-3>.
[Condition of type UNBOUND-SLOT]"
Thanks
Quan
2008/12/31 Ian Eslick <eslick at media.mit.edu>
> Hi Quan,
>
> Thank you again for the information on this bug. The short story is
> that the way we are keeping track of transactions assumed that there
> were only two stores in the nested transaction stack (outer and
> inner). Migrating a nested object inside a with-transaction would not
> pass the parent transaction to the backend so it wouldn't know a
> transaction in progress and attempted to start a new one, which would
> block until the old one was completed.
>
> I just checked in a patch that made transaction tracking a bit more
> intelligent (it depends on the backends taking care of handling nested
> transactions - which I believe they all do fine). I also checked in a
> test which implements Quan's test case.
>
> This patch and the prior patches of the past few days pass all tests
> on SBCL/Mac/BDB.
>
> Ian
>
> PS - I'd love to know the current status of the elephant-unstable tree
> with the postmodern backend.
>
>
> On Dec 30, 2008, at 9:34 AM, quan hu wrote:
>
> > Hello,
> >
> > After more investigation, here are some suspects:
> >
> > In the current elephant codes, the macro my-current-
> > transaction will check if the current transaction belongs to the sc
> > parameter. If not, just return +NULL-CHAR+. That's exactly the
> > case in the following nested transaction where the store-controller
> > is different.
> > The situation becomes:
> > Transaction 1 begins
> > One data is put into DB with Transaction 1
> > ...
> > Another data is put into the same DB with NULL
> > transaction parameter(i.e. no transaction??)
> >
> > It looks like this scenario is not supported by BDB.
> > I simulate above case with a C program to access BDB and get
> > the similar failure symptom.
> > ...
> > txn1 = NULL;
> > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > if (ret != 0) {
> > printf ("Transaction begin failed.\n");
> > return 1;
> > }
> > put_record(FOO,FOO,txn1, dbp1);
> > printf("foo\n");
> > put_record(FOO,FOO,NULL, dbp1);
> > printf("bar\n");
> > txn1->commit(txn1,0);
> > printf("baz\n");
> > ...
> > After "foo" is printed, the C program is blocked. If I change the
> > NULL to txn1, everything is fine.
> >
> >
> > Regarding to the solution, my gut feeling is that it would be
> > difficult to avoid this kind of nested transaction in migrate
> > process, because of the recursive function call. Maybe,
> > *transaction-stack* can be added to track the active transaction
> > stack. Get it pushed and poped in right place of execute-
> > transaction. Then, in my-current-transaction, check *transaction-
> > stack* to see if there is an active transaction belongs to the same
> > store-controller. If it is found, use it. My simple test shows it
> > works. If this is the right direction, I can do more test.
> >
> > Thanks for your help.
> > Quan
> >
> >
> >
> >
> >
> > 2008/12/30 quan hu <ihuquan at gmail.com>
> > Hi Ian, Yarek,
> >
> > Thanks for providing the information.
> > I downloaded the elephant-unstable and test it with BDB4.7.
> > The failure symptom is the same: target store controller get
> > locked.
> >
> > After I make following changes in migrate.lisp; The test passed.
> >
> > (defmethod copy-btree-contents ((sc store-controller) dst src)
> > (let (to-be-migrated)
> > (map-btree (lambda (key value)
> > (let ((newval (migrate sc value))
> > (newkey (migrate sc key)))
> > (push (list newkey
> > newval) to-be-migrated)))
> > ;;;; (setf (get-value
> > newkey dst) newval))) ;;
> > src)
> > (loop for (k v) in to-be-migrated
> > do
> > (setf (get-value k dst) v))))
> >
> > Above debug change just avoids the nested transaction of
> > different store controller for the specific test case.
> > However, it is not the solution, as the nested transaction
> > still could appear in other data pattern.
> >
> > What confuses me is that the nested transaction of the same
> > store controller seem not bring any block.
> > Following test passed:
> > (with-transaction (:store-controller *migrate-dst*)
> >
> > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > (with-transaction (:store-controller *migrate-dst*)
> >
> > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> >
> > But, the nested transaction of different store controller always
> > results in lock.
> >
> > (with-transaction (:store-controller *migrate-dst*)
> >
> > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > (with-transaction (:store-controller *migrate-
> > src*) ;; trigger database lock
> > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> >
> > Thanks
> > Quan
> >
> >
> >
> > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> >
> > Quan,
> >
> > Unstable is here:
> >
> > http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> >
> > Yarek
> >
> >
> > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > That would be elephant-unstable
> >
> > Sent from my iPhone
> >
> > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu> wrote:
> >
> > > Hi Quan,
> > >
> > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some
> > changes
> > > to migration, but I can't recall if they were in the latest darcs or
> > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > please make sure you can reproduce under the above configuration if
> > > you can.
> > >
> > > Thank you,
> > > Ian
> > >
> > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > >
> > >> Hello,
> > >>
> > >> I run into a problem when doing the garbage collection via data
> > >> migration.
> > >> The environment is elephant 0.9.1 and BDB 4.5. I also tried
> > >> the latest elephant in darc and get the same result.
> > >>
> > >> 1. Test case to reproduce the problem.
> > >>
> > >> (defpclass user-profile()
> > >> ((id :initform nil :index t)
> > >> (sms-inbox :initform (make-pset))))
> > >>
> > >> ;; Create a new db for test purpose
> > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/src/")))
> > >> (setf *test-obj* (make-instance 'user-profile :sc
> > >> *migrate-src*))
> > >> (insert-item 'foo (slot-value *test-obj* 'sms-inbox))
> > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/dst/")))
> > >>
> > >> TEST> (migrate *migrate-dst* *migrate-src*)
> > >> Migrating class indexes for: USER-PROFILE
> > >> ; => Input get blocked, not response anymore
> > >>
> > >> 2. Using "db_stat -C A" get following output:
> > >> 1 Lock requests not available due to conflicts, for
> > >> which we waited
> > >>
> > >> 3. I do some debug work and found the problem may be caused
> > >> by nested transaction of different store controller.
> > >> I can reproduce the issue with following code segment
> > >>
> > >> TEST> (ensure-transaction (:store-controller *migrate-
> > dst*)
> > >> (add-to-root 'foo 'foo :sc *migrate-
> > dst*)
> > >> (ensure-transaction (:store-controller
> > >> *migrate-src*)
> > >> (add-to-root 'bar 'bar :sc
> > *migrate-
> > >> dst*)))
> > >>
> > >> This kind of nested transaction scenario can appear in
> > >> the migrate process, because it uses source/destination store
> > >> controller at the same time.
> > >>
> > >> I do not understand why different store controller's
> > >> transaction can result in the lock conflict and want to know how to
> > >> fix it.
> > >>
> > >> Is there anyone meet the similar problem before?
> > >>
> > >> Thanks
> > >> Quan
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> elephant-devel site list
> > >> elephant-devel at common-lisp.net
> > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20081231/5cb81961/attachment.html>
More information about the elephant-devel
mailing list