[elephant-devel] DB migration issue
Ian Eslick
eslick at media.mit.edu
Thu Jan 1 20:39:17 UTC 2009
pushed. -Ian
On Jan 1, 2009, at 8:31 AM, quan hu wrote:
> Hi Ian,
>
> For the pathname fix, I attached the patch.
>
> For the 0.9.1 upgrade fix, I got one failure and one
> successful result.
> I need to double check if the failure is caused by my test
> execution step or other issue.
> I'll send the result later.
>
> Thanks
> Quan
>
>
> 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> I don't think we've tried the upgrade procedure in awhile, so it's
> possible an incompatibility slipped in. It may be that all you need
> to do is make sure that defaults are properly created for slot values
> that are unbound during the store controller opening phase.
>
> Adding a case for pathname is easy enough. I take it one of the slots
> of the pathname structure object is a function; migrating this is
> causing the error.
>
> With a little work you should be able to do both of these fixes,
> certainly the pathname fix. I'd be happy to review and commit them to
> the repository, but unfortunately I won't have any more time to work
> on elephant for a little while.
>
> Ian
>
> On Dec 31, 2008, at 8:37 AM, quan hu wrote:
>
> > Hi Ian,
> >
> > I downloaded the patches and have more testing today. The lock
> > issue is gone!
> > Thanks a lot.
> >
> > 1. But, another issue is found. migrate pathname fails. I test
> > it under SBCL 1.0.21
> > e.g.
> > > (migrate *store-controller* #p"/tmp) => Error: Cannot
> > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > ENOUGH>)
> >
> > It is caused by that (defmethod migrate ((dst store-
> > controller) (src structure-object) is called for this case, as
> > pathname is taken as subtype of structure-object.
> > > (subtypep 'pathname 'structure-object) => t t
> > Maybe, one more migrate method is needed for pathname case.
> >
> > 2. I also want to know if upgrading database file from 0.9.1
> > to unstable release is supported.
> > When opening the DB file generated in 0.9.1 in unstable
> > release, I get error message
> > " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > INDEXED-BTREE oid:-3>.
> > [Condition of type UNBOUND-SLOT]"
> >
> > Thanks
> > Quan
> >
> >
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > Hi Quan,
> >
> > Thank you again for the information on this bug. The short story is
> > that the way we are keeping track of transactions assumed that there
> > were only two stores in the nested transaction stack (outer and
> > inner). Migrating a nested object inside a with-transaction would
> not
> > pass the parent transaction to the backend so it wouldn't know a
> > transaction in progress and attempted to start a new one, which
> would
> > block until the old one was completed.
> >
> > I just checked in a patch that made transaction tracking a bit more
> > intelligent (it depends on the backends taking care of handling
> nested
> > transactions - which I believe they all do fine). I also checked
> in a
> > test which implements Quan's test case.
> >
> > This patch and the prior patches of the past few days pass all tests
> > on SBCL/Mac/BDB.
> >
> > Ian
> >
> > PS - I'd love to know the current status of the elephant-unstable
> tree
> > with the postmodern backend.
> >
> >
> > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> >
> > > Hello,
> > >
> > > After more investigation, here are some suspects:
> > >
> > > In the current elephant codes, the macro my-current-
> > > transaction will check if the current transaction belongs to the
> sc
> > > parameter. If not, just return +NULL-CHAR+. That's exactly the
> > > case in the following nested transaction where the store-
> controller
> > > is different.
> > > The situation becomes:
> > > Transaction 1 begins
> > > One data is put into DB with Transaction 1
> > > ...
> > > Another data is put into the same DB with NULL
> > > transaction parameter(i.e. no transaction??)
> > >
> > > It looks like this scenario is not supported by BDB.
> > > I simulate above case with a C program to access BDB and
> get
> > > the similar failure symptom.
> > > ...
> > > txn1 = NULL;
> > > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > > if (ret != 0) {
> > > printf ("Transaction begin failed.\n");
> > > return 1;
> > > }
> > > put_record(FOO,FOO,txn1, dbp1);
> > > printf("foo\n");
> > > put_record(FOO,FOO,NULL, dbp1);
> > > printf("bar\n");
> > > txn1->commit(txn1,0);
> > > printf("baz\n");
> > > ...
> > > After "foo" is printed, the C program is blocked. If I change
> the
> > > NULL to txn1, everything is fine.
> > >
> > >
> > > Regarding to the solution, my gut feeling is that it would be
> > > difficult to avoid this kind of nested transaction in migrate
> > > process, because of the recursive function call. Maybe,
> > > *transaction-stack* can be added to track the active transaction
> > > stack. Get it pushed and poped in right place of execute-
> > > transaction. Then, in my-current-transaction, check *transaction-
> > > stack* to see if there is an active transaction belongs to the
> same
> > > store-controller. If it is found, use it. My simple test shows
> it
> > > works. If this is the right direction, I can do more test.
> > >
> > > Thanks for your help.
> > > Quan
> > >
> > >
> > >
> > >
> > >
> > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > Hi Ian, Yarek,
> > >
> > > Thanks for providing the information.
> > > I downloaded the elephant-unstable and test it with BDB4.7.
> > > The failure symptom is the same: target store controller get
> > > locked.
> > >
> > > After I make following changes in migrate.lisp; The test
> > passed.
> > >
> > > (defmethod copy-btree-contents ((sc store-controller) dst
> src)
> > > (let (to-be-migrated)
> > > (map-btree (lambda (key value)
> > > (let ((newval (migrate sc
> > value))
> > > (newkey (migrate sc
> > key)))
> > > (push (list newkey
> > > newval) to-be-migrated)))
> > > ;;;; (setf (get-
> value
> > > newkey dst) newval))) ;;
> > > src)
> > > (loop for (k v) in to-be-migrated
> > > do
> > > (setf (get-value k dst) v))))
> > >
> > > Above debug change just avoids the nested transaction of
> > > different store controller for the specific test case.
> > > However, it is not the solution, as the nested transaction
> > > still could appear in other data pattern.
> > >
> > > What confuses me is that the nested transaction of the same
> > > store controller seem not bring any block.
> > > Following test passed:
> > > (with-transaction (:store-controller *migrate-dst*)
> > >
> > > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > (with-transaction (:store-controller *migrate-dst*)
> > >
> > > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > > But, the nested transaction of different store controller
> always
> > > results in lock.
> > >
> > > (with-transaction (:store-controller *migrate-dst*)
> > >
> > > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > (with-transaction (:store-controller *migrate-
> > > src*) ;; trigger database lock
> > > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > >
> > > Quan,
> > >
> > > Unstable is here:
> > >
> > > http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > >
> > > Yarek
> > >
> > >
> > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > That would be elephant-unstable
> > >
> > > Sent from my iPhone
> > >
> > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > >
> > > > Hi Quan,
> > > >
> > > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some
> > > changes
> > > > to migration, but I can't recall if they were in the latest
> > darcs or
> > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > please make sure you can reproduce under the above configuration
> > if
> > > > you can.
> > > >
> > > > Thank you,
> > > > Ian
> > > >
> > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> I run into a problem when doing the garbage collection via
> > data
> > > >> migration.
> > > >> The environment is elephant 0.9.1 and BDB 4.5. I also
> tried
> > > >> the latest elephant in darc and get the same result.
> > > >>
> > > >> 1. Test case to reproduce the problem.
> > > >>
> > > >> (defpclass user-profile()
> > > >> ((id :initform nil :index t)
> > > >> (sms-inbox :initform (make-pset))))
> > > >>
> > > >> ;; Create a new db for test purpose
> > > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > src/")))
> > > >> (setf *test-obj* (make-instance 'user-profile :sc
> > > >> *migrate-src*))
> > > >> (insert-item 'foo (slot-value *test-obj* 'sms-
> inbox))
> > > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > dst/")))
> > > >>
> > > >> TEST> (migrate *migrate-dst* *migrate-src*)
> > > >> Migrating class indexes for: USER-PROFILE
> > > >> ; => Input get blocked, not response anymore
> > > >>
> > > >> 2. Using "db_stat -C A" get following output:
> > > >> 1 Lock requests not available due to conflicts,
> > for
> > > >> which we waited
> > > >>
> > > >> 3. I do some debug work and found the problem may be
> > caused
> > > >> by nested transaction of different store controller.
> > > >> I can reproduce the issue with following code
> segment
> > > >>
> > > >> TEST> (ensure-transaction (:store-controller *migrate-
> > > dst*)
> > > >> (add-to-root 'foo 'foo :sc *migrate-
> > > dst*)
> > > >> (ensure-transaction (:store-
> controller
> > > >> *migrate-src*)
> > > >> (add-to-root 'bar 'bar :sc
> > > *migrate-
> > > >> dst*)))
> > > >>
> > > >> This kind of nested transaction scenario can appear
> > in
> > > >> the migrate process, because it uses source/destination store
> > > >> controller at the same time.
> > > >>
> > > >> I do not understand why different store controller's
> > > >> transaction can result in the lock conflict and want to know
> > how to
> > > >> fix it.
> > > >>
> > > >> Is there anyone meet the similar problem before?
> > > >>
> > > >> Thanks
> > > >> Quan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> elephant-devel site list
> > > >> elephant-devel at common-lisp.net
> > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> <migrate-pathname-
> patch.gz>_______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
More information about the elephant-devel
mailing list