[elephant-devel] DB migration issue
Ian Eslick
eslick at media.mit.edu
Wed Dec 31 15:27:02 UTC 2008
I don't think we've tried the upgrade procedure in awhile, so it's
possible an incompatibility slipped in. It may be that all you need
to do is make sure that defaults are properly created for slot values
that are unbound during the store controller opening phase.
Adding a case for pathname is easy enough. I take it one of the slots
of the pathname structure object is a function; migrating this is
causing the error.
With a little work you should be able to do both of these fixes,
certainly the pathname fix. I'd be happy to review and commit them to
the repository, but unfortunately I won't have any more time to work
on elephant for a little while.
Ian
On Dec 31, 2008, at 8:37 AM, quan hu wrote:
> Hi Ian,
>
> I downloaded the patches and have more testing today. The lock
> issue is gone!
> Thanks a lot.
>
> 1. But, another issue is found. migrate pathname fails. I test
> it under SBCL 1.0.21
> e.g.
> > (migrate *store-controller* #p"/tmp) => Error: Cannot
> migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> ENOUGH>)
>
> It is caused by that (defmethod migrate ((dst store-
> controller) (src structure-object) is called for this case, as
> pathname is taken as subtype of structure-object.
> > (subtypep 'pathname 'structure-object) => t t
> Maybe, one more migrate method is needed for pathname case.
>
> 2. I also want to know if upgrading database file from 0.9.1
> to unstable release is supported.
> When opening the DB file generated in 0.9.1 in unstable
> release, I get error message
> " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> INDEXED-BTREE oid:-3>.
> [Condition of type UNBOUND-SLOT]"
>
> Thanks
> Quan
>
>
>
>
> 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> Hi Quan,
>
> Thank you again for the information on this bug. The short story is
> that the way we are keeping track of transactions assumed that there
> were only two stores in the nested transaction stack (outer and
> inner). Migrating a nested object inside a with-transaction would not
> pass the parent transaction to the backend so it wouldn't know a
> transaction in progress and attempted to start a new one, which would
> block until the old one was completed.
>
> I just checked in a patch that made transaction tracking a bit more
> intelligent (it depends on the backends taking care of handling nested
> transactions - which I believe they all do fine). I also checked in a
> test which implements Quan's test case.
>
> This patch and the prior patches of the past few days pass all tests
> on SBCL/Mac/BDB.
>
> Ian
>
> PS - I'd love to know the current status of the elephant-unstable tree
> with the postmodern backend.
>
>
> On Dec 30, 2008, at 9:34 AM, quan hu wrote:
>
> > Hello,
> >
> > After more investigation, here are some suspects:
> >
> > In the current elephant codes, the macro my-current-
> > transaction will check if the current transaction belongs to the sc
> > parameter. If not, just return +NULL-CHAR+. That's exactly the
> > case in the following nested transaction where the store-controller
> > is different.
> > The situation becomes:
> > Transaction 1 begins
> > One data is put into DB with Transaction 1
> > ...
> > Another data is put into the same DB with NULL
> > transaction parameter(i.e. no transaction??)
> >
> > It looks like this scenario is not supported by BDB.
> > I simulate above case with a C program to access BDB and get
> > the similar failure symptom.
> > ...
> > txn1 = NULL;
> > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > if (ret != 0) {
> > printf ("Transaction begin failed.\n");
> > return 1;
> > }
> > put_record(FOO,FOO,txn1, dbp1);
> > printf("foo\n");
> > put_record(FOO,FOO,NULL, dbp1);
> > printf("bar\n");
> > txn1->commit(txn1,0);
> > printf("baz\n");
> > ...
> > After "foo" is printed, the C program is blocked. If I change the
> > NULL to txn1, everything is fine.
> >
> >
> > Regarding to the solution, my gut feeling is that it would be
> > difficult to avoid this kind of nested transaction in migrate
> > process, because of the recursive function call. Maybe,
> > *transaction-stack* can be added to track the active transaction
> > stack. Get it pushed and poped in right place of execute-
> > transaction. Then, in my-current-transaction, check *transaction-
> > stack* to see if there is an active transaction belongs to the same
> > store-controller. If it is found, use it. My simple test shows it
> > works. If this is the right direction, I can do more test.
> >
> > Thanks for your help.
> > Quan
> >
> >
> >
> >
> >
> > 2008/12/30 quan hu <ihuquan at gmail.com>
> > Hi Ian, Yarek,
> >
> > Thanks for providing the information.
> > I downloaded the elephant-unstable and test it with BDB4.7.
> > The failure symptom is the same: target store controller get
> > locked.
> >
> > After I make following changes in migrate.lisp; The test
> passed.
> >
> > (defmethod copy-btree-contents ((sc store-controller) dst src)
> > (let (to-be-migrated)
> > (map-btree (lambda (key value)
> > (let ((newval (migrate sc
> value))
> > (newkey (migrate sc
> key)))
> > (push (list newkey
> > newval) to-be-migrated)))
> > ;;;; (setf (get-value
> > newkey dst) newval))) ;;
> > src)
> > (loop for (k v) in to-be-migrated
> > do
> > (setf (get-value k dst) v))))
> >
> > Above debug change just avoids the nested transaction of
> > different store controller for the specific test case.
> > However, it is not the solution, as the nested transaction
> > still could appear in other data pattern.
> >
> > What confuses me is that the nested transaction of the same
> > store controller seem not bring any block.
> > Following test passed:
> > (with-transaction (:store-controller *migrate-dst*)
> >
> > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > (with-transaction (:store-controller *migrate-dst*)
> >
> > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> >
> > But, the nested transaction of different store controller always
> > results in lock.
> >
> > (with-transaction (:store-controller *migrate-dst*)
> >
> > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > (with-transaction (:store-controller *migrate-
> > src*) ;; trigger database lock
> > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> >
> > Thanks
> > Quan
> >
> >
> >
> > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> >
> > Quan,
> >
> > Unstable is here:
> >
> > http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> >
> > Yarek
> >
> >
> > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > That would be elephant-unstable
> >
> > Sent from my iPhone
> >
> > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> wrote:
> >
> > > Hi Quan,
> > >
> > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some
> > changes
> > > to migration, but I can't recall if they were in the latest
> darcs or
> > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > please make sure you can reproduce under the above configuration
> if
> > > you can.
> > >
> > > Thank you,
> > > Ian
> > >
> > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > >
> > >> Hello,
> > >>
> > >> I run into a problem when doing the garbage collection via
> data
> > >> migration.
> > >> The environment is elephant 0.9.1 and BDB 4.5. I also tried
> > >> the latest elephant in darc and get the same result.
> > >>
> > >> 1. Test case to reproduce the problem.
> > >>
> > >> (defpclass user-profile()
> > >> ((id :initform nil :index t)
> > >> (sms-inbox :initform (make-pset))))
> > >>
> > >> ;; Create a new db for test purpose
> > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> src/")))
> > >> (setf *test-obj* (make-instance 'user-profile :sc
> > >> *migrate-src*))
> > >> (insert-item 'foo (slot-value *test-obj* 'sms-inbox))
> > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> dst/")))
> > >>
> > >> TEST> (migrate *migrate-dst* *migrate-src*)
> > >> Migrating class indexes for: USER-PROFILE
> > >> ; => Input get blocked, not response anymore
> > >>
> > >> 2. Using "db_stat -C A" get following output:
> > >> 1 Lock requests not available due to conflicts,
> for
> > >> which we waited
> > >>
> > >> 3. I do some debug work and found the problem may be
> caused
> > >> by nested transaction of different store controller.
> > >> I can reproduce the issue with following code segment
> > >>
> > >> TEST> (ensure-transaction (:store-controller *migrate-
> > dst*)
> > >> (add-to-root 'foo 'foo :sc *migrate-
> > dst*)
> > >> (ensure-transaction (:store-controller
> > >> *migrate-src*)
> > >> (add-to-root 'bar 'bar :sc
> > *migrate-
> > >> dst*)))
> > >>
> > >> This kind of nested transaction scenario can appear
> in
> > >> the migrate process, because it uses source/destination store
> > >> controller at the same time.
> > >>
> > >> I do not understand why different store controller's
> > >> transaction can result in the lock conflict and want to know
> how to
> > >> fix it.
> > >>
> > >> Is there anyone meet the similar problem before?
> > >>
> > >> Thanks
> > >> Quan
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> elephant-devel site list
> > >> elephant-devel at common-lisp.net
> > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
More information about the elephant-devel
mailing list