Hi Ian,<br><br> The 0.9.1 upgrade fix patch is attached. <br> I continuously run the upgrade test 3 times. All passed. <br><br> In the fix, if the DB version is <= 0.9.1, (make-instance 'bdb-indexed-btree :from-oid -3 :sc sc :indices (make-hash-table) is always called, regardless of the indices exist before. <br>
In my previous fix, I only call it when the indices does not exist in DB file, but find that approach does not work. A unserialized error is triggered when try to open the source 0.9.1 DB file for the second time. This is caused by that the oid->schema-id has an assumption on that instance-table is only created on a brand new DB file. <br>
<br> If this is not a suitable solution, please let me know. <br> <br>Thanks<br>Quan<br> <br><br><br><br><br><div class="gmail_quote">2009/1/2 Ian Eslick <span dir="ltr"><<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">pushed. -Ian<br>
<div><div></div><div class="Wj3C7c"><br>
On Jan 1, 2009, at 8:31 AM, quan hu wrote:<br>
<br>
> Hi Ian,<br>
><br>
> For the pathname fix, I attached the patch.<br>
><br>
> For the 0.9.1 upgrade fix, I got one failure and one<br>
> successful result.<br>
> I need to double check if the failure is caused by my test<br>
> execution step or other issue.<br>
> I'll send the result later.<br>
><br>
> Thanks<br>
> Quan<br>
><br>
><br>
> 2008/12/31 Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> I don't think we've tried the upgrade procedure in awhile, so it's<br>
> possible an incompatibility slipped in. It may be that all you need<br>
> to do is make sure that defaults are properly created for slot values<br>
> that are unbound during the store controller opening phase.<br>
><br>
> Adding a case for pathname is easy enough. I take it one of the slots<br>
> of the pathname structure object is a function; migrating this is<br>
> causing the error.<br>
><br>
> With a little work you should be able to do both of these fixes,<br>
> certainly the pathname fix. I'd be happy to review and commit them to<br>
> the repository, but unfortunately I won't have any more time to work<br>
> on elephant for a little while.<br>
><br>
> Ian<br>
><br>
> On Dec 31, 2008, at 8:37 AM, quan hu wrote:<br>
><br>
> > Hi Ian,<br>
> ><br>
> > I downloaded the patches and have more testing today. The lock<br>
> > issue is gone!<br>
> > Thanks a lot.<br>
> ><br>
> > 1. But, another issue is found. migrate pathname fails. I test<br>
> > it under SBCL 1.0.21<br>
> > e.g.<br>
> > > (migrate *store-controller* #p"/tmp) => Error: Cannot<br>
> > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-<br>
> > ENOUGH>)<br>
> ><br>
> > It is caused by that (defmethod migrate ((dst store-<br>
> > controller) (src structure-object) is called for this case, as<br>
> > pathname is taken as subtype of structure-object.<br>
> > > (subtypep 'pathname 'structure-object) => t t<br>
> > Maybe, one more migrate method is needed for pathname case.<br>
> ><br>
> > 2. I also want to know if upgrading database file from 0.9.1<br>
> > to unstable release is supported.<br>
> > When opening the DB file generated in 0.9.1 in unstable<br>
> > release, I get error message<br>
> > " The slot DB-BDB::INDICES is unbound in the object #<BDB-<br>
> > INDEXED-BTREE oid:-3>.<br>
> > [Condition of type UNBOUND-SLOT]"<br>
> ><br>
> > Thanks<br>
> > Quan<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > 2008/12/31 Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> > Hi Quan,<br>
> ><br>
> > Thank you again for the information on this bug. The short story is<br>
> > that the way we are keeping track of transactions assumed that there<br>
> > were only two stores in the nested transaction stack (outer and<br>
> > inner). Migrating a nested object inside a with-transaction would<br>
> not<br>
> > pass the parent transaction to the backend so it wouldn't know a<br>
> > transaction in progress and attempted to start a new one, which<br>
> would<br>
> > block until the old one was completed.<br>
> ><br>
> > I just checked in a patch that made transaction tracking a bit more<br>
> > intelligent (it depends on the backends taking care of handling<br>
> nested<br>
> > transactions - which I believe they all do fine). I also checked<br>
> in a<br>
> > test which implements Quan's test case.<br>
> ><br>
> > This patch and the prior patches of the past few days pass all tests<br>
> > on SBCL/Mac/BDB.<br>
> ><br>
> > Ian<br>
> ><br>
> > PS - I'd love to know the current status of the elephant-unstable<br>
> tree<br>
> > with the postmodern backend.<br>
> ><br>
> ><br>
> > On Dec 30, 2008, at 9:34 AM, quan hu wrote:<br>
> ><br>
> > > Hello,<br>
> > ><br>
> > > After more investigation, here are some suspects:<br>
> > ><br>
> > > In the current elephant codes, the macro my-current-<br>
> > > transaction will check if the current transaction belongs to the<br>
> sc<br>
> > > parameter. If not, just return +NULL-CHAR+. That's exactly the<br>
> > > case in the following nested transaction where the store-<br>
> controller<br>
> > > is different.<br>
> > > The situation becomes:<br>
> > > Transaction 1 begins<br>
> > > One data is put into DB with Transaction 1<br>
> > > ...<br>
> > > Another data is put into the same DB with NULL<br>
> > > transaction parameter(i.e. no transaction??)<br>
> > ><br>
> > > It looks like this scenario is not supported by BDB.<br>
> > > I simulate above case with a C program to access BDB and<br>
> get<br>
> > > the similar failure symptom.<br>
> > > ...<br>
> > > txn1 = NULL;<br>
> > > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);<br>
> > > if (ret != 0) {<br>
> > > printf ("Transaction begin failed.\n");<br>
> > > return 1;<br>
> > > }<br>
> > > put_record(FOO,FOO,txn1, dbp1);<br>
> > > printf("foo\n");<br>
> > > put_record(FOO,FOO,NULL, dbp1);<br>
> > > printf("bar\n");<br>
> > > txn1->commit(txn1,0);<br>
> > > printf("baz\n");<br>
> > > ...<br>
> > > After "foo" is printed, the C program is blocked. If I change<br>
> the<br>
> > > NULL to txn1, everything is fine.<br>
> > ><br>
> > ><br>
> > > Regarding to the solution, my gut feeling is that it would be<br>
> > > difficult to avoid this kind of nested transaction in migrate<br>
> > > process, because of the recursive function call. Maybe,<br>
> > > *transaction-stack* can be added to track the active transaction<br>
> > > stack. Get it pushed and poped in right place of execute-<br>
> > > transaction. Then, in my-current-transaction, check *transaction-<br>
> > > stack* to see if there is an active transaction belongs to the<br>
> same<br>
> > > store-controller. If it is found, use it. My simple test shows<br>
> it<br>
> > > works. If this is the right direction, I can do more test.<br>
> > ><br>
> > > Thanks for your help.<br>
> > > Quan<br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > ><br>
> > > 2008/12/30 quan hu <<a href="mailto:ihuquan@gmail.com">ihuquan@gmail.com</a>><br>
> > > Hi Ian, Yarek,<br>
> > ><br>
> > > Thanks for providing the information.<br>
> > > I downloaded the elephant-unstable and test it with BDB4.7.<br>
> > > The failure symptom is the same: target store controller get<br>
> > > locked.<br>
> > ><br>
> > > After I make following changes in migrate.lisp; The test<br>
> > passed.<br>
> > ><br>
> > > (defmethod copy-btree-contents ((sc store-controller) dst<br>
> src)<br>
> > > (let (to-be-migrated)<br>
> > > (map-btree (lambda (key value)<br>
> > > (let ((newval (migrate sc<br>
> > value))<br>
> > > (newkey (migrate sc<br>
> > key)))<br>
> > > (push (list newkey<br>
> > > newval) to-be-migrated)))<br>
> > > ;;;; (setf (get-<br>
> value<br>
> > > newkey dst) newval))) ;;<br>
> > > src)<br>
> > > (loop for (k v) in to-be-migrated<br>
> > > do<br>
> > > (setf (get-value k dst) v))))<br>
> > ><br>
> > > Above debug change just avoids the nested transaction of<br>
> > > different store controller for the specific test case.<br>
> > > However, it is not the solution, as the nested transaction<br>
> > > still could appear in other data pattern.<br>
> > ><br>
> > > What confuses me is that the nested transaction of the same<br>
> > > store controller seem not bring any block.<br>
> > > Following test passed:<br>
> > > (with-transaction (:store-controller *migrate-dst*)<br>
> > ><br>
> > > (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
> > > (with-transaction (:store-controller *migrate-dst*)<br>
> > ><br>
> > > (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
> > ><br>
> > > But, the nested transaction of different store controller<br>
> always<br>
> > > results in lock.<br>
> > ><br>
> > > (with-transaction (:store-controller *migrate-dst*)<br>
> > ><br>
> > > (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
> > > (with-transaction (:store-controller *migrate-<br>
> > > src*) ;; trigger database lock<br>
> > > (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
> > ><br>
> > > Thanks<br>
> > > Quan<br>
> > ><br>
> > ><br>
> > ><br>
> > > 2008/12/30 Yarek Kowalik <<a href="mailto:yarek.kowalik@gmail.com">yarek.kowalik@gmail.com</a>><br>
> > ><br>
> > > Quan,<br>
> > ><br>
> > > Unstable is here:<br>
> > ><br>
> > > <a href="http://www.common-lisp.net/project/elephant/darcs/elephant-unstable" target="_blank">http://www.common-lisp.net/project/elephant/darcs/elephant-unstable</a><br>
> > ><br>
> > > Yarek<br>
> > ><br>
> > ><br>
> > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> > > wrote:<br>
> > > That would be elephant-unstable<br>
> > ><br>
> > > Sent from my iPhone<br>
> > ><br>
> > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> > wrote:<br>
> > ><br>
> > > > Hi Quan,<br>
> > > ><br>
> > > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some<br>
> > > changes<br>
> > > > to migration, but I can't recall if they were in the latest<br>
> > darcs or<br>
> > > > not - I think the latest darcs doesn't play well with BDB 4.5 so<br>
> > > > please make sure you can reproduce under the above configuration<br>
> > if<br>
> > > > you can.<br>
> > > ><br>
> > > > Thank you,<br>
> > > > Ian<br>
> > > ><br>
> > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:<br>
> > > ><br>
> > > >> Hello,<br>
> > > >><br>
> > > >> I run into a problem when doing the garbage collection via<br>
> > data<br>
> > > >> migration.<br>
> > > >> The environment is elephant 0.9.1 and BDB 4.5. I also<br>
> tried<br>
> > > >> the latest elephant in darc and get the same result.<br>
> > > >><br>
> > > >> 1. Test case to reproduce the problem.<br>
> > > >><br>
> > > >> (defpclass user-profile()<br>
> > > >> ((id :initform nil :index t)<br>
> > > >> (sms-inbox :initform (make-pset))))<br>
> > > >><br>
> > > >> ;; Create a new db for test purpose<br>
> > > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/<br>
> > src/")))<br>
> > > >> (setf *test-obj* (make-instance 'user-profile :sc<br>
> > > >> *migrate-src*))<br>
> > > >> (insert-item 'foo (slot-value *test-obj* 'sms-<br>
> inbox))<br>
> > > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/<br>
> > dst/")))<br>
> > > >><br>
> > > >> TEST> (migrate *migrate-dst* *migrate-src*)<br>
> > > >> Migrating class indexes for: USER-PROFILE<br>
> > > >> ; => Input get blocked, not response anymore<br>
> > > >><br>
> > > >> 2. Using "db_stat -C A" get following output:<br>
> > > >> 1 Lock requests not available due to conflicts,<br>
> > for<br>
> > > >> which we waited<br>
> > > >><br>
> > > >> 3. I do some debug work and found the problem may be<br>
> > caused<br>
> > > >> by nested transaction of different store controller.<br>
> > > >> I can reproduce the issue with following code<br>
> segment<br>
> > > >><br>
> > > >> TEST> (ensure-transaction (:store-controller *migrate-<br>
> > > dst*)<br>
> > > >> (add-to-root 'foo 'foo :sc *migrate-<br>
> > > dst*)<br>
> > > >> (ensure-transaction (:store-<br>
> controller<br>
> > > >> *migrate-src*)<br>
> > > >> (add-to-root 'bar 'bar :sc<br>
> > > *migrate-<br>
> > > >> dst*)))<br>
> > > >><br>
> > > >> This kind of nested transaction scenario can appear<br>
> > in<br>
> > > >> the migrate process, because it uses source/destination store<br>
> > > >> controller at the same time.<br>
> > > >><br>
> > > >> I do not understand why different store controller's<br>
> > > >> transaction can result in the lock conflict and want to know<br>
> > how to<br>
> > > >> fix it.<br>
> > > >><br>
> > > >> Is there anyone meet the similar problem before?<br>
> > > >><br>
> > > >> Thanks<br>
> > > >> Quan<br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >> _______________________________________________<br>
> > > >> elephant-devel site list<br>
> > > >> <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > > >> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> > > ><br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > elephant-devel site list<br>
> > > > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > > > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> > ><br>
> > > _______________________________________________<br>
> > > elephant-devel site list<br>
> > > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > elephant-devel site list<br>
> > > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > elephant-devel site list<br>
> > > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > elephant-devel site list<br>
> > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> ><br>
> > _______________________________________________<br>
> > elephant-devel site list<br>
> > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
</div></div>> <migrate-pathname-<br>
> patch.gz>_______________________________________________<br>
<div><div></div><div class="Wj3C7c">> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
<br>
<br>
_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
</div></div></blockquote></div><br>