Hi Ian,<br>
<br>
For the pathname fix, I attached the patch.<br>
<br>
For the 0.9.1 upgrade fix, I got one failure and one successful result.<br>
I need to double check if the failure is caused by my test execution step or other issue.<br>
I'll send the result later.<br>
<br>
Thanks<br>
Quan<br>
<br><br><div class="gmail_quote">2008/12/31 Ian Eslick <span dir="ltr"><<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I don't think we've tried the upgrade procedure in awhile, so it's<br>
possible an incompatibility slipped in. It may be that all you need<br>
to do is make sure that defaults are properly created for slot values<br>
that are unbound during the store controller opening phase.<br>
<br>
Adding a case for pathname is easy enough. I take it one of the slots<br>
of the pathname structure object is a function; migrating this is<br>
causing the error.<br>
<br>
With a little work you should be able to do both of these fixes,<br>
certainly the pathname fix. I'd be happy to review and commit them to<br>
the repository, but unfortunately I won't have any more time to work<br>
on elephant for a little while.<br>
<font color="#888888"><br>
Ian<br>
</font><div><div></div><div class="Wj3C7c"><br>
On Dec 31, 2008, at 8:37 AM, quan hu wrote:<br>
<br>
> Hi Ian,<br>
><br>
> I downloaded the patches and have more testing today. The lock<br>
> issue is gone!<br>
> Thanks a lot.<br>
><br>
> 1. But, another issue is found. migrate pathname fails. I test<br>
> it under SBCL 1.0.21<br>
> e.g.<br>
> > (migrate *store-controller* #p"/tmp) => Error: Cannot<br>
> migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-<br>
> ENOUGH>)<br>
><br>
> It is caused by that (defmethod migrate ((dst store-<br>
> controller) (src structure-object) is called for this case, as<br>
> pathname is taken as subtype of structure-object.<br>
> > (subtypep 'pathname 'structure-object) => t t<br>
> Maybe, one more migrate method is needed for pathname case.<br>
><br>
> 2. I also want to know if upgrading database file from 0.9.1<br>
> to unstable release is supported.<br>
> When opening the DB file generated in 0.9.1 in unstable<br>
> release, I get error message<br>
> " The slot DB-BDB::INDICES is unbound in the object #<BDB-<br>
> INDEXED-BTREE oid:-3>.<br>
> [Condition of type UNBOUND-SLOT]"<br>
><br>
> Thanks<br>
> Quan<br>
><br>
><br>
><br>
><br>
> 2008/12/31 Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> Hi Quan,<br>
><br>
> Thank you again for the information on this bug. The short story is<br>
> that the way we are keeping track of transactions assumed that there<br>
> were only two stores in the nested transaction stack (outer and<br>
> inner). Migrating a nested object inside a with-transaction would not<br>
> pass the parent transaction to the backend so it wouldn't know a<br>
> transaction in progress and attempted to start a new one, which would<br>
> block until the old one was completed.<br>
><br>
> I just checked in a patch that made transaction tracking a bit more<br>
> intelligent (it depends on the backends taking care of handling nested<br>
> transactions - which I believe they all do fine). I also checked in a<br>
> test which implements Quan's test case.<br>
><br>
> This patch and the prior patches of the past few days pass all tests<br>
> on SBCL/Mac/BDB.<br>
><br>
> Ian<br>
><br>
> PS - I'd love to know the current status of the elephant-unstable tree<br>
> with the postmodern backend.<br>
><br>
><br>
> On Dec 30, 2008, at 9:34 AM, quan hu wrote:<br>
><br>
> > Hello,<br>
> ><br>
> > After more investigation, here are some suspects:<br>
> ><br>
> > In the current elephant codes, the macro my-current-<br>
> > transaction will check if the current transaction belongs to the sc<br>
> > parameter. If not, just return +NULL-CHAR+. That's exactly the<br>
> > case in the following nested transaction where the store-controller<br>
> > is different.<br>
> > The situation becomes:<br>
> > Transaction 1 begins<br>
> > One data is put into DB with Transaction 1<br>
> > ...<br>
> > Another data is put into the same DB with NULL<br>
> > transaction parameter(i.e. no transaction??)<br>
> ><br>
> > It looks like this scenario is not supported by BDB.<br>
> > I simulate above case with a C program to access BDB and get<br>
> > the similar failure symptom.<br>
> > ...<br>
> > txn1 = NULL;<br>
> > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);<br>
> > if (ret != 0) {<br>
> > printf ("Transaction begin failed.\n");<br>
> > return 1;<br>
> > }<br>
> > put_record(FOO,FOO,txn1, dbp1);<br>
> > printf("foo\n");<br>
> > put_record(FOO,FOO,NULL, dbp1);<br>
> > printf("bar\n");<br>
> > txn1->commit(txn1,0);<br>
> > printf("baz\n");<br>
> > ...<br>
> > After "foo" is printed, the C program is blocked. If I change the<br>
> > NULL to txn1, everything is fine.<br>
> ><br>
> ><br>
> > Regarding to the solution, my gut feeling is that it would be<br>
> > difficult to avoid this kind of nested transaction in migrate<br>
> > process, because of the recursive function call. Maybe,<br>
> > *transaction-stack* can be added to track the active transaction<br>
> > stack. Get it pushed and poped in right place of execute-<br>
> > transaction. Then, in my-current-transaction, check *transaction-<br>
> > stack* to see if there is an active transaction belongs to the same<br>
> > store-controller. If it is found, use it. My simple test shows it<br>
> > works. If this is the right direction, I can do more test.<br>
> ><br>
> > Thanks for your help.<br>
> > Quan<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > 2008/12/30 quan hu <<a href="mailto:ihuquan@gmail.com">ihuquan@gmail.com</a>><br>
> > Hi Ian, Yarek,<br>
> ><br>
> > Thanks for providing the information.<br>
> > I downloaded the elephant-unstable and test it with BDB4.7.<br>
> > The failure symptom is the same: target store controller get<br>
> > locked.<br>
> ><br>
> > After I make following changes in migrate.lisp; The test<br>
> passed.<br>
> ><br>
> > (defmethod copy-btree-contents ((sc store-controller) dst src)<br>
> > (let (to-be-migrated)<br>
> > (map-btree (lambda (key value)<br>
> > (let ((newval (migrate sc<br>
> value))<br>
> > (newkey (migrate sc<br>
> key)))<br>
> > (push (list newkey<br>
> > newval) to-be-migrated)))<br>
> > ;;;; (setf (get-value<br>
> > newkey dst) newval))) ;;<br>
> > src)<br>
> > (loop for (k v) in to-be-migrated<br>
> > do<br>
> > (setf (get-value k dst) v))))<br>
> ><br>
> > Above debug change just avoids the nested transaction of<br>
> > different store controller for the specific test case.<br>
> > However, it is not the solution, as the nested transaction<br>
> > still could appear in other data pattern.<br>
> ><br>
> > What confuses me is that the nested transaction of the same<br>
> > store controller seem not bring any block.<br>
> > Following test passed:<br>
> > (with-transaction (:store-controller *migrate-dst*)<br>
> ><br>
> > (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
> > (with-transaction (:store-controller *migrate-dst*)<br>
> ><br>
> > (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
> ><br>
> > But, the nested transaction of different store controller always<br>
> > results in lock.<br>
> ><br>
> > (with-transaction (:store-controller *migrate-dst*)<br>
> ><br>
> > (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
> > (with-transaction (:store-controller *migrate-<br>
> > src*) ;; trigger database lock<br>
> > (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
> ><br>
> > Thanks<br>
> > Quan<br>
> ><br>
> ><br>
> ><br>
> > 2008/12/30 Yarek Kowalik <<a href="mailto:yarek.kowalik@gmail.com">yarek.kowalik@gmail.com</a>><br>
> ><br>
> > Quan,<br>
> ><br>
> > Unstable is here:<br>
> ><br>
> > <a href="http://www.common-lisp.net/project/elephant/darcs/elephant-unstable" target="_blank">http://www.common-lisp.net/project/elephant/darcs/elephant-unstable</a><br>
> ><br>
> > Yarek<br>
> ><br>
> ><br>
> > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> > wrote:<br>
> > That would be elephant-unstable<br>
> ><br>
> > Sent from my iPhone<br>
> ><br>
> > On Dec 29, 2008, at 7:36 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a>><br>
> wrote:<br>
> ><br>
> > > Hi Quan,<br>
> > ><br>
> > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some<br>
> > changes<br>
> > > to migration, but I can't recall if they were in the latest<br>
> darcs or<br>
> > > not - I think the latest darcs doesn't play well with BDB 4.5 so<br>
> > > please make sure you can reproduce under the above configuration<br>
> if<br>
> > > you can.<br>
> > ><br>
> > > Thank you,<br>
> > > Ian<br>
> > ><br>
> > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:<br>
> > ><br>
> > >> Hello,<br>
> > >><br>
> > >> I run into a problem when doing the garbage collection via<br>
> data<br>
> > >> migration.<br>
> > >> The environment is elephant 0.9.1 and BDB 4.5. I also tried<br>
> > >> the latest elephant in darc and get the same result.<br>
> > >><br>
> > >> 1. Test case to reproduce the problem.<br>
> > >><br>
> > >> (defpclass user-profile()<br>
> > >> ((id :initform nil :index t)<br>
> > >> (sms-inbox :initform (make-pset))))<br>
> > >><br>
> > >> ;; Create a new db for test purpose<br>
> > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/<br>
> src/")))<br>
> > >> (setf *test-obj* (make-instance 'user-profile :sc<br>
> > >> *migrate-src*))<br>
> > >> (insert-item 'foo (slot-value *test-obj* 'sms-inbox))<br>
> > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/<br>
> dst/")))<br>
> > >><br>
> > >> TEST> (migrate *migrate-dst* *migrate-src*)<br>
> > >> Migrating class indexes for: USER-PROFILE<br>
> > >> ; => Input get blocked, not response anymore<br>
> > >><br>
> > >> 2. Using "db_stat -C A" get following output:<br>
> > >> 1 Lock requests not available due to conflicts,<br>
> for<br>
> > >> which we waited<br>
> > >><br>
> > >> 3. I do some debug work and found the problem may be<br>
> caused<br>
> > >> by nested transaction of different store controller.<br>
> > >> I can reproduce the issue with following code segment<br>
> > >><br>
> > >> TEST> (ensure-transaction (:store-controller *migrate-<br>
> > dst*)<br>
> > >> (add-to-root 'foo 'foo :sc *migrate-<br>
> > dst*)<br>
> > >> (ensure-transaction (:store-controller<br>
> > >> *migrate-src*)<br>
> > >> (add-to-root 'bar 'bar :sc<br>
> > *migrate-<br>
> > >> dst*)))<br>
> > >><br>
> > >> This kind of nested transaction scenario can appear<br>
> in<br>
> > >> the migrate process, because it uses source/destination store<br>
> > >> controller at the same time.<br>
> > >><br>
> > >> I do not understand why different store controller's<br>
> > >> transaction can result in the lock conflict and want to know<br>
> how to<br>
> > >> fix it.<br>
> > >><br>
> > >> Is there anyone meet the similar problem before?<br>
> > >><br>
> > >> Thanks<br>
> > >> Quan<br>
> > >><br>
> > >><br>
> > >><br>
> > >><br>
> > >><br>
> > >> _______________________________________________<br>
> > >> elephant-devel site list<br>
> > >> <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > >> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> > ><br>
> > ><br>
> > > _______________________________________________<br>
> > > elephant-devel site list<br>
> > > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> ><br>
> > _______________________________________________<br>
> > elephant-devel site list<br>
> > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > elephant-devel site list<br>
> > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > elephant-devel site list<br>
> > <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
<br>
<br>
_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
</div></div></blockquote></div><br>