Hi Ian,<br><br> I downloaded the patches and have more testing today. The lock issue is gone!<br> Thanks a lot.<br><br> 1. But, another issue is found. migrate pathname fails. I test it under SBCL 1.0.21<br> e.g. <br>
> (migrate *store-controller* #p"/tmp) => Error: Cannot migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-ENOUGH>)<br><br> It is caused by that (defmethod migrate ((dst store-controller) (src structure-object) is called for this case, as pathname is taken as subtype of structure-object.<br>
> (subtypep 'pathname 'structure-object) => t t<br> Maybe, one more migrate method is needed for pathname case.<br> <br> 2. I also want to know if upgrading database file from 0.9.1 to unstable release is supported.<br>
When opening the DB file generated in 0.9.1 in unstable release, I get error message <br> " The slot DB-BDB::INDICES is unbound in the object #<BDB-INDEXED-BTREE oid:-3>.<br> [Condition of type UNBOUND-SLOT]" <br>
<br>Thanks<br>Quan<br><br><br><br><br><div class="gmail_quote">2008/12/31 Ian Eslick <span dir="ltr"><<a href="mailto:eslick@media.mit.edu" target="_blank">eslick@media.mit.edu</a>></span><br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hi Quan,<br>
<br>
Thank you again for the information on this bug. The short story is<br>
that the way we are keeping track of transactions assumed that there<br>
were only two stores in the nested transaction stack (outer and<br>
inner). Migrating a nested object inside a with-transaction would not<br>
pass the parent transaction to the backend so it wouldn't know a<br>
transaction in progress and attempted to start a new one, which would<br>
block until the old one was completed.<br>
<br>
I just checked in a patch that made transaction tracking a bit more<br>
intelligent (it depends on the backends taking care of handling nested<br>
transactions - which I believe they all do fine). I also checked in a<br>
test which implements Quan's test case.<br>
<br>
This patch and the prior patches of the past few days pass all tests<br>
on SBCL/Mac/BDB.<br>
<font color="#888888"><br>
Ian<br>
</font><br>
PS - I'd love to know the current status of the elephant-unstable tree<br>
with the postmodern backend.<br>
<div><div></div><div><br>
<br>
On Dec 30, 2008, at 9:34 AM, quan hu wrote:<br>
<br>
> Hello,<br>
><br>
> After more investigation, here are some suspects:<br>
><br>
> In the current elephant codes, the macro my-current-<br>
> transaction will check if the current transaction belongs to the sc<br>
> parameter. If not, just return +NULL-CHAR+. That's exactly the<br>
> case in the following nested transaction where the store-controller<br>
> is different.<br>
> The situation becomes:<br>
> Transaction 1 begins<br>
> One data is put into DB with Transaction 1<br>
> ...<br>
> Another data is put into the same DB with NULL<br>
> transaction parameter(i.e. no transaction??)<br>
><br>
> It looks like this scenario is not supported by BDB.<br>
> I simulate above case with a C program to access BDB and get<br>
> the similar failure symptom.<br>
> ...<br>
> txn1 = NULL;<br>
> ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);<br>
> if (ret != 0) {<br>
> printf ("Transaction begin failed.\n");<br>
> return 1;<br>
> }<br>
> put_record(FOO,FOO,txn1, dbp1);<br>
> printf("foo\n");<br>
> put_record(FOO,FOO,NULL, dbp1);<br>
> printf("bar\n");<br>
> txn1->commit(txn1,0);<br>
> printf("baz\n");<br>
> ...<br>
> After "foo" is printed, the C program is blocked. If I change the<br>
> NULL to txn1, everything is fine.<br>
><br>
><br>
> Regarding to the solution, my gut feeling is that it would be<br>
> difficult to avoid this kind of nested transaction in migrate<br>
> process, because of the recursive function call. Maybe,<br>
> *transaction-stack* can be added to track the active transaction<br>
> stack. Get it pushed and poped in right place of execute-<br>
> transaction. Then, in my-current-transaction, check *transaction-<br>
> stack* to see if there is an active transaction belongs to the same<br>
> store-controller. If it is found, use it. My simple test shows it<br>
> works. If this is the right direction, I can do more test.<br>
><br>
> Thanks for your help.<br>
> Quan<br>
><br>
><br>
><br>
><br>
><br>
> 2008/12/30 quan hu <<a href="mailto:ihuquan@gmail.com" target="_blank">ihuquan@gmail.com</a>><br>
> Hi Ian, Yarek,<br>
><br>
> Thanks for providing the information.<br>
> I downloaded the elephant-unstable and test it with BDB4.7.<br>
> The failure symptom is the same: target store controller get<br>
> locked.<br>
><br>
> After I make following changes in migrate.lisp; The test passed.<br>
><br>
> (defmethod copy-btree-contents ((sc store-controller) dst src)<br>
> (let (to-be-migrated)<br>
> (map-btree (lambda (key value)<br>
> (let ((newval (migrate sc value))<br>
> (newkey (migrate sc key)))<br>
> (push (list newkey<br>
> newval) to-be-migrated)))<br>
> ;;;; (setf (get-value<br>
> newkey dst) newval))) ;;<br>
> src)<br>
> (loop for (k v) in to-be-migrated<br>
> do<br>
> (setf (get-value k dst) v))))<br>
><br>
> Above debug change just avoids the nested transaction of<br>
> different store controller for the specific test case.<br>
> However, it is not the solution, as the nested transaction<br>
> still could appear in other data pattern.<br>
><br>
> What confuses me is that the nested transaction of the same<br>
> store controller seem not bring any block.<br>
> Following test passed:<br>
> (with-transaction (:store-controller *migrate-dst*)<br>
><br>
> (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
> (with-transaction (:store-controller *migrate-dst*)<br>
><br>
> (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
><br>
> But, the nested transaction of different store controller always<br>
> results in lock.<br>
><br>
> (with-transaction (:store-controller *migrate-dst*)<br>
><br>
> (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
> (with-transaction (:store-controller *migrate-<br>
> src*) ;; trigger database lock<br>
> (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
><br>
> Thanks<br>
> Quan<br>
><br>
><br>
><br>
> 2008/12/30 Yarek Kowalik <<a href="mailto:yarek.kowalik@gmail.com" target="_blank">yarek.kowalik@gmail.com</a>><br>
><br>
> Quan,<br>
><br>
> Unstable is here:<br>
><br>
> <a href="http://www.common-lisp.net/project/elephant/darcs/elephant-unstable" target="_blank">http://www.common-lisp.net/project/elephant/darcs/elephant-unstable</a><br>
><br>
> Yarek<br>
><br>
><br>
> On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu" target="_blank">eslick@media.mit.edu</a>><br>
> wrote:<br>
> That would be elephant-unstable<br>
><br>
> Sent from my iPhone<br>
><br>
> On Dec 29, 2008, at 7:36 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu" target="_blank">eslick@media.mit.edu</a>> wrote:<br>
><br>
> > Hi Quan,<br>
> ><br>
> > Can you try the latest darcs with BDB 4.6 or 4.7? I made some<br>
> changes<br>
> > to migration, but I can't recall if they were in the latest darcs or<br>
> > not - I think the latest darcs doesn't play well with BDB 4.5 so<br>
> > please make sure you can reproduce under the above configuration if<br>
> > you can.<br>
> ><br>
> > Thank you,<br>
> > Ian<br>
> ><br>
> > On Dec 29, 2008, at 3:40 AM, quan hu wrote:<br>
> ><br>
> >> Hello,<br>
> >><br>
> >> I run into a problem when doing the garbage collection via data<br>
> >> migration.<br>
> >> The environment is elephant 0.9.1 and BDB 4.5. I also tried<br>
> >> the latest elephant in darc and get the same result.<br>
> >><br>
> >> 1. Test case to reproduce the problem.<br>
> >><br>
> >> (defpclass user-profile()<br>
> >> ((id :initform nil :index t)<br>
> >> (sms-inbox :initform (make-pset))))<br>
> >><br>
> >> ;; Create a new db for test purpose<br>
> >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/src/")))<br>
> >> (setf *test-obj* (make-instance 'user-profile :sc<br>
> >> *migrate-src*))<br>
> >> (insert-item 'foo (slot-value *test-obj* 'sms-inbox))<br>
> >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/dst/")))<br>
> >><br>
> >> TEST> (migrate *migrate-dst* *migrate-src*)<br>
> >> Migrating class indexes for: USER-PROFILE<br>
> >> ; => Input get blocked, not response anymore<br>
> >><br>
> >> 2. Using "db_stat -C A" get following output:<br>
> >> 1 Lock requests not available due to conflicts, for<br>
> >> which we waited<br>
> >><br>
> >> 3. I do some debug work and found the problem may be caused<br>
> >> by nested transaction of different store controller.<br>
> >> I can reproduce the issue with following code segment<br>
> >><br>
> >> TEST> (ensure-transaction (:store-controller *migrate-<br>
> dst*)<br>
> >> (add-to-root 'foo 'foo :sc *migrate-<br>
> dst*)<br>
> >> (ensure-transaction (:store-controller<br>
> >> *migrate-src*)<br>
> >> (add-to-root 'bar 'bar :sc<br>
> *migrate-<br>
> >> dst*)))<br>
> >><br>
> >> This kind of nested transaction scenario can appear in<br>
> >> the migrate process, because it uses source/destination store<br>
> >> controller at the same time.<br>
> >><br>
> >> I do not understand why different store controller's<br>
> >> transaction can result in the lock conflict and want to know how to<br>
> >> fix it.<br>
> >><br>
> >> Is there anyone meet the similar problem before?<br>
> >><br>
> >> Thanks<br>
> >> Quan<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> _______________________________________________<br>
> >> elephant-devel site list<br>
> >> <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
> >> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > elephant-devel site list<br>
> > <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
> > <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
<br>
<br>
_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
</div></div></blockquote></div><br>