<html><body bgcolor="#FFFFFF"><div>This scenario worked for migration as of the last release so a bug must have<br>crept in with the new features. Thank you for the investigation, it only works if you are very careful. There should be some info on this in the txn src code or in the manual - I'll look into this when I can if you don't identify it first. Is it related to the specific migrate code for set-valued slots?</div><div><br>Sent from my iPhone</div><div><br>On Dec 30, 2008, at 9:34 AM, "quan hu" <<a href="mailto:ihuquan@gmail.com">ihuquan@gmail.com</a>> wrote:<br><br></div><div></div><blockquote type="cite"><div>Hello,<br><br> After more investigation, here are some suspects:<br><br> In the current elephant codes, the macro my-current-transaction will check if the current transaction belongs to the sc parameter. If not, just return +NULL-CHAR+. That's exactly the case in the following nested transaction where the store-controller is different. <br>
The situation becomes:<br> Transaction 1 begins<br> One data is put into DB with Transaction 1 <br> ... <br> Another data is put into the same DB with NULL transaction parameter(i.e. no transaction??) <br>
<br> It looks like this scenario is not supported by BDB. <br> I simulate above case with a C program to access BDB and get the similar failure symptom.<br>...<br> txn1 = NULL;<br> ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);<br>
if (ret != 0) {<br> printf ("Transaction begin failed.\n");<br> return 1;<br> }<br> put_record(FOO,FOO,txn1, dbp1);<br> printf("foo\n");<br> put_record(FOO,FOO,NULL, dbp1);<br> printf("bar\n");<br>
txn1->commit(txn1,0);<br> printf("baz\n");<br>...<br> After "foo" is printed, the C program is blocked. If I change the NULL to txn1, everything is fine.<br><br><br> Regarding to the solution, my gut feeling is that it would be difficult to avoid this kind of nested transaction in migrate process, because of the recursive function call. Maybe, *transaction-stack* can be added to track the active transaction stack. Get it pushed and poped in right place of execute-transaction. Then, in my-current-transaction, check *transaction-stack* to see if there is an active transaction belongs to the same store-controller. If it is found, use it. My simple test shows it works. If this is the right direction, I can do more test.<br>
<br>Thanks for your help.<br>Quan<br><br><br><br> <br><br><div class="gmail_quote">2008/12/30 quan hu <span dir="ltr"><<a href="mailto:ihuquan@gmail.com"><a href="mailto:ihuquan@gmail.com">ihuquan@gmail.com</a></a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Ian, Yarek,<br><br> Thanks for providing the information.<br> I downloaded the elephant-unstable and test it with BDB4.7.<br> The failure symptom is the same: target store controller get locked.<br><br> After I make following changes in migrate.lisp; The test passed.<br>
<br> (defmethod copy-btree-contents ((sc store-controller) dst src)<br> (let (to-be-migrated)<br> (map-btree (lambda (key value)<br> (let ((newval (migrate sc value))<br>
(newkey (migrate sc key)))<br> (push (list newkey newval) to-be-migrated)))<br> ;;;; (setf (get-value newkey dst) newval))) ;; <br>
src)<br> (loop for (k v) in to-be-migrated<br> do<br> (setf (get-value k dst) v))))<br><br> Above debug change just avoids the nested transaction of different store controller for the specific test case.<br>
However, it is not the solution, as the nested transaction still could appear in other data pattern.<br><br> What confuses me is that the nested transaction of the same store controller seem not bring any block.<br>
Following test passed:<br> (with-transaction (:store-controller *migrate-dst*)<div class="Ih2E3d"><br> (add-to-root 'foo 'foo :sc *migrate-dst*)<br></div> (with-transaction (:store-controller *migrate-dst*)<div class="Ih2E3d">
<br>
(add-to-root 'bar 'bar :sc *migrate-dst*)))<br><br></div> But, the nested transaction of different store controller always results in lock.<br><br> (with-transaction (:store-controller *migrate-dst*)<div class="Ih2E3d">
<br>
(add-to-root 'foo 'foo :sc *migrate-dst*)<br></div> (with-transaction (:store-controller *<b>migrate-src</b>*) ;; trigger database lock <br><div class="Ih2E3d"> (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
<br></div>Thanks<br>Quan<br><br><br><br><div class="gmail_quote">2008/12/30 Yarek Kowalik <span dir="ltr"><<a href="mailto:yarek.kowalik@gmail.com" target="_blank"><a href="mailto:yarek.kowalik@gmail.com">yarek.kowalik@gmail.com</a></a>></span><div><div></div>
<div class="Wj3C7c"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Quan, <br><br>Unstable is here:<br><br> <a href="http://www.common-lisp.net/project/elephant/darcs/elephant-unstable" target="_blank"><a href="http://www.common-lisp.net/project/elephant/darcs/elephant-unstable">http://www.common-lisp.net/project/elephant/darcs/elephant-unstable</a></a><br><font color="#888888"><br>
Yarek</font><div><div></div><div><br><br><div class="gmail_quote">
On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <span dir="ltr"><<a href="mailto:eslick@media.mit.edu" target="_blank"><a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a></a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
That would be elephant-unstable<br>
<br>
Sent from my iPhone<br>
<div><div></div><div><br>
On Dec 29, 2008, at 7:36 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu" target="_blank"><a href="mailto:eslick@media.mit.edu">eslick@media.mit.edu</a></a>> wrote:<br>
<br>
> Hi Quan,<br>
><br>
> Can you try the latest darcs with BDB 4.6 or 4.7? I made some changes<br>
> to migration, but I can't recall if they were in the latest darcs or<br>
> not - I think the latest darcs doesn't play well with BDB 4.5 so<br>
> please make sure you can reproduce under the above configuration if<br>
> you can.<br>
><br>
> Thank you,<br>
> Ian<br>
><br>
> On Dec 29, 2008, at 3:40 AM, quan hu wrote:<br>
><br>
>> Hello,<br>
>><br>
>> I run into a problem when doing the garbage collection via data<br>
>> migration.<br>
>> The environment is elephant 0.9.1 and BDB 4.5. I also tried<br>
>> the latest elephant in darc and get the same result.<br>
>><br>
>> 1. Test case to reproduce the problem.<br>
>><br>
>> (defpclass user-profile()<br>
>> ((id :initform nil :index t)<br>
>> (sms-inbox :initform (make-pset))))<br>
>><br>
>> ;; Create a new db for test purpose<br>
>> (setf *migrate-src* (open-store '(:bdb "/tmp/db/src/")))<br>
>> (setf *test-obj* (make-instance 'user-profile :sc<br>
>> *migrate-src*))<br>
>> (insert-item 'foo (slot-value *test-obj* 'sms-inbox))<br>
>> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/dst/")))<br>
>><br>
>> TEST> (migrate *migrate-dst* *migrate-src*)<br>
>> Migrating class indexes for: USER-PROFILE<br>
>> ; => Input get blocked, not response anymore<br>
>><br>
>> 2. Using "db_stat -C A" get following output:<br>
>> 1 Lock requests not available due to conflicts, for<br>
>> which we waited<br>
>><br>
>> 3. I do some debug work and found the problem may be caused<br>
>> by nested transaction of different store controller.<br>
>> I can reproduce the issue with following code segment<br>
>><br>
>> TEST> (ensure-transaction (:store-controller *migrate-dst*)<br>
>> (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
>> (ensure-transaction (:store-controller<br>
>> *migrate-src*)<br>
>> (add-to-root 'bar 'bar :sc *migrate-<br>
>> dst*)))<br>
>><br>
>> This kind of nested transaction scenario can appear in<br>
>> the migrate process, because it uses source/destination store<br>
>> controller at the same time.<br>
>><br>
>> I do not understand why different store controller's<br>
>> transaction can result in the lock conflict and want to know how to<br>
>> fix it.<br>
>><br>
>> Is there anyone meet the similar problem before?<br>
>><br>
>> Thanks<br>
>> Quan<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> elephant-devel site list<br>
>> <a href="mailto:elephant-devel@common-lisp.net" target="_blank"><a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a></a><br>
>> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank"><a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a></a><br>
><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net" target="_blank"><a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a></a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank"><a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a></a><br>
<br>
_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net" target="_blank"><a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a></a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank"><a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a></a><br>
</div></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net" target="_blank"><a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a></a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank"><a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a></a><br></blockquote></div></div></div><br>
</blockquote></div><br>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>elephant-devel site list</span><br><span><a href="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</a></span><br><span><a href="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</a></span></div></blockquote></body></html>