Hello,<br><br> After more investigation, here are some suspects:<br><br> In the current elephant codes, the macro my-current-transaction will check if the current transaction belongs to the sc parameter. If not, just return +NULL-CHAR+. That's exactly the case in the following nested transaction where the store-controller is different. <br>
The situation becomes:<br> Transaction 1 begins<br> One data is put into DB with Transaction 1 <br> ... <br> Another data is put into the same DB with NULL transaction parameter(i.e. no transaction??) <br>
<br> It looks like this scenario is not supported by BDB. <br> I simulate above case with a C program to access BDB and get the similar failure symptom.<br>...<br> txn1 = NULL;<br> ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);<br>
if (ret != 0) {<br> printf ("Transaction begin failed.\n");<br> return 1;<br> }<br> put_record(FOO,FOO,txn1, dbp1);<br> printf("foo\n");<br> put_record(FOO,FOO,NULL, dbp1);<br> printf("bar\n");<br>
txn1->commit(txn1,0);<br> printf("baz\n");<br>...<br> After "foo" is printed, the C program is blocked. If I change the NULL to txn1, everything is fine.<br><br><br> Regarding to the solution, my gut feeling is that it would be difficult to avoid this kind of nested transaction in migrate process, because of the recursive function call. Maybe, *transaction-stack* can be added to track the active transaction stack. Get it pushed and poped in right place of execute-transaction. Then, in my-current-transaction, check *transaction-stack* to see if there is an active transaction belongs to the same store-controller. If it is found, use it. My simple test shows it works. If this is the right direction, I can do more test.<br>
<br>Thanks for your help.<br>Quan<br><br><br><br> <br><br><div class="gmail_quote">2008/12/30 quan hu <span dir="ltr"><<a href="mailto:ihuquan@gmail.com">ihuquan@gmail.com</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Hi Ian, Yarek,<br><br> Thanks for providing the information.<br> I downloaded the elephant-unstable and test it with BDB4.7.<br> The failure symptom is the same: target store controller get locked.<br><br> After I make following changes in migrate.lisp; The test passed.<br>
<br> (defmethod copy-btree-contents ((sc store-controller) dst src)<br> (let (to-be-migrated)<br> (map-btree (lambda (key value)<br> (let ((newval (migrate sc value))<br>
(newkey (migrate sc key)))<br> (push (list newkey newval) to-be-migrated)))<br> ;;;; (setf (get-value newkey dst) newval))) ;; <br>
src)<br> (loop for (k v) in to-be-migrated<br> do<br> (setf (get-value k dst) v))))<br><br> Above debug change just avoids the nested transaction of different store controller for the specific test case.<br>
However, it is not the solution, as the nested transaction still could appear in other data pattern.<br><br> What confuses me is that the nested transaction of the same store controller seem not bring any block.<br>
Following test passed:<br> (with-transaction (:store-controller *migrate-dst*)<div class="Ih2E3d"><br> (add-to-root 'foo 'foo :sc *migrate-dst*)<br></div> (with-transaction (:store-controller *migrate-dst*)<div class="Ih2E3d">
<br>
(add-to-root 'bar 'bar :sc *migrate-dst*)))<br><br></div> But, the nested transaction of different store controller always results in lock.<br><br> (with-transaction (:store-controller *migrate-dst*)<div class="Ih2E3d">
<br>
(add-to-root 'foo 'foo :sc *migrate-dst*)<br></div> (with-transaction (:store-controller *<b>migrate-src</b>*) ;; trigger database lock <br><div class="Ih2E3d"> (add-to-root 'bar 'bar :sc *migrate-dst*)))<br>
<br></div>Thanks<br>Quan<br><br><br><br><div class="gmail_quote">2008/12/30 Yarek Kowalik <span dir="ltr"><<a href="mailto:yarek.kowalik@gmail.com" target="_blank">yarek.kowalik@gmail.com</a>></span><div><div></div>
<div class="Wj3C7c"><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Quan, <br><br>Unstable is here:<br><br> <a href="http://www.common-lisp.net/project/elephant/darcs/elephant-unstable" target="_blank">http://www.common-lisp.net/project/elephant/darcs/elephant-unstable</a><br><font color="#888888"><br>
Yarek</font><div><div></div><div><br><br><div class="gmail_quote">
On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <span dir="ltr"><<a href="mailto:eslick@media.mit.edu" target="_blank">eslick@media.mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
That would be elephant-unstable<br>
<br>
Sent from my iPhone<br>
<div><div></div><div><br>
On Dec 29, 2008, at 7:36 AM, Ian Eslick <<a href="mailto:eslick@media.mit.edu" target="_blank">eslick@media.mit.edu</a>> wrote:<br>
<br>
> Hi Quan,<br>
><br>
> Can you try the latest darcs with BDB 4.6 or 4.7? I made some changes<br>
> to migration, but I can't recall if they were in the latest darcs or<br>
> not - I think the latest darcs doesn't play well with BDB 4.5 so<br>
> please make sure you can reproduce under the above configuration if<br>
> you can.<br>
><br>
> Thank you,<br>
> Ian<br>
><br>
> On Dec 29, 2008, at 3:40 AM, quan hu wrote:<br>
><br>
>> Hello,<br>
>><br>
>> I run into a problem when doing the garbage collection via data<br>
>> migration.<br>
>> The environment is elephant 0.9.1 and BDB 4.5. I also tried<br>
>> the latest elephant in darc and get the same result.<br>
>><br>
>> 1. Test case to reproduce the problem.<br>
>><br>
>> (defpclass user-profile()<br>
>> ((id :initform nil :index t)<br>
>> (sms-inbox :initform (make-pset))))<br>
>><br>
>> ;; Create a new db for test purpose<br>
>> (setf *migrate-src* (open-store '(:bdb "/tmp/db/src/")))<br>
>> (setf *test-obj* (make-instance 'user-profile :sc<br>
>> *migrate-src*))<br>
>> (insert-item 'foo (slot-value *test-obj* 'sms-inbox))<br>
>> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/dst/")))<br>
>><br>
>> TEST> (migrate *migrate-dst* *migrate-src*)<br>
>> Migrating class indexes for: USER-PROFILE<br>
>> ; => Input get blocked, not response anymore<br>
>><br>
>> 2. Using "db_stat -C A" get following output:<br>
>> 1 Lock requests not available due to conflicts, for<br>
>> which we waited<br>
>><br>
>> 3. I do some debug work and found the problem may be caused<br>
>> by nested transaction of different store controller.<br>
>> I can reproduce the issue with following code segment<br>
>><br>
>> TEST> (ensure-transaction (:store-controller *migrate-dst*)<br>
>> (add-to-root 'foo 'foo :sc *migrate-dst*)<br>
>> (ensure-transaction (:store-controller<br>
>> *migrate-src*)<br>
>> (add-to-root 'bar 'bar :sc *migrate-<br>
>> dst*)))<br>
>><br>
>> This kind of nested transaction scenario can appear in<br>
>> the migrate process, because it uses source/destination store<br>
>> controller at the same time.<br>
>><br>
>> I do not understand why different store controller's<br>
>> transaction can result in the lock conflict and want to know how to<br>
>> fix it.<br>
>><br>
>> Is there anyone meet the similar problem before?<br>
>><br>
>> Thanks<br>
>> Quan<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> elephant-devel site list<br>
>> <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
>> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
><br>
><br>
> _______________________________________________<br>
> elephant-devel site list<br>
> <a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
> <a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
<br>
_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br>
</div></div></blockquote></div><br>
</div></div><br>_______________________________________________<br>
elephant-devel site list<br>
<a href="mailto:elephant-devel@common-lisp.net" target="_blank">elephant-devel@common-lisp.net</a><br>
<a href="http://common-lisp.net/mailman/listinfo/elephant-devel" target="_blank">http://common-lisp.net/mailman/listinfo/elephant-devel</a><br></blockquote></div></div></div><br>
</blockquote></div><br>