<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
To my knowledge, you won't be able to do nested transactions.<BR>
<BR>
You can of course, write it to do a block of 100 at a time yourself.<BR>
That is, you program each transaction to do blocks of 100 by maging<BR>
the blocking yourself.<BR>
<BR>
This will probably give you all the speed-up that you require.<BR>
<BR>
It would of course be more convenient if you could write a giant <BR>
transaction, but that is difficult in any system.<BR>
<BR>
I am not absolutely sure, but I suspect based on the error message<BR>
that this is being passed up from BDB itself, or is very deep in the <BR>
sleepycat.lisp code.<BR>
<BR>
I would not expect there to be a code work-around that I would<BR>
be competent to suggest beyond the one above: Do one loop of 100<BR>
objects inside the transaction, and a loop of 130, or (+ 1 (mod n 100)),<BR>
as the outer loop.<BR>
<BR>
<BR>
On Wed, 2006-01-11 at 22:48 -0500, Waldo Rubinstein wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">I'm testing the with-transaction macro bulk loading custom classes </FONT>
<FONT COLOR="#000000">into elephant and encountered a little snag.</FONT>
<FONT COLOR="#000000">I'm initially loading about 300 objects in the database. The </FONT>
<FONT COLOR="#000000">performance using with-transaction as opposed to not using it </FONT>
<FONT COLOR="#000000">(meaning committing after each object is "written") is significant. </FONT>
<FONT COLOR="#000000">See below:</FONT>
<FONT COLOR="#000000">** not using with-transaction:</FONT>
<FONT COLOR="#000000">CL-USER> (time (oonet:load-834-tape "01053rdSupp.EDI"))</FONT>
<FONT COLOR="#000000">(COM.OONET.EDI:LOAD-834-TAPE "01053rdSupp.EDI") took 2,341 </FONT>
<FONT COLOR="#000000">milliseconds (2.341 seconds) to run.</FONT>
<FONT COLOR="#000000">Of that, 487 milliseconds (0.487 seconds) were spent in user mode</FONT>
<FONT COLOR="#000000"> 422 milliseconds (0.422 seconds) were spent in system mode</FONT>
<FONT COLOR="#000000"> 1,432 milliseconds (1.432 seconds) were spent executing </FONT>
<FONT COLOR="#000000">other OS processes.</FONT>
<FONT COLOR="#000000">309,800 bytes of memory allocated.</FONT>
<FONT COLOR="#000000">** using with-transaction:</FONT>
<FONT COLOR="#000000">CL-USER> (time (oonet:load-834-tape "01053rdSupp.EDI"))</FONT>
<FONT COLOR="#000000">(COM.OONET.EDI:LOAD-834-TAPE "01053rdSupp.EDI") took 624 milliseconds </FONT>
<FONT COLOR="#000000">(0.624 seconds) to run.</FONT>
<FONT COLOR="#000000">Of that, 270 milliseconds (0.270 seconds) were spent in user mode</FONT>
<FONT COLOR="#000000"> 19 milliseconds (0.019 seconds) were spent in system mode</FONT>
<FONT COLOR="#000000"> 335 milliseconds (0.335 seconds) were spent executing other </FONT>
<FONT COLOR="#000000">OS processes.</FONT>
<FONT COLOR="#000000">314,160 bytes of memory allocated.</FONT>
<FONT COLOR="#000000">It seems to be approximately 4 times faster using with-transaction, </FONT>
<FONT COLOR="#000000">which makes sense.</FONT>
<FONT COLOR="#000000">When I try loading a larger set of objects (approximately 13,000), </FONT>
<FONT COLOR="#000000">the program works as follows:</FONT>
<FONT COLOR="#000000">** not using with-transaction:</FONT>
<FONT COLOR="#000000">CL-USER> (time (oonet:load-834-tape "0106.EDI"))</FONT>
<FONT COLOR="#000000">(COM.OONET.EDI:LOAD-834-TAPE "0106.EDI") took 480,914 milliseconds </FONT>
<FONT COLOR="#000000">(480.914 seconds) to run.</FONT>
<FONT COLOR="#000000">Of that, 118,046 milliseconds (118.046 seconds) were spent in user mode</FONT>
<FONT COLOR="#000000"> 101,881 milliseconds (101.881 seconds) were spent in system </FONT>
<FONT COLOR="#000000">mode</FONT>
<FONT COLOR="#000000"> 260,987 milliseconds (260.987 seconds) were spent executing </FONT>
<FONT COLOR="#000000">other OS processes.</FONT>
<FONT COLOR="#000000">301 milliseconds (0.301 seconds) was spent in GC.</FONT>
<FONT COLOR="#000000">66,954,792 bytes of memory allocated.</FONT>
<FONT COLOR="#000000">** using with-transaction:</FONT>
<FONT COLOR="#000000">CL-USER> (time (oonet:load-834-tape "RDTLMCNA0106.EDI"))</FONT>
<FONT COLOR="#000000">Berkeley DB error: Cannot allocate memory</FONT>
<FONT COLOR="#000000"> [Condition of type SLEEPYCAT:DB-ERROR]</FONT>
<FONT COLOR="#000000">Restarts:</FONT>
<FONT COLOR="#000000"> 0: [ABORT-REQUEST] Abort handling SLIME request.</FONT>
<FONT COLOR="#000000"> 1: [ABORT-BREAK] Reset this process</FONT>
<FONT COLOR="#000000"> 2: [ABORT] Kill this process</FONT>
<FONT COLOR="#000000">Backtrace:</FONT>
<FONT COLOR="#000000"> 0: (#<STANDARD-METHOD SLOT-BOUNDP-USING-CLASS :AROUND </FONT>
<FONT COLOR="#000000">(ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT </FONT>
<FONT COLOR="#000000">ELEPHANT::PERSISTENT-SLOT-DEFINITION)> #<ELEPHANT:PERSISTENT- </FONT>
<FONT COLOR="#000000">METACLASS COM.OONET.EDI:834-ENTRY> #<PERSISTENT-EFFECTIVE-SLOT- </FONT>
<FONT COLOR="#000000">DEFINITION for database slot COM.OONET.EDI::EFFECTIVE-DATE-QUALIFIER </FONT>
<FONT COLOR="#000000">#x8BE961E> #<PERSISTENT-EFFECTIVE-SLOT-DEFINITION for database slot </FONT>
<FONT COLOR="#000000">COM.OONET.EDI::EFFECTIVE-DATE-QUALIFIER #x8BE961E>)</FONT>
<FONT COLOR="#000000"> 1: (#<STANDARD-METHOD SHARED-INITIALIZE :AROUND </FONT>
<FONT COLOR="#000000">(ELEPHANT:PERSISTENT-OBJECT T)> #<COM.OONET.EDI:834-ENTRY #x8CCB1A6> 'T)</FONT>
<FONT COLOR="#000000"> 2: (CCL::%%CNM-WITH-ARGS-COMBINED-METHOD-DCODE '(NIL </FONT>
<FONT COLOR="#000000">#<CCL::STANDARD-KERNEL-METHOD INITIALIZE-INSTANCE (STANDARD-OBJECT) </FONT>
<FONT COLOR="#000000"> > . 10540825) '((#<STANDARD-METHOD INITIALIZE-INSTANCE :BEFORE #>) </FONT>
<FONT COLOR="#000000">NIL #<CCL::STANDARD-KERNEL-METHOD INITIALIZE-INSTANCE (STANDARD- </FONT>
<FONT COLOR="#000000">OBJECT)>) 10540825)</FONT>
<FONT COLOR="#000000"> 3: (CCL::%%STANDARD-COMBINED-METHOD-DCODE 'NIL </FONT>
<FONT COLOR="#000000">#<ELEPHANT:PERSISTENT-METACLASS COM.OONET.EDI:834-ENTRY>)</FONT>
<FONT COLOR="#000000"> 4: (CCL::%MAKE-STD-INSTANCE #<COM.OONET.EDI:834-ENTRY #x8CCC40E> </FONT>
<FONT COLOR="#000000">'("INS" "Y" "18" "001" "25" "A" "" "" "FT"))</FONT>
<FONT COLOR="#000000"> 5: (#<CCL::STANDARD-KERNEL-METHOD MAKE-INSTANCE (STANDARD-CLASS)> </FONT>
<FONT COLOR="#000000">#<ELEPHANT:PERSISTENT-METACLASS COM.OONET.EDI:834-ENTRY>)</FONT>
<FONT COLOR="#000000"> 6: (CCL::%MAKE-INSTANCE '(COM.OONET.EDI:834-ENTRY . </FONT>
<FONT COLOR="#000000">#<ELEPHANT:PERSISTENT-METACLASS COM.OONET.EDI:834-ENTRY>))</FONT>
<FONT COLOR="#000000"> 7: (COM.OONET.EDI::PARSE-834-FILE #<FILE-CHARACTER-INPUT-STREAM ("/ </FONT>
<FONT COLOR="#000000">Users/dsalama/Documents/Projects/MCNA/DHACS/RDTLMCNA0106.EDI"/14) </FONT>
<FONT COLOR="#000000">#x8BF207E>)</FONT>
<FONT COLOR="#000000"> 8: (COM.OONET.EDI:LOAD-834-TAPE "RDTLMCNA0106.EDI")</FONT>
<FONT COLOR="#000000"> 9: (CCL::REPORT-TIME '(COM.OONET.EDI:LOAD-834-TAPE </FONT>
<FONT COLOR="#000000">"RDTLMCNA0106.EDI") #<Anonymous Function #x8BF20EE>)</FONT>
<FONT COLOR="#000000">10: (CCL::CALL-CHECK-REGS 'CCL::REPORT-TIME)</FONT>
<FONT COLOR="#000000">11: (SWANK::EVAL-REGION "(time (oonet:load-834-tape \"RDTLMCNA0106.EDI </FONT>
<FONT COLOR="#000000">\"))</FONT>
<FONT COLOR="#000000">" 'T)</FONT>
<FONT COLOR="#000000">12: (#<Anonymous Function #x85286E6> "(time (oonet:load-834-tape </FONT>
<FONT COLOR="#000000">\"RDTLMCNA0106.EDI\"))</FONT>
<FONT COLOR="#000000">")</FONT>
<FONT COLOR="#000000">13: (SWANK::CALL-WITH-BUFFER-SYNTAX #<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x8BF2716>)</FONT>
<FONT COLOR="#000000">14: (CCL::CALL-CHECK-REGS 'SWANK:LISTENER-EVAL)</FONT>
<FONT COLOR="#000000">15: (#<Anonymous Function #x8516CA6> '(SWANK:LISTENER-EVAL "(time </FONT>
<FONT COLOR="#000000">(oonet:load-834-tape \"RDTLMCNA0106.EDI\"))</FONT>
<FONT COLOR="#000000">") 47 "COMMON-LISP-USER")</FONT>
<FONT COLOR="#000000">16: (#<STANDARD-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)> </FONT>
<FONT COLOR="#000000">#<Compiled-function SWANK:SWANK-DEBUGGER-HOOK #x8527576> #<COMPILED- </FONT>
<FONT COLOR="#000000">LEXICAL-CLOSURE #x8BF2756>)</FONT>
<FONT COLOR="#000000">17: (FUNCALL 'SWANK::EVAL-FOR-EMACS)</FONT>
<FONT COLOR="#000000">18: (#<Anonymous Function #x8504C96>)</FONT>
<FONT COLOR="#000000">19: (#<Anonymous Function #x8508D56> #<Anonymous Function #x8504C96>)</FONT>
<FONT COLOR="#000000">20: (SWANK::CALL-WITH-REDIRECTED-IO #<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x8B9D596> #<CONNECTION #x899E8F6>)</FONT>
<FONT COLOR="#000000">21: (SWANK::CALL-WITH-CONNECTION #<CONNECTION #x899E8F6> #<Anonymous </FONT>
<FONT COLOR="#000000">Function #x8504C96>)</FONT>
<FONT COLOR="#000000">22: (SWANK::HANDLE-REQUEST #<CONNECTION #x899E8F6>)</FONT>
<FONT COLOR="#000000">23: (#<Anonymous Function #x85013B6> #<CONNECTION #x899E8F6>)</FONT>
<FONT COLOR="#000000">24: (#<Anonymous Function #x8508D56> #<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x89D9606>)</FONT>
<FONT COLOR="#000000">25: (SWANK::CALL-WITH-REDIRECTED-IO #<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x89D95EE> 'NIL)</FONT>
<FONT COLOR="#000000">26: (SWANK::CALL-WITH-CONNECTION #<CONNECTION #x899E8F6> #<COMPILED- </FONT>
<FONT COLOR="#000000">LEXICAL-CLOSURE #x89D9606>)</FONT>
<FONT COLOR="#000000">27: (SWANK::CALL-WITH-BINDINGS 'NIL #<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x89D961E>)</FONT>
<FONT COLOR="#000000">28: (CCL::RUN-PROCESS-INITIAL-FORM '(#<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x89D87AE>) #<PROCESS new-repl-thread(38) [Active] #x89D87E6>)</FONT>
<FONT COLOR="#000000">29: (#<Anonymous Function #x80DB61E> '(#<COMPILED-LEXICAL-CLOSURE </FONT>
<FONT COLOR="#000000">#x89D87AE>) 0)</FONT>
<FONT COLOR="#000000">30: (#<Anonymous Function #x80CE806> 790176 #<LISP-THREAD new-repl- </FONT>
<FONT COLOR="#000000">thread [tcr @ #x303A80] #x89D88D6>)</FONT>
<FONT COLOR="#000000">It seems it dies after processing about 430 objects.</FONT>
<FONT COLOR="#000000">Is there a fix for this? Is there a work around? How about, is there </FONT>
<FONT COLOR="#000000">a way that I could commit every 100 objects within a nested </FONT>
<FONT COLOR="#000000">transaction? Will it help?</FONT>
<FONT COLOR="#000000">Thanks,</FONT>
<FONT COLOR="#000000">Waldo</FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">elephant-devel site list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000"><A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>