[rucksack-devel] many small xacts vs. one large xact

[Cyrus Harmon]

Ok, so I'm trying to create about .5M persistent objects. If I use a  
single transaction, performance starts to degrade after 50k objects  
or so. If I use a transaction per object, things are ok in the  
beginning, but after 1400 objects or so, I get the following:

There is no applicable method for the generic function
   #<STANDARD-GENERIC-FUNCTION RUCKSACK:P-CAR (1)>
when called with arguments
   (#<RUCKSACK:PERSISTENT-ARRAY #2653 in #<STANDARD-CACHE of size  
10000, heap #P"/Users/sly/projects/cyrusharmon.org/cl-bio/rucksack/ 
heap" and 4634 objects in memory.>>).
    [Condition of type SIMPLE-ERROR]

Restarts:
   0: [ABORT] Abort #<RUCKSACK:STANDARD-TRANSACTION #337747670400025  
with 4 dirty objects>
   1: [RETRY] Retry #<RUCKSACK:STANDARD-TRANSACTION #337747670400025  
with 4 dirty objects>
   2: [ABORT-REQUEST] Abort handling SLIME request.
   3: [ABORT] Exit debugger, returning to top level.

Backtrace:
   0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<unused  
argument> #<unused argument> #<STANDARD-GENERIC-FUNCTION RUCKSACK:P- 
CAR (1)> #<RUCKSACK:PERSISTENT-ARRAY #2653 in #<STANDARD-CACHE of  
size 10000, heap #P"/Users/sly/projects/cyrusharmon.org/cl-bio/ 
rucksack/heap" and 4634 objects in memory.>>)
   1: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) #<unavailable  
argument> #<unavailable argument> #<STANDARD-GENERIC-FUNCTION  
RUCKSACK:P-CAR (1)>)
   2: (RUCKSACK::FIND-BINDING-IN-NODE 1413 #<RUCKSACK:BTREE-NODE with  
64 bindings {117D5AF9}> #<RUCKSACK:BTREE #23 in #<STANDARD-CACHE of  
size 10000, heap #P"/Users/sly/projects/cyrusharmon.org/cl-bio/ 
rucksack/heap" and 4634 objects in memory.>>)
   3: (RUCKSACK::LEAF-INSERT #<RUCKSACK:BTREE #23 in #<STANDARD-CACHE  
of size 10000, heap #P"/Users/sly/projects/cyrusharmon.org/cl-bio/ 
rucksack/heap" and 4634 objects in memory.>> #<unavailable argument>  
(#<RUCKSACK:BTREE-NODE with 23 bindings {14494DF1}> NIL) 1413  
2646 :OVERWRITE)
   4: ((SB-PCL::FAST-METHOD RUCKSACK:BTREE-INSERT (RUCKSACK:BTREE  
#1="#<...>" . #1#)) (#(NIL) . #()) #<unused argument>  
#<RUCKSACK:BTREE #23 in #<STANDARD-CACHE of size 10000, heap #P"/ 
Users/sly/projects/cyrusharmon.org/cl-bio/rucksack/heap" and 4634  
objects in memory.>> 1413 2646 :IF-EXISTS :OVERWRITE)
   5: ((SB-PCL::FAST-METHOD INITIALIZE-INSTANCE :AROUND  
(#1="#<...>" . #1#)) (#(NIL NIL) . #()) #S(SB-PCL::FAST-METHOD- 
CALL :FUNCTION #<FUNCTION #> :PV-CELL NIL :NEXT-METHOD-CALL NIL :ARG- 
INFO (1 . T)) #<P-TAXON #2646 in #<STANDARD-CACHE of size 10000, heap  
#P"/Users/sly/projects/cyrusharmon.org/cl-bio/rucksack/heap" and 4634  
objects in memory.>> :TAX-ID 1413 :PARENT-ID 1386 :RANK  
"species" :EMBL-CODE "BC" :DIVISION-ID 0 :DIVISION-INHERITED  
T :GENETIC-CODE-ID 11 :GENETIC-CODE-INHERITED T :MITOCHONDRIAL- 
GENETIC-CODE-ID 0 :MITOCHONDRIAL-GENETIC-CODE-INHERITED T :GENBANK- 
HIDDEN T :HIDDEN-SUBTREE NIL :COMMENTS "" :RUCKSACK  
#<RUCKSACK::SERIAL-TRANSACTION-RUCKSACK in #P"/Users/sly/projects/ 
cyrusharmon.org/cl-bio/rucksack/" with 0 roots {11762239}>)
   6: ((SB-PCL::FAST-METHOD MAKE-INSTANCE (CLASS)) #<unused argument>  
#<unused argument> #<RUCKSACK:PERSISTENT-CLASS P-TAXON> #<unavailable  
&REST argument>)
   7: (PARSE-TAX-NODES :FILE NIL)
   8: (SB-INT:SIMPLE-EVAL-IN-LEXENV (PARSE-TAX-NODES) #<NULL-LEXENV>)
   9: (SWANK::EVAL-REGION
       "(parse-tax-nodes)
      "
       T)
10: ((LAMBDA ()))
11: ((LAMBDA (SWANK-BACKEND::FN)) #<CLOSURE (LAMBDA #) {1175E165}>)
12: (SWANK::CALL-WITH-BUFFER-SYNTAX #<CLOSURE (LAMBDA #) {1175E165}>)
13: (SWANK:LISTENER-EVAL
       "(parse-tax-nodes)
      ")
14: (SB-INT:SIMPLE-EVAL-IN-LEXENV
       (SWANK:LISTENER-EVAL "(parse-tax-nodes)
      ")
       #<NULL-LEXENV>)

has anyone done any stress testing to see how rucksack works with  
lots of objects and/or have any advice on storing large numbers of  
objects?

Thanks,

Cyrus