[rucksack-devel] PATCH: without-rucksack-gcing

[Cyrus Harmon]

On Jan 12, 2007, at 12:09 PM, Arthur Lemmens wrote:

> Cyrus Harmon wrote:
>
>> The following patch defines a macro without-rucksack-gcing that can
>> be used to cause the enclosed form to be executed without calling
>> collect-some-garbage upon commit.
>
> Thanks.  I try to keep special variables to a minimum, so I'll  
> probably
> move this flag to a slot in TRANSACTION that gets initialized by
> WITH-TRANSACTION.

Yes, that sounds like a reasonable approach. I thought about making  
this an argument to with-transaction like:

(with-transaction (:gc nil)
  ...)

>> Using this patch allows me to load 400k objects into rucksack in a
>> reasonable amount of time
>
> That's good news.  With or without indexes?  And can you be more  
> specific
> about 'reasonable amount of time'?

with indices. ah, yes, reasonable. I suppose that's somewhat  
subjective. The "unreasonable" efforts before were characterized by  
progressive slowdowns, such that things would start fast (say ~50-100  
obj/second) and after 50-100k objects or so, slow down to 1/obj  
second. I left for breakfast and didn't time the last big insert, but  
the one I've been working on now has been going for about 15 minutes  
and has done over 150k objects. 10k obj/min is fine for my purposes.  
100 obj/min wouldn't be.

One thing to note is that eventually you do have to pay the GC cost,  
and on the first transaction with GC after my big insert, there was a  
multiple minute delay while it GCed the rucksack. I can live with  
that and it's certainly better than not working at all!

Cyrus