[rucksack-devel] continuing the rucksack saga...

[Cyrus Harmon]

On Jan 11, 2007, at 11:13 PM, Arthur Lemmens wrote:

> Cyrus Harmon wrote:
>
>> I'm not sure whether to lay the blame at rucksacks' feet or at SBCLs,
>> but now that i've resolved my performance problems to the point where
>> I can make-instance over 150k objects, I run out of heap space:
>
> OK, it sounds like you're now reaching the interesting part, where all
> your data doesn't fit in memory anymore and you really need Rucksack's
> cache.

Well, I'm not convinced I'm really at that point as these objects are  
fairly small, but the combination of the memory for the objects and  
whatever other junks gets consed along the way doesn't seem to fit in  
memory.

>> and this happens when trying to allocate all .5M objects in a single
>> transaction, also, perhaps, a bad idea.
>
> Yes, that's a bad idea because in the current design Rucksack has  
> to keep
> all those objects in memory until the transaction has committed.

yes, i figured as much. again, this was a workaround for the 1 obj/ 
transaction problem.

>> I think I got basically the same error when trying 1 obj/transaction
>
> If you do, I'd say that's a bug (or at least a misunderstanding)  
> somewhere.
> But it could be anywhere: in Rucksack's cache or transactions, in  
> SBCL's
> garbage collector, or in your own objects.  If it's possible to  
> somehow
> reach all your other objects from something that's 'active' in memory,
> then SBCL's garbage collector can't collect any garbage.
>
> Here's what I would do: first I'd have a serious look at your objects
> (and data structures), then I'd switch to something like 1000 objects
> per transaction, and if that doesn't solve anything I'd want to see if
> Rucksack's cache or transactions don't keep in-memory object  
> references
> longer than it should.

what's the best way to do this? can I just drop a  
(rucksack:transaction-commit rucksack:*transaction*) into my loop and  
have the right thing happen or do I need to break up the loop to  
explicitly do a (with-transaction ... for batches of 1000 or so?

Thanks,

Cyrus