[rucksack-devel] continuing the rucksack saga...

[Arthur Lemmens]

Cyrus Harmon wrote:

> I'm not sure whether to lay the blame at rucksacks' feet or at SBCLs,
> but now that i've resolved my performance problems to the point where
> I can make-instance over 150k objects, I run out of heap space:

OK, it sounds like you're now reaching the interesting part, where all
your data doesn't fit in memory anymore and you really need Rucksack's
cache.

> and this happens when trying to allocate all .5M objects in a single
> transaction, also, perhaps, a bad idea.

Yes, that's a bad idea because in the current design Rucksack has to keep
all those objects in memory until the transaction has committed.

> I think I got basically the same error when trying 1 obj/transaction

If you do, I'd say that's a bug (or at least a misunderstanding) somewhere.
But it could be anywhere: in Rucksack's cache or transactions, in SBCL's
garbage collector, or in your own objects.  If it's possible to somehow
reach all your other objects from something that's 'active' in memory,
then SBCL's garbage collector can't collect any garbage.

Here's what I would do: first I'd have a serious look at your objects
(and data structures), then I'd switch to something like 1000 objects
per transaction, and if that doesn't solve anything I'd want to see if
Rucksack's cache or transactions don't keep in-memory object references
longer than it should.

Arthur