[rucksack-devel] rucksack performance

[Arthur Lemmens]

Cyrus Harmon wrote:

> Following up on my post from yesterday, even when I can coerce the
> "lots of transactions" into working, at some point performance breaks
> down rather severely.

I haven't looked at this in detail, but my first guess would be that
the garbage collector settings and/or performance is critical here.

> yes, along the way there is some fluctuation, as, I imagine, the
> indices and caches grow, etc... but we reach a threshold where it
> takes roughly .5 sec and 3M of consing for every object.

3M of consing per object is ridiculously much of course.  I'm pretty
sure that it should be possible to reduce this a lot by tracing some
of the garbage collector routines and looking at how much work they do.

One thing you could consider is to turn the garbage collector off
during the phase where you're creating very many objects (initializing
your database maybe?).  In fact, you could just turn it off, period.
As long as your disk is big enough, of course...

Let me know if turning the GC off doesn't help.

> And it would be nice if this approach worked as well.

Yes.  Having a separate transaction for each created object is not the
most efficient way and should not be necessary, but obviously it should
work and it shouldn't be ridiculously slow.

Arthur