[rucksack-devel] Re: Fwd: State of the nation and heap patch

[Arthur Lemmens]

Cyrus Harmon wrote:

> Thanks for these changes. My initial tests suggest that the new
> version is faster, but not overwhelmingly so. I'll try to do some more
> rigorous benchmarking (and profiling) and see what I can came up with.

I expect that most of the time is caused by class and slot indexing,
but it would be interesting to test that first.  For example, maybe you
could time how much time it takes to import your data without any
indexing at all?  And then with class indexing but no slot indexing?

Do you have slot indexes where relatively few slot values map to many
objects?  In that case, the current implementation is far too slow,
because it uses a plain list to represent the set of all btree values
that belong to to one key.  I'm working on changing that to a different
data structure, but I haven't finished that yet.

Arthur