[rucksack-devel] memory usage of rucksack.

[Arthur Lemmens]

Helmut Enders wrote:

> Can somebody estimate the memory usage of a rucksack query,
>    does it grow (disproportionate) to the increasing size of the
>    data in rucksack (e.g. loading big btrees into the image etc.)
>    (rough estimate +- 20 MB would be ok).

The basic idea is that this is controlled by the cache (see cache.lisp).
In particular, you can use the :SIZE initarg for caches to control how
many objects the cache will keep in memory.  When you open a rucksack,
you can use the :CACHE-ARGS keyword parameter to pass initargs to the
cache.  So something like the following should work (but I didn't test
it):

  (with-rucksack (rucksack "/tmp/" :cache-args '(:size 10000))
     ...)

There are a few things that you should be aware of here:

- All dirty objects will be kept in memory until they are saved
  by a transaction-commit.  So if you have transactions that change
  very many objects, the cache may grow much bigger than the specified
  size.

- This one should be obvious: if you use a very small cache, almost
  everything will be kept on disk all the time (which could slow down
  your programs considerably).


> I have bought an Alix Board
>      http://www.alix-board.de/produkte/alix2c3.html
> in a
>      http://www.alix-board.de/produkte/case-alix2c3.html
>
> using a standard Debian installation and sbcl.
>
> It is an ideal hunchentoot server, fast,
> no moving parts, no fan, no noise,
> energy consumption ca. 5W, powered over Ethernet,
> three NICs (for the folks, who can't live without iptables (DMZ) :-)

Sounds like fun.


> One of my apps (warehousing) has an 800 MB image (all articles
> are in the image). The result of a query are max 3-4 MB of article
> data.  Are the remaining ca. 190MB of the board enough, if porting
> it to rucksack, even if the rucksack grows to 5 or 10 GB
> (within the same scenario)?

I think it should be possible in principle.  But in practice, you may
need to do some tuning (of your data, or of the indexes that you use,
or of the Rucksack code) to get the performance you need.

Arthur