From helmut at cybernetic-systems.de Tue Sep 2 08:53:56 2008 From: helmut at cybernetic-systems.de (Dr. Helmut G. Enders) Date: Tue, 02 Sep 2008 10:53:56 +0200 Subject: [rucksack-devel] Answer of cross post to [clbuild] Message-ID: <48BCFF24.8050200@cybernetic-systems.de> > > .... > > and would like to add a rucksack - line to the clbuild (wnpp-) projects: > > > > rucksack get_cvs_clnet #Persistence library for Common Lisp. Davir Lichtenblau [clbuild-devel]: Done. Helmut From helmut at cybernetic-systems.de Thu Sep 18 10:22:31 2008 From: helmut at cybernetic-systems.de (Helmut G. Enders) Date: Thu, 18 Sep 2008 12:22:31 +0200 Subject: [rucksack-devel] memory usage of rucksack. Message-ID: <48D22BE7.9030901@cybernetic-systems.de> Can somebody estimate the memory usage of a rucksack query, does it grow (disproportionate) to the increasing size of the data in rucksack (e.g. loading big btrees into the image etc.) (rough estimate +- 20 MB would be ok). Helmut Background: I have bought an Alix Board http://www.alix-board.de/produkte/alix2c3.html in a http://www.alix-board.de/produkte/case-alix2c3.html using a standard Debian installation and sbcl. It is an ideal hunchentoot server, fast, no moving parts, no fan, no noise, energy consumption ca. 5W, powered over Ethernet, three NICs (for the folks, who can't live without iptables (DMZ) :-) the only bottleneck is memory 256 MB. The system wird iptables, squid, dnsmasq, etc. needs ca. 20 MB, sbcl with my lines of code, hunchentoot, rucksack et al. needs ca. 45 MB. Remains ca. 190 MB. One of my apps (warehousing) has an 800 MB image (all articles are in the image). The result of a query are max 3-4 MB of article data. Are the remaining ca. 190MB of the board enough, if porting it to rucksack, even if the rucksack grows to 5 or 10 GB (within the same scenario)? From alemmens at xs4all.nl Thu Sep 18 14:27:39 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 18 Sep 2008 16:27:39 +0200 Subject: [rucksack-devel] memory usage of rucksack. In-Reply-To: <48D22BE7.9030901@cybernetic-systems.de> References: <48D22BE7.9030901@cybernetic-systems.de> Message-ID: Helmut Enders wrote: > Can somebody estimate the memory usage of a rucksack query, > does it grow (disproportionate) to the increasing size of the > data in rucksack (e.g. loading big btrees into the image etc.) > (rough estimate +- 20 MB would be ok). The basic idea is that this is controlled by the cache (see cache.lisp). In particular, you can use the :SIZE initarg for caches to control how many objects the cache will keep in memory. When you open a rucksack, you can use the :CACHE-ARGS keyword parameter to pass initargs to the cache. So something like the following should work (but I didn't test it): (with-rucksack (rucksack "/tmp/" :cache-args '(:size 10000)) ...) There are a few things that you should be aware of here: - All dirty objects will be kept in memory until they are saved by a transaction-commit. So if you have transactions that change very many objects, the cache may grow much bigger than the specified size. - This one should be obvious: if you use a very small cache, almost everything will be kept on disk all the time (which could slow down your programs considerably). > I have bought an Alix Board > http://www.alix-board.de/produkte/alix2c3.html > in a > http://www.alix-board.de/produkte/case-alix2c3.html > > using a standard Debian installation and sbcl. > > It is an ideal hunchentoot server, fast, > no moving parts, no fan, no noise, > energy consumption ca. 5W, powered over Ethernet, > three NICs (for the folks, who can't live without iptables (DMZ) :-) Sounds like fun. > One of my apps (warehousing) has an 800 MB image (all articles > are in the image). The result of a query are max 3-4 MB of article > data. Are the remaining ca. 190MB of the board enough, if porting > it to rucksack, even if the rucksack grows to 5 or 10 GB > (within the same scenario)? I think it should be possible in principle. But in practice, you may need to do some tuning (of your data, or of the indexes that you use, or of the Rucksack code) to get the performance you need. Arthur From alemmens at xs4all.nl Thu Sep 18 15:13:22 2008 From: alemmens at xs4all.nl (Arthur Lemmens) Date: Thu, 18 Sep 2008 17:13:22 +0200 Subject: [rucksack-devel] classes In-Reply-To: <48B7B626.7050505@cybernetic-systems.de> References: <48B7B626.7050505@cybernetic-systems.de> Message-ID: [Sorry for the late reply.] Helmut Enders wrote: > Using rucksack with many applications, and many users > pro application (one rucksack for one user), > I'm looking for a function to request the "meta" data > of a rucksack (what psql is for postgresql). > - what classes (slots) are there etc. > > I think, it is not to difficult, to do that on user-level, > like a doc-list of "(defclass xyz ()..." strings, > with a defined name in root. After an eval on these strings, > the rest can be done with the map-... functions. > > Is such a functionality in rucksack, or is a plan already in place? It sounds like the schema table would be a good basis for something like this (see schema-table.lisp). I don't think that it should be necessary to do an EVAL on strings containing DEFCLASS forms. Arthur