<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
On Wed, 2006-10-11 at 17:57 -0700, Red Daly wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">I was importing into sleepycat using standard elephant routines. I am </FONT>
<FONT COLOR="#000000">not aware of an 'import mode' for sleepycat, but I will look into that </FONT>
<FONT COLOR="#000000">when I have a chance. </FONT>
</PRE>
</BLOCKQUOTE>
I do not think there is anyway to import things directly and to deal with Elephants serialization at<BR>
the same time. I think this is a dead end.
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000"> Another consideration using sleepycat is that </FONT>
<FONT COLOR="#000000">using BTrees with a large working set demands large amounts of memory </FONT>
<FONT COLOR="#000000">relative to a Hash representation. I am unfamiliar with the internals </FONT>
<FONT COLOR="#000000">of elephant and sleepycat, but it feels like the basic access method is </FONT>
<FONT COLOR="#000000">restricting performance, which seems to be described here:</FONT>
<FONT COLOR="#000000"><A HREF="http://www.sleepycat.com/docs/gsg/C/accessmethods.html#BTreeVSHash">http://www.sleepycat.com/docs/gsg/C/accessmethods.html#BTreeVSHash</A></FONT>
<FONT COLOR="#000000">My problem so far has been importing the data, which goes very fast </FONT>
<FONT COLOR="#000000">until sleepycat requires extensive disk access. The in-memory rate is </FONT>
<FONT COLOR="#000000">reasonable and would complete in a few hours. However, once disk </FONT>
<FONT COLOR="#000000">operations begin the import speed suggests it would take many days to </FONT>
<FONT COLOR="#000000">complete. I have yet to perform extensive benchmarks, but I estimate </FONT>
<FONT COLOR="#000000">the instantiation rate shifts from 1800 persistent class instantiations </FONT>
<FONT COLOR="#000000">/second to 120 / s.</FONT>
<FONT COLOR="#000000">here are the two approaches that I hypothesize may help performance. I </FONT>
<FONT COLOR="#000000">am admittedly unaware of innards of the two systems in question, so you </FONT>
<FONT COLOR="#000000">expert developers will know best. If either sounds appropriate or you </FONT>
<FONT COLOR="#000000">envision another possibility for allowing this kind of scaling, I will </FONT>
<FONT COLOR="#000000">look into implementing such a system.</FONT>
<FONT COLOR="#000000">1. decreasing the size of the working set is one possibility for </FONT>
<FONT COLOR="#000000">decreasing run-time memory requirements and disk access. I'm not sure </FONT>
<FONT COLOR="#000000">how the concept of a 'working set' translates from the sleepycat world </FONT>
<FONT COLOR="#000000">to the elephant world, but perhaps you do.</FONT>
</PRE>
</BLOCKQUOTE>
Elephant keeps very little in memory (unless you used my DCM module in the contrib directory.)<BR>
So I have to admit I don't quite understand why any import operation would suddenly shift to more <BR>
disk access. Now, doing a simple import of a lot of data WOULD tend to leave objects in memory<BR>
that could not be garbage collected, if you weren't careful. That could be solved by either <BR>
making SURE you don't keep a reference to the objects or by doing your import in large batches<BR>
(restarting LISP processes in between times). But I am just guesssing --- do you feel comfortable <BR>
posting your code, or pseudocode for how you do the import?
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">2. using a Hash instead of a BTree in the primary database? I am </FONT>
<FONT COLOR="#000000">unsure what this means for elephant.</FONT>
</PRE>
</BLOCKQUOTE>
I don't think that you make progress with that. Elephant depends very deeply on the BTree, <BR>
and, as far as I know, so does Sleepycat.
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">In the mean time I will depart from the every-class-is-persistent </FONT>
<FONT COLOR="#000000">approach and also use more traditional data structures.</FONT>
<FONT COLOR="#000000">Thanks again,</FONT>
<FONT COLOR="#000000">Red Daly</FONT>
<FONT COLOR="#000000">Robert L. Read wrote:</FONT>
<FONT COLOR="#000000">> Yes, it's amusing.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> In my own work I use the Postgres backend; I know very little about </FONT>
<FONT COLOR="#000000">> SleepyCat. It seems</FONT>
<FONT COLOR="#000000">> to me that this is more of a SleepyCat issue, then an Elephant issue. </FONT>
<FONT COLOR="#000000">> Perhaps you should</FONT>
<FONT COLOR="#000000">> ask the SleepyCat list?</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> Are you importing things into SleepyCat directly in the correct </FONT>
<FONT COLOR="#000000">> serialization format that</FONT>
<FONT COLOR="#000000">> they can be read by Elephant? If so, I assume it is just a question </FONT>
<FONT COLOR="#000000">> of solving the SleepyCat</FONT>
<FONT COLOR="#000000">> problems.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> An alternative would be to use the SQL-based backend. However, I </FONT>
<FONT COLOR="#000000">> doubt this will solve</FONT>
<FONT COLOR="#000000">> your problem, since at present we (well, I wrote it) use a very </FONT>
<FONT COLOR="#000000">> inefficient serialization scheme</FONT>
<FONT COLOR="#000000">> for the SQL-based backend that base64 encodes everything. This had </FONT>
<FONT COLOR="#000000">> the advantage that</FONT>
<FONT COLOR="#000000">> it makes it work trouble-free with different database backends, but </FONT>
<FONT COLOR="#000000">> could clearly be improved upon.</FONT>
<FONT COLOR="#000000">> However, it is more than efficient enough for all my work, and at </FONT>
<FONT COLOR="#000000">> present nobody is clamoring</FONT>
<FONT COLOR="#000000">> to have it improved.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> Is your problem importing the data or using it once it is imported? </FONT>
<FONT COLOR="#000000">> It's hard for me to imagine</FONT>
<FONT COLOR="#000000">> a problem so large that even the import time is a problem --- suppose </FONT>
<FONT COLOR="#000000">> it takes 24 hours --- can</FONT>
<FONT COLOR="#000000">> you not afford to pay that?</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> A drastic measure and potentially expensive measure would be to switch </FONT>
<FONT COLOR="#000000">> to a 64-bit architecture</FONT>
<FONT COLOR="#000000">> with a huge memory. I intend to do that when forced by performance </FONT>
<FONT COLOR="#000000">> issues in my own work.</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote:</FONT>
<FONT COLOR="#000000">>> I will be running experiments in informatics and modeling in the future </FONT>
<FONT COLOR="#000000">>> that may contain (tens or hundreds of) millions of objects. Given the </FONT>
<FONT COLOR="#000000">>> ease of use of elephant so far, it would be great to use it as the </FONT>
<FONT COLOR="#000000">>> persistent store and avoid creating too many custom data structures.</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> I have recently run up against some performance bottlenecks when using </FONT>
<FONT COLOR="#000000">>> elephant to work with very large datasets (in the hundreds of millions </FONT>
<FONT COLOR="#000000">>> of objects). Using SleepyCat, I am able to import data very quickly </FONT>
<FONT COLOR="#000000">>> with a DB_CONFIG file with the following contents:</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> set_lk_max_locks 500000</FONT>
<FONT COLOR="#000000">>> set_lk_max_objects 500000</FONT>
<FONT COLOR="#000000">>> set_lk_max_lockers 500000</FONT>
<FONT COLOR="#000000">>> set_cachesize 1 0 0</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> I can import data very quickly until the 1 gb cache is too small to </FONT>
<FONT COLOR="#000000">>> allow complete in-memory access to the database. at this point it seems </FONT>
<FONT COLOR="#000000">>> that disk IO makes additional writes happen much slower. (I have also </FONT>
<FONT COLOR="#000000">>> tried increasing the 1 gb cache size, but the database fails to open if </FONT>
<FONT COLOR="#000000">>> it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb </FONT>
<FONT COLOR="#000000">>> swap, so the constraint seems to be physical memory.) the max_lock, </FONT>
<FONT COLOR="#000000">>> etc. lines allow transactions to contain hundreds of thousands of </FONT>
<FONT COLOR="#000000">>> individual locks, limiting the transaction throughput bottleneck</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> What are the technical restrictions on writing several million objects </FONT>
<FONT COLOR="#000000">>> to the datastore? Is it feasible to create a batch import feature to </FONT>
<FONT COLOR="#000000">>> allow large datasets to be imported using reasonable amounts of memory </FONT>
<FONT COLOR="#000000">>> for a desktop computer?</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> I hope this email is at least amusing!</FONT>
<FONT COLOR="#000000">>></FONT>
<FONT COLOR="#000000">>> Thanks again,</FONT>
<FONT COLOR="#000000">>> red daly</FONT>
<FONT COLOR="#000000">>> _______________________________________________</FONT>
<FONT COLOR="#000000">>> elephant-devel site list</FONT>
<FONT COLOR="#000000">>> <A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A> <mailto:<A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A>></FONT>
<FONT COLOR="#000000">>> <A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
<FONT COLOR="#000000">>> </FONT>
<FONT COLOR="#000000">> ------------------------------------------------------------------------</FONT>
<FONT COLOR="#000000">></FONT>
<FONT COLOR="#000000">> _______________________________________________</FONT>
<FONT COLOR="#000000">> elephant-devel site list</FONT>
<FONT COLOR="#000000">> <A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000">> <A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">elephant-devel site list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000"><A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>