<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
Yes, it's amusing.<BR>
<BR>
In my own work I use the Postgres backend; I know very little about SleepyCat. It seems<BR>
to me that this is more of a SleepyCat issue, then an Elephant issue. Perhaps you should<BR>
ask the SleepyCat list?<BR>
<BR>
Are you importing things into SleepyCat directly in the correct serialization format that <BR>
they can be read by Elephant? If so, I assume it is just a question of solving the SleepyCat<BR>
problems.<BR>
<BR>
An alternative would be to use the SQL-based backend. However, I doubt this will solve<BR>
your problem, since at present we (well, I wrote it) use a very inefficient serialization scheme<BR>
for the SQL-based backend that base64 encodes everything. This had the advantage that <BR>
it makes it work trouble-free with different database backends, but could clearly be improved upon.<BR>
However, it is more than efficient enough for all my work, and at present nobody is clamoring<BR>
to have it improved.<BR>
<BR>
Is your problem importing the data or using it once it is imported? It's hard for me to imagine<BR>
a problem so large that even the import time is a problem --- suppose it takes 24 hours --- can<BR>
you not afford to pay that?<BR>
<BR>
A drastic measure and potentially expensive measure would be to switch to a 64-bit architecture<BR>
with a huge memory. I intend to do that when forced by performance issues in my own work.<BR>
<BR>
<BR>
<BR>
On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">I will be running experiments in informatics and modeling in the future </FONT>
<FONT COLOR="#000000">that may contain (tens or hundreds of) millions of objects. Given the </FONT>
<FONT COLOR="#000000">ease of use of elephant so far, it would be great to use it as the </FONT>
<FONT COLOR="#000000">persistent store and avoid creating too many custom data structures.</FONT>
<FONT COLOR="#000000">I have recently run up against some performance bottlenecks when using </FONT>
<FONT COLOR="#000000">elephant to work with very large datasets (in the hundreds of millions </FONT>
<FONT COLOR="#000000">of objects). Using SleepyCat, I am able to import data very quickly </FONT>
<FONT COLOR="#000000">with a DB_CONFIG file with the following contents:</FONT>
<FONT COLOR="#000000">set_lk_max_locks 500000</FONT>
<FONT COLOR="#000000">set_lk_max_objects 500000</FONT>
<FONT COLOR="#000000">set_lk_max_lockers 500000</FONT>
<FONT COLOR="#000000">set_cachesize 1 0 0</FONT>
<FONT COLOR="#000000">I can import data very quickly until the 1 gb cache is too small to </FONT>
<FONT COLOR="#000000">allow complete in-memory access to the database. at this point it seems </FONT>
<FONT COLOR="#000000">that disk IO makes additional writes happen much slower. (I have also </FONT>
<FONT COLOR="#000000">tried increasing the 1 gb cache size, but the database fails to open if </FONT>
<FONT COLOR="#000000">it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb </FONT>
<FONT COLOR="#000000">swap, so the constraint seems to be physical memory.) the max_lock, </FONT>
<FONT COLOR="#000000">etc. lines allow transactions to contain hundreds of thousands of </FONT>
<FONT COLOR="#000000">individual locks, limiting the transaction throughput bottleneck</FONT>
<FONT COLOR="#000000">What are the technical restrictions on writing several million objects </FONT>
<FONT COLOR="#000000">to the datastore? Is it feasible to create a batch import feature to </FONT>
<FONT COLOR="#000000">allow large datasets to be imported using reasonable amounts of memory </FONT>
<FONT COLOR="#000000">for a desktop computer?</FONT>
<FONT COLOR="#000000">I hope this email is at least amusing!</FONT>
<FONT COLOR="#000000">Thanks again,</FONT>
<FONT COLOR="#000000">red daly</FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">elephant-devel site list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000"><A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>