<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
Dear Kevin,<BR>
I would not go so far as to see it is outside what you should expect of Elephant, but <BR>
certainly if I had to store thousands of datapoints in conceptual groups numbering in the <BR>
hundreds my first instinct would be to place them all in one database, and put them<BR>
in different btrees there.<BR>
Sleepycat should be perfectly happy with millions and millions of data objects.<BR>
I vaguely remember running into a problem similar to what you are having. <BR>
The idiom you are using, the macro "with-open-store", should be closing everything,<BR>
and appears to have the code in place to do the right thing. You are using it correctly,<BR>
at least conceptually.<BR>
I cannot guess whether this is a bug in the way that we close things, or some bug<BR>
in the Sleepycat library itself that will be harder to work around. <BR>
A heroic programmer would take your code and turn it in to a test, and add the test<BR>
the suite, so that even if there is some limit we can not easily work around, we can <BR>
at least explore the limit. If you will post or send me an example of that, I will <BR>
work it into the test suite. It will be good to make sure that both the Sleepycat and <BR>
SQL implementations work correctly in this respect.<BR>
<BR>
I don't know anything about your needs or application, but you certainly should consider<BR>
simply using one database, unless there is some obvious reason this is bad for you.<BR>
(In my opinion, the issue of "data ownership" is the only one that typically drives you<BR>
to split a datastore across separate databases: if client X things they own the data<BR>
and want to be able to wipe it out and back it up independent of client Y, it is often<BR>
easiest to use completely separate datastores. But there is not much of a performance<BR>
reason to do so.<BR>
<BR>
<BR>
On Mon, 2006-03-06 at 18:25 -0500, Kevin Y. Kim (Lists) wrote:
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">Hi,</FONT>
<FONT COLOR="#000000">(Perhaps I'm going about this all wrong, but...)</FONT>
<FONT COLOR="#000000">I'm trying to populate a large number of Sleepycat databases.</FONT>
<FONT COLOR="#000000">I've created the necessary underlying directories.</FONT>
<FONT COLOR="#000000">What I'm attempting is something like this:</FONT>
<FONT COLOR="#000000">(defun populate-db (tree data-pairs)</FONT>
<FONT COLOR="#000000"> (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p))) </FONT>
<FONT COLOR="#000000">data-pairs))</FONT>
<FONT COLOR="#000000">(mapcar #'(lambda (db)</FONT>
<FONT COLOR="#000000"> (with-open-store db</FONT>
<FONT COLOR="#000000"> (setf btree (make-btree)</FONT>
<FONT COLOR="#000000"> (add-to-root "tree" btree)</FONT>
<FONT COLOR="#000000"> (populate-db *btree* list-of-data-pairs)))</FONT>
<FONT COLOR="#000000"> list-of-databases)</FONT>
<FONT COLOR="#000000">I have hundreds of databases, with each database having thousands of </FONT>
<FONT COLOR="#000000">data-pairs.</FONT>
<FONT COLOR="#000000">So, after loading up around 150 databases, I get an error from </FONT>
<FONT COLOR="#000000">Sleepycat that</FONT>
<FONT COLOR="#000000">I've got too many databases open. I *assume* that even though I've </FONT>
<FONT COLOR="#000000">wrapped</FONT>
<FONT COLOR="#000000">everything in the "with-open-store" call, the databases are not closing</FONT>
<FONT COLOR="#000000">because I've got a reference to a btree. Is that correct? If so, is </FONT>
<FONT COLOR="#000000">there an</FONT>
<FONT COLOR="#000000">explicit way I can ensure that each database is closed before I move </FONT>
<FONT COLOR="#000000">on to the</FONT>
<FONT COLOR="#000000">next one?</FONT>
<FONT COLOR="#000000">If anyone has alternate suggestions on how I should store this data, </FONT>
<FONT COLOR="#000000">feel free to</FONT>
<FONT COLOR="#000000">pass them along.</FONT>
<FONT COLOR="#000000">Thanks,</FONT>
<FONT COLOR="#000000">-kevin</FONT>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">elephant-devel site list</FONT>
<FONT COLOR="#000000"><A HREF="mailto:elephant-devel@common-lisp.net">elephant-devel@common-lisp.net</A></FONT>
<FONT COLOR="#000000"><A HREF="http://common-lisp.net/mailman/listinfo/elephant-devel">http://common-lisp.net/mailman/listinfo/elephant-devel</A></FONT>
</PRE>
</BLOCKQUOTE>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
----<BR>
Robert L. Read, PhD read &T robertlread.net<BR>
Consider visiting Progressive Engineering: http://robertlread.net/pe<BR>
In Austin: 912-8593 "Think globally, Act locally." -- RBF<BR>
<BR>
<BR>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>