[elephant-devel] Too many files open

[Robert L. Read]

Dear Kevin,
    I would not go so far as to see it is outside what you should expect
of Elephant, but 
certainly if I had to store thousands of datapoints in conceptual groups
numbering in the 
hundreds my first instinct would be to place them all in one database,
and put them
in different btrees there.
    Sleepycat should be perfectly happy with millions and millions of
data objects.
    I vaguely remember running into a problem similar to what you are
having. 
    The idiom you are using, the macro "with-open-store", should be
closing everything,
and appears to have the code in place to do the right thing.  You are
using it correctly,
at least conceptually.
    I cannot guess whether this is a bug in the way that we close
things, or some bug
in the Sleepycat library itself that will be harder to work around.  
    A heroic programmer would take your code and turn it in to a test,
and add the test
the suite, so that even if there is some limit we can not easily work
around, we can 
at least explore the limit.  If you will post or send me an example of
that, I will 
work it into the test suite.  It will be good to make sure that both the
Sleepycat and 
SQL implementations work correctly in this respect.
    
    I don't know anything about your needs or application, but you
certainly should consider
simply using one database, unless there is some obvious reason this is
bad for you.
(In my opinion, the issue of "data ownership" is the only one that
typically drives you
to split a datastore across separate databases:  if client X things they
own the data
and want to be able to wipe it out and back it up independent of client
Y, it is often
easiest to use completely separate datastores.  But there is not much of
a performance
reason to do so.


On Mon, 2006-03-06 at 18:25 -0500, Kevin Y. Kim (Lists) wrote:

> Hi,
> 
> (Perhaps I'm going about this all wrong, but...)
> 
> I'm trying to populate a large number of Sleepycat databases.
> I've created the necessary underlying directories.
> What I'm attempting is something like this:
> 
> (defun populate-db (tree data-pairs)
>      (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p)))  
> data-pairs))
> 
> (mapcar #'(lambda (db)
>               (with-open-store db
>                   (setf btree (make-btree)
>                   (add-to-root "tree" btree)
> 		 (populate-db *btree* list-of-data-pairs)))
>          list-of-databases)
> 
> I have hundreds of databases, with each database having thousands of  
> data-pairs.
> 
> So, after loading up around 150 databases, I get an error from  
> Sleepycat that
> I've got too many databases open.  I *assume* that even though I've  
> wrapped
> everything in the "with-open-store" call, the databases are not closing
> because I've got a reference to a btree.  Is that correct?  If so, is  
> there an
> explicit way I can ensure that each database is closed before I move  
> on to the
> next one?
> 
> If anyone has alternate suggestions on how I should store this data,  
> feel free to
> pass them along.
> 
> Thanks,
> -kevin
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel

----
Robert L. Read, PhD                                     read &T
robertlread.net
Consider visiting Progressive Engineering:
http://robertlread.net/pe
In Austin: 912-8593                                        "Think
globally, Act locally." -- RBF


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20060306/28b54738/attachment.html>