[elephant-devel] Too many files open

[Ian Eslick]

Another note on lots of databases.  with-open-store is a pretty
expensive operation, so unless you are only opening them infrequently
and for batch operation, a fine-grained use of with-open-store is likely
to cost you a great deal of performance. 

Ian

Robert L. Read wrote:
> Dear Kevin,
>     I would not go so far as to see it is outside what you should
> expect of Elephant, but
> certainly if I had to store thousands of datapoints in conceptual
> groups numbering in the
> hundreds my first instinct would be to place them all in one database,
> and put them
> in different btrees there.
>     Sleepycat should be perfectly happy with millions and millions of
> data objects.
>     I vaguely remember running into a problem similar to what you are
> having.
>     The idiom you are using, the macro "with-open-store", should be
> closing everything,
> and appears to have the code in place to do the right thing.  You are
> using it correctly,
> at least conceptually.
>     I cannot guess whether this is a bug in the way that we close
> things, or some bug
> in the Sleepycat library itself that will be harder to work around. 
>     A heroic programmer would take your code and turn it in to a test,
> and add the test
> the suite, so that even if there is some limit we can not easily work
> around, we can
> at least explore the limit.  If you will post or send me an example of
> that, I will
> work it into the test suite.  It will be good to make sure that both
> the Sleepycat and
> SQL implementations work correctly in this respect.
>    
>     I don't know anything about your needs or application, but you
> certainly should consider
> simply using one database, unless there is some obvious reason this is
> bad for you.
> (In my opinion, the issue of "data ownership" is the only one that
> typically drives you
> to split a datastore across separate databases:  if client X things
> they own the data
> and want to be able to wipe it out and back it up independent of
> client Y, it is often
> easiest to use completely separate datastores.  But there is not much
> of a performance
> reason to do so.
>
>
> On Mon, 2006-03-06 at 18:25 -0500, Kevin Y. Kim (Lists) wrote:
>> Hi,
>>
>> (Perhaps I'm going about this all wrong, but...)
>>
>> I'm trying to populate a large number of Sleepycat databases.
>> I've created the necessary underlying directories.
>> What I'm attempting is something like this:
>>
>> (defun populate-db (tree data-pairs)
>>      (mapcar #'(lambda (p) (setf (get-value (car p) tree) (cadr p)))  
>> data-pairs))
>>
>> (mapcar #'(lambda (db)
>>               (with-open-store db
>>                   (setf btree (make-btree)
>>                   (add-to-root "tree" btree)
>> 		 (populate-db *btree* list-of-data-pairs)))
>>          list-of-databases)
>>
>> I have hundreds of databases, with each database having thousands of  
>> data-pairs.
>>
>> So, after loading up around 150 databases, I get an error from  
>> Sleepycat that
>> I've got too many databases open.  I *assume* that even though I've  
>> wrapped
>> everything in the "with-open-store" call, the databases are not closing
>> because I've got a reference to a btree.  Is that correct?  If so, is  
>> there an
>> explicit way I can ensure that each database is closed before I move  
>> on to the
>> next one?
>>
>> If anyone has alternate suggestions on how I should store this data,  
>> feel free to
>> pass them along.
>>
>> Thanks,
>> -kevin
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net <mailto:elephant-devel at common-lisp.net>
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>     
> ----
> Robert L. Read, PhD                                     read &T
> robertlread.net
> Consider visiting Progressive Engineering:      http://robertlread.net/pe
> In Austin: 912-8593                                        "Think
> globally, Act locally." -- RBF
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel