[elephant-devel] Berkeley DB error: Cannot allocate memory.

[Ian Eslick]

You could be running out of cache or locks.  I believe there are now  
parameters in config.sexp you can set to raise the default limits.   
The lack of robustness to allocation failures is a problem with  
Berkeley DB.

Unfortunately, running recovery isn't a trivial process.  You have to  
guarantee that all other threads have released any Berkeley DB  
resources (abort all active transactions) and don't try to request any  
more (meaning no persistent slot reads/writes for Elephant) so you  
essentially need to get inside the state of each process, abort any  
transactions, and then suspend each thread.  This isn't something that  
you can canonicalize inside the Elephant library.

Chalk this up as another reason to someday implement a lisp-only  
version of the library!

Ian

On Sep 21, 2008, at 10:25 PM, Red Daly wrote:

> I have recently run into "Cannot allocate memory" problems with  
> elephant on a production server.  Unfortunately when one transaction  
> is too large, it seems to blow the database until a manual recover  
> is done.
>
> The occasional failure is slightly worrisome but it the whole  
> database requiring a manual recover from one extra-large transaction  
> is a scary thought for a live application with thousands of users.
>
> Why does the memory allocation failure sour the whole database  
> instead of aborting a single transaction?  I think that elephant  
> should try to encapsulate this failure, recovering the database or  
> whatever is necessary to make the store usable for the next  
> transaction.
>
> Best,
> Red Daly
>
>
> On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov <victor.kryukov at gmail.com 
> > wrote:
> On Jan 4, 2008 2:54 AM, Ian Eslick <eslick at csail.mit.edu> wrote:
> > Hi Victor,
> >
> > Sounds like your transaction is blowing out the shared memory
> > allocated by Berkeley DB to store dirty pages.  This is caused by
> > transactions that are too large; putting an entire file of data  
> could
> > well accomplish this.  (We really should change the error message to
> > be more informative in these cases).
> >
> > Try pushing with-transaction into the loop in import-movie as  
> follows:
>
> Thanks for your suggestion, Ian - the problem was solved once I've
> moved with-transaction inside the collect-rating-info.
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel