[elephant-devel] Berkeley DB error: Cannot allocate memory.

[Marc]

Ian Eslick wrote:
> You could be running out of cache or locks.  I believe there are now
> parameters in config.sexp you can set to raise the default limits. 
> The lack of robustness to allocation failures is a problem with
> Berkeley DB.
>
> Unfortunately, running recovery isn't a trivial process.  You have to
> guarantee that all other threads have released any Berkeley DB
> resources (abort all active transactions) and don't try to request any
> more (meaning no persistent slot reads/writes for Elephant) so you
> essentially need to get inside the state of each process, abort any
> transactions, and then suspend each thread.  This isn't something that
> you can canonicalize inside the Elephant library.
>
> Chalk this up as another reason to someday implement a lisp-only
> version of the library!
Indeed, that'd be a dream. For us, at least, this prevents us from
seriously using transactions at all in elephant. We do use them to speed
up some bulk inserts when we know that the number of inserts in one go
won't be too big, but not for larger logical transactions where they'd
really be called for. Raising the limits at best slightly lessens the
pain, but is not really a solution.

Best regards,

Marc
>
> On Sep 21, 2008, at 10:25 PM, Red Daly wrote:
>
>> I have recently run into "Cannot allocate memory" problems with
>> elephant on a production server.  Unfortunately when one transaction
>> is too large, it seems to blow the database until a manual recover is
>> done.
>>
>> The occasional failure is slightly worrisome but it the whole
>> database requiring a manual recover from one extra-large transaction
>> is a scary thought for a live application with thousands of users.
>>
>> Why does the memory allocation failure sour the whole database
>> instead of aborting a single transaction?  I think that elephant
>> should try to encapsulate this failure, recovering the database or
>> whatever is necessary to make the store usable for the next transaction.
>>
>> Best,
>> Red Daly
>>
>>
>> On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov
>> <victor.kryukov at gmail.com> wrote:
>> On Jan 4, 2008 2:54 AM, Ian Eslick <eslick at csail.mit.edu> wrote:
>> > Hi Victor,
>> >
>> > Sounds like your transaction is blowing out the shared memory
>> > allocated by Berkeley DB to store dirty pages.  This is caused by
>> > transactions that are too large; putting an entire file of data could
>> > well accomplish this.  (We really should change the error message to
>> > be more informative in these cases).
>> >
>> > Try pushing with-transaction into the loop in import-movie as follows:
>>
>> Thanks for your suggestion, Ian - the problem was solved once I've
>> moved with-transaction inside the collect-rating-info.
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>