[elephant-devel] Berkeley DB error: Cannot allocate memory.
Ian Eslick
eslick at media.mit.edu
Tue Sep 23 18:38:11 UTC 2008
Comments on original "out of memory problems"
- Running out of disk space for logs can also be a problem as well as
transaction size
- The max # of locks needed will grow with the log of the DB size and
linearly with the number of concurrent transactions. You can run
db_stat to see what your resource usage is.
- When bulk loading be sure to pull the data from a file into memory
before running the transaction if there might be any concurrent
transactions on that DB. (think about what happens when you have to
restart the transaction and the DB state is reset but the stream &
loop variables in memory are not).
> but not for larger logical transactions where they'd
> really be called for. Raising the limits at best slightly lessens the
> pain, but is not really a solution.
If raising the limits isn't a solution, then your application is
probably mismatched with Elephant/BDB. There might be some artificial
reasons you are seeing these problems.
1) If you have a large DB with deep indices and highly concurrent
transactions, then you may start hit lock limits as described above.
You should be able to set these limits high enough that you'll never
hit problems. Fortunately increasing locks is a good solution. max-
concurrent-sessions * max-objects-per-session * 2 + 20% is a good
number for max locks. You can use db_stat to see what kind of locks-
per-session numbers you're seeing and an analysis of your application
should tell you the max-concurrent-sessions.
Max concurrent transactions is upper-bounded by your machine's
processing capacity and max # of locks is proportional to the
logarithm of the number of objects which will grow very slowly so can
be given practical upper bounds.
(Set lock sizes in config.sexp - :berkeley-db-max-locks & :berkeley-db-
max-objects)
2) If you really are blowing out the BDB data cache in normal
operation (vs. bulk loads) then your application is either not using
transactions efficiently or is highly mismatched to any transactional
DB whether that be BDB or some future lisp implementation.
If a logical transaction results in writing or modifying 10's of
megabytes of data, then no transactional DB will ever work for you.
This is just physics. To get ACID properties you have to be able to
stage all side effects in memory to ensure that they are written as an
atomic unit. If you don't have enough memory to perform this staging,
then you can't implement a transaction.
There is a special case where you are manipulating large data objects
such as full-res images or uploaded files. I use a hybrid solution
for these where BDB keeps pointers to disk files which store just the
large items. If you can handle potential inconsistency between the on-
disk storage and the DB, then you can write the file to disk, keep a
pointer to it and do the bookkeeping in the DB inside a single
transaction. You'll run into serializer limits before you run into
cache size limits for large data items anyway.
The following may be obvious, but it's easy to get complacent and
forget that Elephant is backed up by a transactional DB and that you
can't really ignore those implications for real-world applications.
- The concept behind the transaction is to combine small DB operations
into a single atomic chunk. Sometimes you can wrap a small set of
such chunks for performance. You shouldn't be wrapping lots of
unrelated operations into a single transaction. This is better for
avoiding concurrency conflicts as well.
For example, all composite operations inside Elephant use transactions
to ensure consistency of slot and index values. Slot writes to
indexed slots wrap both the write and the index updates into a single
transaction. Otherwise every sub-operation ends up in its own little
transaction and you can get inconsistent indices if you have an error
in between the slot write and index update.
- One downside of a transactional store on BDB is every little
transaction results in a set of pages being flushed to disk (any Btree
index pages and the leaf pages with data). Flushing pages to disk
waits for a write-done from the driver; this can be very expensive!
(10's of ms if I remember correctly). Flushing one page takes about
the same time as flushing a few dozen pages so you can save
significant run time by wrapping your leaf operations in a transaction
- plus all the usual ACID properties.
In my web application (using weblocks) I wrap every ajax callback on
the server side in one big transaction since I know apriori that
operations within these callbacks aren't going to blow out 20M of
cache or 1000's of locks. Typically I update a couple of objects and
indices. This may be a few dozen 4k or 8k pages.
Cheers,
Ian
On Sep 23, 2008, at 9:42 AM, Marc wrote:
> Ian Eslick wrote:
>> You could be running out of cache or locks. I believe there are now
>> parameters in config.sexp you can set to raise the default limits.
>> The lack of robustness to allocation failures is a problem with
>> Berkeley DB.
>>
>> Unfortunately, running recovery isn't a trivial process. You have to
>> guarantee that all other threads have released any Berkeley DB
>> resources (abort all active transactions) and don't try to request
>> any
>> more (meaning no persistent slot reads/writes for Elephant) so you
>> essentially need to get inside the state of each process, abort any
>> transactions, and then suspend each thread. This isn't something
>> that
>> you can canonicalize inside the Elephant library.
>>
>> Chalk this up as another reason to someday implement a lisp-only
>> version of the library!
> Indeed, that'd be a dream. For us, at least, this prevents us from
> seriously using transactions at all in elephant. We do use them to
> speed
> up some bulk inserts when we know that the number of inserts in one go
> won't be too big, but not for larger logical transactions where they'd
> really be called for. Raising the limits at best slightly lessens the
> pain, but is not really a solution.
>
> Best regards,
>
> Marc
>>
>> On Sep 21, 2008, at 10:25 PM, Red Daly wrote:
>>
>>> I have recently run into "Cannot allocate memory" problems with
>>> elephant on a production server. Unfortunately when one transaction
>>> is too large, it seems to blow the database until a manual recover
>>> is
>>> done.
>>>
>>> The occasional failure is slightly worrisome but it the whole
>>> database requiring a manual recover from one extra-large transaction
>>> is a scary thought for a live application with thousands of users.
>>>
>>> Why does the memory allocation failure sour the whole database
>>> instead of aborting a single transaction? I think that elephant
>>> should try to encapsulate this failure, recovering the database or
>>> whatever is necessary to make the store usable for the next
>>> transaction.
>>>
>>> Best,
>>> Red Daly
>>>
>>>
>>> On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov
>>> <victor.kryukov at gmail.com> wrote:
>>> On Jan 4, 2008 2:54 AM, Ian Eslick <eslick at csail.mit.edu> wrote:
>>>> Hi Victor,
>>>>
>>>> Sounds like your transaction is blowing out the shared memory
>>>> allocated by Berkeley DB to store dirty pages. This is caused by
>>>> transactions that are too large; putting an entire file of data
>>>> could
>>>> well accomplish this. (We really should change the error message
>>>> to
>>>> be more informative in these cases).
>>>>
>>>> Try pushing with-transaction into the loop in import-movie as
>>>> follows:
>>>
>>> Thanks for your suggestion, Ian - the problem was solved once I've
>>> moved with-transaction inside the collect-rating-info.
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
More information about the elephant-devel
mailing list