[elephant-devel] Berkeley DB error: Cannot allocate memory.

[Marc]

Ian Eslick wrote:
>
>> I'm a bit surprised about this. Most relational databases use
>> journalling and / or rollback files to ensure that they can commit or
>> rollback large transactions as one atomic unit. The procedure is quite
>> neatly described in http://www.sqlite.org/atomiccommit.html and
>> http://www.sqlite.org/tempfiles.html, but works AFAIK more or less the
>> same on other RDMSs.
>
> Actually, as I understand it, SQlite makes all the edits in user
> memory rather than in a shared memory region so it's just easier to
> allocate the memory in that case.  The tradeoff as you point out is
> serializing side-effecting transactions at the whole-DB level. 
> Actually if you had shared locks you could probably still pull of the
> same rough model...
>
> BDB is tuned for highly concurrent, but independent, write
> transactions across multiple threads/processes.  The shared memory
> region is used for sharing the fine-grained locks as well as for a
> shared cache so each process doesn't have to separately allocate
> memory and that you can have processes with various levels of read
> isolation.  This unfortunately leads to the requirement to
> pre-allocate that shared memory region and your current dilemma.
>
that sounds like the key for this specific mismatch of the BDB locking
philosophy – which for the scenarios you describe is perfect – and our
quite different needs. We can (well, have to) live with these
restrictions for now. However, if there ever were a new lisp backend, I
don't think it'd necessarily have to have precisely the same constraints.
>> I've just tried it out with sqlite which quite happily handles a
>> transaction of 10 million inserts as an atomic unit both for commit and
>> rollback – to be sure, in this case at the price of an extended read /
>> write lock on the DB and, of course, not in one physical write
>> operation. Main memory hardly enters the equation there.
>
> Yep, this is definitely a sweet spot for SQLite.
>
> We actually might be able to configure BDB to behave similarly (tune
> it for single-writer, multiple-readers) as an optional setup to
> Elephant; but we'd have to look into this a bit as I'm not entirely
> sure BDB can meet your constraints here.  I'm a little stale on all
> this these days...
>
that'd be great, of course.

Best regards,

Marc