[elephant-devel] Berkeley DB error: Cannot allocate memory.

[Victor Kryukov]

On Jan 4, 2008 2:15 AM, Leslie P. Polzer <leslie.polzer at gmx.net> wrote:
>
> > I've decided to put elephant 0.9.1 under some heavy load test, and
> > play with Netflix data set a little bit. The attached program that
> > tries to import everything in BerkeleyDB fails when trying to import
> > movie file number 8 with the following traceback. Do you have any idea
> > what could be the problem?
>
> No, but we might if you tell us
>
> - OS
> - RAM
> - Swap limits
> - Kernel limits on memory (e.g. /etc/security/limits.conf for GNU/Linux)

Of course :).

# uname -a
Linux esculap 2.6.22-14-generic #1 SMP Tue Dec 18 05:28:27 UTC 2007
x86_64 GNU/Linux
(Ubuntu 7.10)

RAM: Physical 2GB, SWAP 4GB
Lisp: SBCL 1.0.13

/etc/security/limits.conf contains only commented out strings.

> - size of the data you're trying to pull

Well, I'm trying to import Netflix data set. It consists of ratings
(from 1 to 5) for 17770 movies given by 480189 different users; total
number of ratings is 100480507.

Honestly, I don't know how to map that nicely into the Elephant world,
so I'm using the following approach; each user (or movies) is a class
with 2 slots: id of type (integer 32), and ratings, which is btree
mapping movies (or users) ids into respective ratings.

I *suspect* that data mapping above is horribly inefficient (too many
btrees?), but my intuition may be wrong.

Anyway, first 8 files contain exactly 20008 ratings, so at the point
of failure we should have created:

No more then 8 movie objects
No more then 20008 user objects
No more then 20016 (= 20008 + 8) btrees (slot elements of the above
two classes).

I'd be happy to provide more details.

Regards,
Victor.