[elephant-devel] Berkeley DB error: Cannot allocate memory.

[Ian Eslick]

Hi Victor,

Sounds like your transaction is blowing out the shared memory  
allocated by Berkeley DB to store dirty pages.  This is caused by  
transactions that are too large; putting an entire file of data could  
well accomplish this.  (We really should change the error message to  
be more informative in these cases).

Try pushing with-transaction into the loop in import-movie as follows:

(defun import-movie (n)
  "Import ratings information for movie number N."
  (iter (for line in-file (movie-filename n) using 'read-line)
    (unless (first-time-p) ; skip first line
      (with-transaction ()
        (collect-rating-info n line)))))

I expect that will fix it.

Ian

PS - Your data representation sounds fine as a first start.  You may  
want to rethink it when you start doing queries and in fact may want  
to build a custom data structure in static memory for performance  
reasons once your algorithms and related queries are clear to you.

PS - You may get some additional mileage by increasing the total  
memory cache size (you should lookup whether the transaction size is  
correlated with the overall cache size) using the my-config.sexp  
option (:berkeley-db-cachesize . 20971520)


On Jan 4, 2008, at 3:15 AM, Leslie P. Polzer wrote:

>
>> I've decided to put elephant 0.9.1 under some heavy load test, and
>> play with Netflix data set a little bit. The attached program that
>> tries to import everything in BerkeleyDB fails when trying to import
>> movie file number 8 with the following traceback. Do you have any  
>> idea
>> what could be the problem?
>
> No, but we might if you tell us
>
> - OS
> - RAM
> - Swap limits
> - Kernel limits on memory (e.g. /etc/security/limits.conf for GNU/ 
> Linux)
> - size of the data you're trying to pull
>
>  Leslie
>
> -- 
> My personal blog: http://blog.viridian-project.de/
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel