[elephant-devel] get-instances-by-value andget-instances-by-class

[Alex Mizrahi]

 IE> As I recall, when allocating space in the heap for new objects, SBCL 
will periodically
 IE> ask the OS for more heap memory.  Even if the objects are garbage 
collected later,
 IE> the allocated memory will remain (SBCL never releases heap memory back 
to the
 IE> OS even if it isn't using most of it).  It also asks for new heap space 
in increasingly
 IE> large chunks much like a hash table.

SBCL allocates a memory chunk of a fixed size, but it is only reserved at 
the beginning -- it is a normal behaviour on Linux that memory is actually 
allocated only on the first write to a page.  So, yes, memory usage (RSS --  
Resident Set Size) grows after use, but it does not grow indefinitely. There 
is --dynamic-space-size option in SBCL to control heap size.

It makes sense to set --dynamic-space-size to some reasonable value, and 
then watch if application dies of memory starvation.

If it does not die, then it is not a leak. Maybe SBCL just does not feel a 
need to clean up weak hash table or some memory generations, and if it is 
within allocated heap, it has a right to keep them.

 IE> It's possible that some of this memory is the shared memory allocated 
by BDB - but that can is upper-bounded in the my-config.sexp file and should 
be static.

It is also possible that there is a bug in Elephant, somewhere in FFI code 
or in functions which do foreign calls.

Technically it is also possible that leak is caused by SBCL itself -- SBCL 
has a conservative GC, but it only is a problem in some rare situations, 
when you build very long lists, for example, and probably it's not a case 
here.

---------------------
(dotimes (i 1000000)
    (elephant:get-instances-by-value 'd::TopicIdentificationC 'd::uri 
"goethe"))
;--> memory that is not released after operation 29.0 MB


(dotimes (i 1000000)
    (elephant:get-instance-by-value 'd::TopicIdentificationC 'd::uri 
"goethe"))
;--> memory that is not released after operation 0.1 MB
---------------------

This is pretty interesting, if indeed get-instances-by-value leak and 
get-instance-by-value does not, it should be easy to pinpoint the problem.

How many instances does get-instances-by-value return? Is it a long list?