[Ecls-list] ECL 10.3.1 GC finalization bug and patch

[Ram Krishnan]

On Sun, Mar 14, 2010 at 1:54 AM, Juan Jose Garcia-Ripoll
<juanjose.garciaripoll at googlemail.com> wrote:
> On Sun, Mar 14, 2010 at 2:30 AM, Ram Krishnan <kriyative at gmail.com> wrote:
>>
>> There's a fairly serious bug in the GC finalization code in
>> 10.3.1 (and 9.12.3). The effect of the bug is that if a program
>> allocates multiple ffi objects, and calls SET-FINALIZER on them, then
>> under some circumstances the finalizer may be called with a completely
>> bogus pointer object, or just crash with a SIGSEGV. The issue is in
>> the queueing_finalizer() function in src/c/alloc_2.d.
>
> It would be nice if you could provide a test case.
>
>>
>> After staring at this function for a while I gave up on what the
>> motivation behind it is. All that queueing_finalizer() does is to
>> create a list of the objects and their finalizers, which were passed
>> to it, so that those finalizers may be called at some later point. I
>> can't figure out what the benefit of deferring these finalization
>> calls is. Anyway, the simplest fix is to not do any queueing, and just
>> invoke the finalizer, like so:
>
> This can not be done. An arbitrary lisp finalizer can not be invoked inside
> the garbage collector when the code has been marked as not live.
>
> Among other things the problem is that some of its components may be
> destroyed when you try this approach. Say you have object A and B that point
> each other and suddenly they become  garbage-collectable. If you invoke the
> finalizer in the garbage collector then A's finalizer may be called before
> or after B's, but what is more important is that B or A might have been
> already destroyed by the collector.
>
> The approach we took was different. When an object becomes finalizable (and
> it is not a simple object such as a stream or so) the object is brought back
> to life and put in a queue which is known to the garbage collector. This
> queue is processed later at another point in time and then the object is
> removed from the queue and eventually garbage collected.
>
>> If there's a reason for the current behavior of queueing_finalizer(),
>> I look forward to understanding it.
>
> I hope the  text above helped you understand it. The situation is not at all
> trivial and the current one is the best solution I could come up with.
>
> Juanjo
>
> --
> Instituto de Física Fundamental, CSIC
> c/ Serrano, 113b, Madrid 28006 (Spain)
> http://juanjose.garciaripoll.googlepages.com
>

Hi Juan,

Firstly, sorry for rushing off the bug report without including a test
case. Here it is:

(defun finalize-foreign-object (o)
  (ffi:c-inline (o) (:pointer-void) :void
                "
{
  void *o = #0;
  printf(\";; freeing %x\\\n\", (int) o);
  fflush(stdout);
  free(o);
}"))

(defun alloc-foreign-object (n)
  (let ((obj (ffi:c-inline (n) (:int) :pointer-void "malloc(#0)" :one-liner t)))
    (format t "~&;; allocated foreign object ~s~%" obj)
    (ext:set-finalizer obj #'finalize-foreign-object)
    obj))

;; this should work correctly, with corresponding alloc and free
;; messages
(progn
  (alloc-foreign-object 10)
  (si:gc t)
  (si:gc t))

;; this will however not finalize all the allocated objects
(progn
  (let (x)
    (dotimes (i 10)
      (push (alloc-foreign-object 20) x)))
  (si:gc t)
  (si:gc t))

Secondly, I was wrong about the crashing behavior. After I'd sent my
bug report, I realized I had a call to FORMAT inside of the finalizer,
which is likely to cons and presumably cause unexpected behavior.
Removing that call to FORMAT eliminated the crashing behavior. So,
please disregard that.

However, the issue with only part of the queued objects being gc'ed
remains, as evidenced by the test case above.

Lastly, thanks for explaining queueing_finalizer(), and I appreciate
the motivation, but, I still don't follow the logic. Just so I have it
right, this is what I understood from your explanation:

 - new object gets assigned a finalizer
 - eventually, object is marked garbage
 - gc calls finalizer
 - finalizer is added to a queue, and will be called when that queue
becomes garbage

If a set of objects have inter-dependencies when they are first passed
into queueing_finalizer() (and get pushed into to_be_finalized), won't
those same dependencies exist when queueing_finalizer() is reinvoked
on the to_be_finalized queue? How is the dependency situation any
different in the second go around?

The basic_finalizer() I proposed assuredly doesn't address anything
more complex than the simplest cases, such as my test case. In my test
case of freeing foreign object pointers, which is pretty much the only
use of finalizers (other than closing file descriptors) I've ever
seen, there are ostensibly no dependencies. I suppose there may be
more exotic uses for finalizer, but I still don't see how the
queueing_finalizer() addresses those.

I completely agree that situation is not trivial, and I would just
like to find a balance between the needs of the now, versus the needs
of the future :)

Cheers,

-ram