[cffi-devel] how to treat expected failures in tests

[Luís Oliveira]

On Tue, Jan 10, 2012 at 12:00 PM, Anton Vodonosov <avodonosov at yandex.ru> wrote:
> What meaning do you put into the term "expected failure"? Does it mean the library is buggy, but
> these bugs are known? Or it means that some non-required features are absent, but the
> library in general OK?

I may have not been consistent in my usage of marking expected
failures, but they mark known bugs unlikely to be fixed in the
short-term. Either in CFFI or in the Lisp implementation. Now that we
have a bugtracker, known failures should definitely point to their
respective issue.

Non-required features are marked using cffi-sys::foo symbols in
*features* so we shouldn't have failures related to that.


> I am interested in both short answer - as a library author, how do you think CFFI test suite
> should be marked if only expected failures present - OK or FAIL?

Depends on the use case. In terms of notifications, I would rather be
warned about new failures. In terms of a summary, I'd like to see the
results broken down into OK, FAIL, KNOWNFAIL.


> And also I am curious in this concrete example, what these 4 failures mean for CFFI on ECL?

DEFCFUN.NOOP fails because ECL's :void type returns NIL instead of
(VALUES) and we'd need to implement a work-around in CFFI. Since this
is a such a minor detail, it's sitting unfixed in the foreseeable
future. CALLBACKS.BFF.1 is an ECL bug: it used to crash (or hang?)
while compiling the SUM-126-NO-LL defcallback. It might have been
fixed in the meantime. The string encoding failures are a known CFFI
bug, IIRC.

HTH,

-- 
Luís Oliveira
http://r42.eu/~luis/