[cffi-devel] how to treat expected failures in tests

[Jeff Cunningham]

On 01/10/2012 04:00 AM, Anton Vodonosov wrote:
> Hello.
>
> For my common lisp testing project I aggregate results of a library test suite into single value - ok/fail.
>
> I just tested ECL and have the following output from CFFI test suite:
>
> 4 out of 228 total tests failed: DEFCFUN.NOOP, CALLBACKS.BFF.1, STRING.ENCODING.UTF-16.BASIC,
>     STRING.ENCODINGS.ALL.BASIC.
> No unexpected failures.
>
> What meaning do you put into the term "expected failure"? Does it mean the library is buggy, but
> these bugs are known? Or it means that some non-required features are absent, but the
> library in general OK?
>
> I am interested in both short answer - as a library author, how do you think CFFI test suite
> should be marked if only expected failures present - OK or FAIL?
>
> And also I am curious in this concrete example, what these 4 failures mean for CFFI on ECL?
>

In a testing scenario, "expected failure" to me means the test was 
designed to fail and it did. Usually, these are set up to test error 
handling. In a large testing environment we would periodically toss in a 
couple tests we knew would fail - one thing it tests it that the people 
running the tests are actually running them. If they didn't come back 
with these failures we knew there was a breakdown in the process.

Jeff