[cffi-devel] how to treat expected failures in tests

[Anton Vodonosov]

On 01/10/2012 04:28 AM, Luís Oliveira wrote:
> I may have not been consistent in my usage of marking expected
> failures, but they mark known bugs unlikely to be fixed in the
> short-term. Either in CFFI or in the Lisp implementation. 
> ...
> In terms of notifications, I would rather be warned about new failures. 
> In terms of a summary, I'd like to see the results broken down 
> into OK, FAIL, KNOWNFAIL.

10.01.2012, 23:12, "Jeff Cunningham" <jeffrey at jkcunningham.com>:
> How about OK, FAIL, UNEXPECTEDOK, and EXPECTEDFAIL? You have to consider
> the the cases where one expects a failure but it passes too. 

I think it is rather theoretical. If no test frameworks provide a notion of UNEXPECTEDOK,
this means it was not needed in regression testing practice.

I am even reluctant to the EXPECTEDFAIL, because the word is contradictory and
the meaning is not obvious and confusing. 

If take into account that test results are observed not only by developers, but also 
by library users, we can imagine a user seeing EXPECTEDFAIL and asking himself: 
"Excpected FAIL... Is it OK? Can I use the library?"

But I see that several regression testing frameworks provide a notion of expected
failures and developers use it. 

And now I understand the goal - to simplify detection of _new_ regressions.

Therefore I think I will introduce an expected failure status in the cl-test-grid
(in the near feature).

Jeff Cunningham:
> In a testing scenario, "expected failure" to me means the test was
> designed to fail and it did. Usually, these are set up to test error
> handling. 

Robert Goldman:
> That is not how the term is used in the CFFI tests, or in most of the
> unit testing libraries. 

Indeed, Robert is right. If we want to test error handling by designing
a test which should signal an error, than if error is really signaled, 
the test status is OK, and if error is not signaled, the status is FAIL. 

> In a large testing environment we would periodically toss in a
> couple tests we knew would fail - one thing it tests it that the people
> running the tests are actually running them. If they didn't come back
> with these failures we knew there was a breakdown in the process.

The goal of cl-test-grid is that if people are running tests, the results
are shared and we can always check the online reports :)

PS: even today it is possible to distinguish expected failure from 
unexpected in cl-test-grid - one just needs to click the
"fail" link to open the logs, where the tests suite prints if 
the failures are unexpected or not. BTW, you might have noticed
that CFFI has unexpected failures almost on all the lisp implementations.

Best regards,
- Anton