[cffi-devel] [cl-test-grid] Re: how to treat expected failures in tests

[Anton Vodonosov]

14.03.2012, 04:01, "Luís Oliveira" <luismbo at gmail.com>:
>>  Lisp compilation failures are present in logs, but in this case we have C library compilation
>>  error. This may be seen by component description in the ASDF error message:
>>  #<C-TEST-LIB "cffi-tests" "tests" "libtest">
>
> Ah, my bad. While I'm making feature requests, perhaps a backtrace
> would be useful, though? :-)

They would of course.. but there is no portable way to retrieve a backtrace of CL condition.

>
>>  In cffi-tests.asd the
>>
>>  (defmethod perform ((o compile-op) (c c-test-lib))
>>
>>  does not try to run "make" on windows,
>>
>>   #-windows
>>   (unless (zerop (run-shell-command
>>                   "cd ~A; make"
>>
>>  and on windows the only error signaled in absense of the native library is
>>  cffi:load-foreign-library-error. I rely on it when detecting :no-resource.
>
> Should we assume that 'make' is available on windows? Is that a common
> setup for cygwin/mingw? What about other toolsets such as Microsoft's?
> Do they ship with make?

Gnu make is often installed with cygwin (but not so many people install cygwin).
Microsoft Visual Studio ships with a program called nmake - mostly compatible
with make, but not completely. Again, not everyone have Visual Studio. Actually,
make is not that necessary, it's just one C file we need to compile, the compile.bat
from the tests directory is OK (but anyway requires Microsoft Visual C compiler). 

In short, even if we implement the compile-op for windows, it is going to fail for 85% of users.

I think it's OK to ask user to compile the library manually.

What I need to fix, is to more correctly recognize the cases when tests are impossible
to run. Probably some improvements in the compile-op implementation will help here,
but I don't know yet what it would be.

BTW, to be clear, the failure on ABCL represented in the logs as
    
    Class not found: com.sun.jna.Native

is also :no-resouce - CFFI on  ABCL requires jna.jar to present in classpaths, 
which I don't have. 

I have a TODO item recognize this situation as :no-resource, too.