[cffi-devel] Unit tests failures on different lisps

[Anton Vodonosov]

29.12.2011, 15:08, "Luís Oliveira" <luismbo at gmail.com>:
> On Thu, Dec 29, 2011 at 3:23 AM, Anton Vodonosov <avodonosov at yandex.ru> wrote:
>
>>  Not yet, because I am in the beginning of the project and only
>>  starting to see the first results (it's not really a buildfarm, but
>>  a small system where anyone may run a simple command, like
>>  (load "agent.lisp"), and it runs the tests and submits the results
>>  to the central server; so that we will be able to accumulate
>>  the statictics; currently this command just runs tests on
>>  whatever is the current quicklisp distribution is installed
>>  on the system, and I am the only person who runs this command
>>  - I am testing how it works and polishing the corner cases).
>
> OK, cool stuff. Have you considered using something like Jenkins?

Jenkins is an interesting software (thanks for the reference BTW), but
it (and other continuous integration servers) has not so much overlap 
with what I do.

For Jenkins to run tests we need to provide a command "run tests".
Out of box it will be unable to run CL tests. So it's the first thing
I do - unify the interface of running test suites of different CL
libraries.

Also it is important to represent results so that we can see
side by side results of a particular library on different CL
implementations; or results of all the libraries on different
quicklisp distributions and so on. Continuous integration
servers will not provide such a representation.

Another goal is that people can contribute test results for
their platforms. This will save us from dependency
and maintenance of some central build farm with
all the possible combinations of OS and CL implementations.

So, while continue integration servers main function is to 
run some command - "run tests" - on some schedule
or by evens like source control commits. 

In my opinion scheduling is not the main task to solve. 
With cl-test-grid I go bottom-up by providing an easy way to 
run tests on any platform and share results. 

If we have several contributors who agree to run a simple 
command manually say once a month, it should be enough 
(for the beginning at least) to monitor the CL landscape 
and detect regressions. 

After this will work, probably someone will want to configure a cron job 
for this command and don't spend any attention on this at alll,
but on some platforms there might be limitations which will require
starting tests manually. Today it's a bit early to think about this.

Best regards,
- Anton