[Ecls-list] Release candidate?

[Anton Vodonosov]

23.07.2012, 17:30, "Juan Jose Garcia-Ripoll" <juanjose.garciaripoll at gmail.com>:
> Is it possible to reproduce your tests with my own platforms?

Yes, it is one of the primary goals to make it easy to run tests for everyone and to share the results.

If you are interested, I will provide you instructions somewhat later, because I am now
working on improvements of the test running program, and my work will affect the instructions,
so it's better to discuss it after I finish the current tasks.

As a short overview. I have a function libtest(libname). This function runs test suite
of the specified library, downloading it with quicklisp if necessary, and converts the test
result from the format of test framework of this particular library into a unified format.

On top of this there is a program called test-grid-agent. It's a lisp program, which
bootstraps private quicklisp in a local directory, runs each test suite in a separate lisp process,
collects test statuses and logs and uploads them admin (to me) for publishing.

I am now making some final changes in the agent and testing it.

> Would it also be possible to get one table per platform?

Customizing the reports would require some programming. Out of box
only reports described on this overview page are ready: 
http://common-lisp.net/project/cl-test-grid/.

The test result are stored in single lisp file [1] and based on the
current reporting code [2] it is probably not difficult to get the 
presentation you need.

[1] https://github.com/cl-test-grid/cl-test-grid/blob/master/db.lisp
[2] https://github.com/cl-test-grid/cl-test-grid/blob/master/reporting/reporting.lisp

In one of the next milestones I plan to improve the reports (simplify and document
the test results data access, add filtering and matching functions).
But I will switch to it after I will finish with the agent program. 

Could you explain what do you mean by one table per platform?
And in general I am interested to know what a compiler maintainer
would like to see in reports.

> Many of the libraries seem to have failed tests also on other ports.

Yes, there are lot of failures on other compilers.

> Would it be possible to distinguish between test failure and build failure? That would set up priorities straight. 

There is no separate test status for build failure (I saw on ECL test farm record build status and test status
separately and considered but haven't decided to introduce it yet).
There are cases when we can distinguish build failure in test-grid, but I don't want to overload you by the details now.

Today you have 4 ECL version x 56 libraries = 224 test results. Lot for work to review everything.
And we will add more libraries / more ECL versions.

I came to a conclusion that a better strategy is not to review every test result, but to concentrate
on regressions. We capture situation as it is, and then monitor how it changes. 
If yesterday libraryX reported 3 test cases failed, but today it has 4 test cases
failed, then we know someone's recent changes cause this and we can react. I.e. a strategy
of prevent the situation of moving into wrong direction. (BTW, the regression
detection code would be able to detect if a library yesterday had test failures,
but today it just doesn't build) 

Still, the short answer, you can't easily distinguish build failures in the test results.

Best regards,
- Anton