[Ecls-list] [cl-test-grid] cl-test-grid incremental reports

[Juan Jose Garcia-Ripoll]

On Thu, Dec 20, 2012 at 3:59 PM, Anton Vodonosov <avodonosov at yandex.ru>wrote:

> 29.11.2012, 01:21, "Juan Jose Garcia-Ripoll" <
> juanjose.garciaripoll at gmail.com>:
> > On Mon, Nov 26, 2012 at 11:06 PM, Anton Vodonosov <avodonosov at yandex.ru>
> wrote:
> >> I see that according reports situation is not always improving with
> commits.
> >
> > It turns out that the code I used for sorting reports was broken! Now I
> fixed the code that separates dates from file names and the comparison is
> way better :-)
> >
> > http://ecls.sourceforge.net/reports-generated/ecl/index.html
>
> I look at the reports and see that changes between two consecutive versions
> sometimes have regressions, sometimes things are improved.
>

Indeed. There are reappearances which are some mysterious and I believe
there is some non-reproducibility of some results. There is also the fact
that along the table a new quicklisp version was introduced and this broke
a lot of things. There was also another bug in cffi which broke further
things. In such circumstance the utility of the tests is questionable.

And there are sometimes when I can hardly interpret the tables generated.
See for instance
http://ecls.sourceforge.net/reports-generated/ecl/ecl-diff-lisp-to-c-20121207231628.htmlwhich
comes right after
http://ecls.sourceforge.net/reports-generated/ecl/ecl-diff-lisp-to-c-20121206231812.html
I have the impression that the code that generates the tables is still
somehow broken.

There are also sometimes when cl-test-grid reports OK and this is not so.
See http://cl-test-grid.appspot.com/blob?key=854116 I believe this is
another source of "regressions"


> But it's difficult for me to see the overall direction of quality change.
> How do you monitor it?
>

Right now I don't. The last month I have not monitored the cl-test-grid
tests because I have focused on open bug reports.


> I.e. as soon as we reach new level of quality - next dev version without
> regressios but with improvements - we capture this quality level by
> shifting
> base version to it, and do not allow the quality go back.
> Thus trying to keep quality a non-decreasing function of time.
>

In an ideal world, indeed this would be true. In practice I am finding that
the evolution of libraries introduces itself the possibility of going back.
Moreover, some bug solutions lead to regressions but in that situation I
cannot afford not to fix the bug.


> I've have tried this approach when monitored the ABCL development
> towards the new release. [...] If you want, I can help you to setup this
> procedure for ECL.
>

Help is always welcome.

Probably the table may be extended also with columns for bytecode version
> of ECL.
>

Right now this is not of the greatest priority to me. Getting quicklisp to
work with the compiled version is _the_ priority, specially when there are
many libraries that cannot work with the interpreter because of the
features they use.

P.S.
> I've fixed small bug in test-grid-reporting code, so that this report will
> have two columns:
>
> http://ecls.sourceforge.net/reports-generated/ecl/ecl-diff-lisp-to-c-20121207231628.html
> You may want to git pull to get this fix.
>

The reports do "git pull" every night.

Best

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/ecl-devel/attachments/20121220/c931afae/attachment.html>