[Ecls-list] [cl-test-grid] cl-test-grid incremental reports

[Anton Vodonosov]

29.11.2012, 01:21, "Juan Jose Garcia-Ripoll" <juanjose.garciaripoll at gmail.com>:
> On Mon, Nov 26, 2012 at 11:06 PM, Anton Vodonosov <avodonosov at yandex.ru> wrote:
>> I see that according reports situation is not always improving with commits.
>
> It turns out that the code I used for sorting reports was broken! Now I fixed the code that separates dates from file names and the comparison is way better :-)
>
> http://ecls.sourceforge.net/reports-generated/ecl/index.html

I look at the reports and see that changes between two consecutive versions
sometimes have regressions, sometimes things are improved.

But it's difficult for me to see the overall direction of quality change.
How do you monitor it?

Consider the following approach.

We chose some "base version", and compare all the next versions
to it. Initially the base-version may be the previous release.

If the next tested versions have regressions comparing to the base-version,
the regressions are noticed and planned to fix.

As soon as we have some development version without regressions
and maybe with improvements comparing to the base-version, we start
to use this new version as the base-version - all the further versions
are compared to it.

I.e. as soon as we reach new level of quality - next dev version without
regressios but with improvements - we capture this quality level by shifting
base version to it, and do not allow the quality go back.
Thus trying to keep quality a non-decreasing function of time.

I've have tried this approach when monitored the ABCL development
towards the new release. Initially I compared all the deve versions to the
previous release. Later, as revision 14200 of ABCL has only improvements
comparing with the prev release, I compared all the further versions with this revision.

If you want, I can help you to setup this procedure for ECL.

I guess it will be some variable *base-version* in your reporting script, and
another column in the table, which compares the current version with the *base-version*.

Probably the table may be extended also with columns for bytecode version of ECL.

P.S.
I've fixed small bug in test-grid-reporting code, so that this report will have two columns:
http://ecls.sourceforge.net/reports-generated/ecl/ecl-diff-lisp-to-c-20121207231628.html

You may want to git pull to get this fix.