[Ecls-list] [cl-test-grid] cl-test-grid incremental reports

[Anton Vodonosov]

Hi.

Fist about the things that look strange now.

21.12.2012, 01:30, "Juan Jose Garcia-Ripoll" <juanjose.garciaripoll at gmail.com>:
> Indeed. There are reappearances which are some mysterious and I believe there is some non-reproducibility of some results. 

Non-reproducibility is quite rare. If test case uses random values, or written incorrectly.

> And there are sometimes when I can hardly interpret the tables generated. 
> See for instance http://ecls.sourceforge.net/reports-generated/ecl/ecl-diff-lisp-to-c-20121207231628.html 
> which comes right after http://ecls.sourceforge.net/reports-generated/ecl/ecl-diff-lisp-to-c-20121206231812.html
> I have the impression that the code that generates the tables is still somehow broken.

I reviewed this more carefully and now I understand what it is. 
ECL 5b7258e8 has the only difference ECL ab933fa5 - additional failed test case of usocked.
Here are the usocket logs of these ECLs:
http://cl-test-grid.appspot.com/blob?key=860701
http://cl-test-grid.appspot.com/blob?key=882267

This table has only single row because it is build by first computing set-exclusive-or between test
results of two ECL versions, and then laying-out the result into a table. 
Each failed test case is are separate result item, therefore set-exclusive-or returns a set of a single
element - the failed usocket test case on 5b7258e8.

> There are also sometimes when cl-test-grid reports OK and this is not so.
> See http://cl-test-grid.appspot.com/blob?key=854116 I believe this is another source of "regressions"

cl-test-grid works correnctly here, that's a problem of this particular system vs Quicklisp.
I've build this version of ECL and tried (ql:quickload :buildnode).
The result is 

;; Error while compiling /home/testgrid/quicklisp/dists/quicklisp/software/slime-20121125-cvs/swank-ecl.lisp:
;;   COMPILE-FILE returned NIL.
;; Aborting.
;;
(:BUILDNODE)

No errors is signaled. This happens because :buildnode depens on :swank. swank.asd uses
swank-loader::init, which calls (abort) in case compilation is failed. But ql:quicload installs
ABORT restart wich silently aborts the system load, without signalling any errors.

I've asked in Quicklisp mailing list how to think about this case:
https://groups.google.com/forum/?fromgroups=#!topic/quicklisp/rbyog9Hn7xY


Now about the testing procedure.

It only makes sence to compare test resutls when we change only in one dimension of environment.

If we compare two ECL versions, then we keep the same quicklisp.
If we what to compare two quicklisp versions, we consider only the same versions of lisp implementations.

When I tested ABCL, I had "base version" as I described previously. 
When new quicklis is out, I've retested old release, the "base version" and ensured
the absence of regressions holds on the new quicklisp too.

This means in such testing we need 3 binaries available:
- old release
- the base version
- current head

I will try perform this procedure running tests from my machine. Will first compare 12.12.1 with the current head.
If you will like how it works, we could then setup it for you.

Best regards,
- Anton