[Ecls-list] [cl-test-grid] cl-test-grid incremental reports

[Anton Vodonosov]

07.01.2013, 02:23, "Juan Jose Garcia-Ripoll" <juanjose.garciaripoll at gmail.com>:
> Dear Anton, thanks again for your work on this and for the good news.. It seems your manual comparison of versions works better than the automated process I tried to construct. Would it be very difficult to set up your process in a more robust manner than mine?
>
> Best,
>
> Juanjo


Hello Juanjo.

Sorry it took long to reply.

I want to say that the approach is not so manual.

I run few commands (git pull, ecl-rebuild.sh, run-agent.sh)
and computer tests 800+ libraries on the new ECL.

Then I run another command from slime and have a
comparison of these results with results
of the old version.

I would say it's an automated system, but the user
interface consists of a lisp script opened in emacs,
where I run test-grid-reporting::print-compiler-diff
on desired versions (or print all results, or print
load failures sorted by number of blocked dependencies)

I think this level of automation is the most convenient.

We may add some more automation, for example you already
have a setup where new commits initiate a test run.
But IMHO it's not the biggest priority, because
an automated system will require coding, debugging,
modification and maintenance by itself, and will disctruct
you from doing these activities on ECL. And will add
relatevely little convenience, because starting run-agent.sh
manually is easy.

What is not so easy, but really deserves time and attention
is reviewing the difference between two versions, analyzing
whether the regressions are caused by ECL or it is a problem
in environment/infrastructure. Then fixing ECL it it was a bug.
This can not be automated.

Example of what I mean by infrustructure problem is
https://github.com/quicklisp/quicklisp-client/issues/71.
It lays in inter-operation between Quicklisp, CCL and ECL:
if during quicklisp:update-all-dists performed by CCL some ASDF
systems with dot in file name were updated, then ECL can not
load these systems, because CCL escaped dots in pathname
when storing the pathname in quicklisp system file.
I encountered this when testing ECL recently, and it took some time
to investigate. Unfortunately, this kind of problems is unavoidable,
they will happen sometimes.

To minimize your efforts I believe it is not necessary to test
every commit, it's enough to run tests once in a several weeks
or in a month.
   
So, after the above general thoughts, the concrete actions I propose
to do:

1. The key point I suggest to change is to not compare each version
   with it's predecessor, but instead compare HEAD with
   the last version reviewed and verified to not have regressions
   (what I call "base version" in previous emails).

   I propose you to start doing it manually.
   
   If you do this kind of testing, you will end up having 3 ECL
   binaries on your machine: HEAD, base version and previous release.
   Because when new Quicklisp is out, it's necessary to retest
   not only HEAD, but also the base version and previous release
   so that ECL test results are compared on the same Quicklisp.

4. Submit test results to online storage instead of local-result-store
   (i.e. using test-grid-storage module with storage name "ecl")
   This will make the results available for me too.
   Please send me your run-agent.lisp file, I will show how to change it.

3. Could you share the scripts which build this page:
   http://ecls.sourceforge.net/reports-generated/ecl/index.html
   I would like to see how you determine order between
   ECL version - just by date or somehow else. If we can
   determine HEAD automatically, then probably it's
   not difficult to include HEAD/base-version comparison
   into the automated process.

Best regards,
- Anton

PS

Below is the script I currently have in slime session when analyze ECL test results.
I just change the version numbers, execute the functions calls I want and review HTML:

(ql:quickload :test-grid-storage)
(ql:quickload :test-grid-reporting)

(defpackage #:ecl-reports
  (:use :cl)
  (:import-from #:test-grid-reporting
                #:lisp
                #:libname
                #:subset
                #:lib-world
                #:results-cell-printer))
(in-package #:ecl-reports)

(defparameter *db-snapshot-file* (asdf:system-relative-pathname :test-grid-storage
                                                                "db-ecl.lisp"))
(defparameter *r* (test-grid-storage:make-replica "ecl" *db-snapshot-file*))
(test-grid-storage:sync *r*)

(defparameter *all-results* (test-grid-reporting::list-results (test-grid-storage:data *r*)))

(test-grid-reporting::print-compiler-diff "ecl/ecl-diff-bytecode.html"
                                          *all-results*
                                          "quicklisp 2012-12-23"
                                          "ecl-12.12.1-ebafa5f2-linux-x86-bytecode"
                                          "ecl-12.12.1-89a8201b-linux-x86-bytecode")

(test-grid-reporting::print-compiler-diff "ecl/ecl-diff-lisp-to-c.html"
                                          *all-results*
                                          "quicklisp 2012-12-23"
                                          "ecl-12.12.1-b6519e5c-linux-x86-lisp-to-c"
                                          "ecl-12.12.1-89a8201b-linux-x86-lisp-to-c")


(test-grid-reporting::print-pivot "ecl/ecl-results.html"
                                  (subset *all-results*
                                          (lambda (result)
                                            (and (string= "quicklisp 2012-12-23" (lib-world result))
                                                 (some (alexandria:rcurry #'search (lisp result))
                                                       (list "12.12.1-unknown" "12.12.1-ebafa5f2" "12.12.1-89a8201b")))))
                                  :rows '((libname string<))
                                  :cols '((lib-world string>) (lisp string<))
                                  :cell-printer #'results-cell-printer)