[mcclim-devel] Test suites and code coverage

[Christophe Rhodes]

Hi,

Some people may know that Juho Snellman committed a nice new tool to
SBCL recently: a tool to measure code coverage[1].  For laughs, I
thought I'd try it out on McCLIM and its test suite...

To do so, I first loaded clx, spatial-trees and flexichain.  Then
  (require :sb-cover)
  (declaim (optimize sb-cover:store-coverage-data))
  (asdf:oos 'asdf:load-op :mcclim)
  (sb-cover:reset-coverage) ; debatable.  See below.
  (defpackage :clim-tests (:use :clim :clim-lisp))
  (mapcar #'load (directory "/path/to/mcclim/Tests/*.lisp"))

The results I got are available at
<http://doc.gold.ac.uk/~mas01cr/tmp/mcclim-tests/>.

So, firstly, this is clearly not automated enough; I plan to spend a
little time making running tests easier.  The point about clearing
coverage before running the tests is to remove all the uses of
internal code (such as macros) in the building of mcclim itself.  I
don't know whether that's fair or not, but in doing so I noticed that
there are some load-time tests (in e.g. bezier.lisp) which might be
better placed in a test suite.

I also am aware that I didn't run Drei or automaton test suites in the
process of doing this; running those should be a simple matter.

What does this show?  Well, even though it's difficult to test a UI
library in an automated fashion, I'm pretty sure that there are large
swathes of things that /are/ testable in an automated fashion that
aren't being so tested -- and that maybe by making these statistics
available people could find bits that interest them and write tests to
exercise those bits.

There's also the obvious possibility of collecting the coverage
statistics while running demodemo -- which will exercise backend
functions and some of the UI code, and show us where some gaps are.
(There's also my long-standing idea of using the Null backend to test
UI in an automated fashion, but I'm sure more work needs to be done to
make that possible.)

Anyway.  Comments, ideas?

Cheers,

Christophe

[1] Weblog entry at
<http://jsnell.iki.fi/blog/archive/2007-05-03-code-coverage-tool-for-sbcl.html>;
some details at <http://www.sbcl.org/manual/sb_002dcover.html>.