[mcclim-devel] Test suites and code coverage

[Robert Goldman]

Christophe Rhodes wrote:
> Hi,
> 
> Some people may know that Juho Snellman committed a nice new tool to
> SBCL recently: a tool to measure code coverage[1].  For laughs, I
> thought I'd try it out on McCLIM and its test suite...
> 
> To do so, I first loaded clx, spatial-trees and flexichain.  Then
>   (require :sb-cover)
>   (declaim (optimize sb-cover:store-coverage-data))
>   (asdf:oos 'asdf:load-op :mcclim)
>   (sb-cover:reset-coverage) ; debatable.  See below.
>   (defpackage :clim-tests (:use :clim :clim-lisp))
>   (mapcar #'load (directory "/path/to/mcclim/Tests/*.lisp"))
> 
> The results I got are available at
> <http://doc.gold.ac.uk/~mas01cr/tmp/mcclim-tests/>.
> 
> So, firstly, this is clearly not automated enough; I plan to spend a
> little time making running tests easier.  The point about clearing
> coverage before running the tests is to remove all the uses of
> internal code (such as macros) in the building of mcclim itself.  I
> don't know whether that's fair or not, but in doing so I noticed that
> there are some load-time tests (in e.g. bezier.lisp) which might be
> better placed in a test suite.
> 
> I also am aware that I didn't run Drei or automaton test suites in the
> process of doing this; running those should be a simple matter.
> 
> What does this show?  Well, even though it's difficult to test a UI
> library in an automated fashion, I'm pretty sure that there are large
> swathes of things that /are/ testable in an automated fashion that
> aren't being so tested -- and that maybe by making these statistics
> available people could find bits that interest them and write tests to
> exercise those bits.
> 
> There's also the obvious possibility of collecting the coverage
> statistics while running demodemo -- which will exercise backend
> functions and some of the UI code, and show us where some gaps are.
> (There's also my long-standing idea of using the Null backend to test
> UI in an automated fashion, but I'm sure more work needs to be done to
> make that possible.)
> 
> Anyway.  Comments, ideas?
> 
These results are very thought provoking, but I was wondering --- is
there a source of information about what the counts mean somewhere?  I
have looked at the sb-cover manual section and Juho's page, and I don't
see a definition for these terms.  I am asking because I'm trying to
grope towards an understanding of how to use them.  For example, I see
in your statistics that graph-formatting.lisp (one of the few pieces of
McCLIM I can claim to somewhat understand), has 54.2% expression
coverage and 46.8% branch coverage.  I'm not sure how to relate that to
what I know, which is that graph-formatting is radically unfinished (it
supports only tree formatting, neither DAG nor directed graph formatting
options, at least as of about a month ago, which was when I last pulled
a CVS update).

Thanks,
r