Running tests with test coverage enabled

[Alexey Veretennikov]

Hi,

I've done something slightly more brute-force and probably not elegant.
I've modified the test asd file for my library in the following way (see
below)

Basically I've tried to introduce the same operation as test-op. This
helps me to keep dependencies loaded/compiled via quicklisp. But it will
1) Turn on the coverage
2) Force reload of test subject with load-op and :force t
3) Run tests
4) Dump coverage results and [optionally] open browser with them.

Please let me know if it could be done in an easier/more elegant way or
if there are any drawbacks of this approach.

#|
  Test package.
  Usage:
  (ql:quickload :mylib-api-test)
  (asdf/operate:test-system :mylib-api)

  In order to perform code coverage with these tests (currenty supported
  only on LispWorks 7), run the following:
  (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test)
|#



(in-package :cl-user)
(defpackage mylib-api-test-asd
  (:use :cl :asdf)
  (:export coverage-op))

(in-package :mylib-api-test-asd)


(defclass coverage-op (selfward-operation)
  ((selfward-operation :initform 'load-op :allocation :class))
  (:documentation "Test coverage operation"))


(defsystem mylib-api-test
  :author "Alexey Veretennikov"
  :license "BSD"
  :depends-on (:mylib-api
               :cl-fad
               :flexi-streams
               :prove)
  :components ((:module "t"
                :components
                ((:file "base")
                 (:test-file "pack-test")
                 (:test-file "utils-test"))))
  :description "Test system for mylib-api"

  :defsystem-depends-on (:prove-asdf)
  :perform (test-op :after (op c)
                    (funcall (intern #.(string :run-test-system) :prove-asdf) c)
                    (asdf:clear-system c))
  :perform (coverage-op (op c)
                        (run-tests-with-coverage)
                        (asdf:clear-system c)))


#+lispworks7
(defun generate-coverage-output-path ()
  (multiple-value-bind (second minute hour date month year);; day)
      (get-decoded-time)
    (let ((results-directory-name
           (pathname
            (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html"
                    year month date hour minute second))))
      
      (merge-pathnames results-directory-name (hcl:get-temp-directory)))))

  
#+lispworks7
(defun run-lw-test-coverage ()
  (hcl:clear-code-coverage)
  (hcl:with-code-coverage-generation ()
    (asdf/operate:load-system :mylib-api :force t))
  (asdf/operate:test-system :mylib-api-test)
  (let ((output-file (generate-coverage-output-path)))
    (hcl:code-coverage-data-generate-coloring-html output-file)
    (format *standard-output* "Generated coverage report to ~a" output-file)
    #+macosx
    (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:"
                 (objc:invoke "NSURL" "URLWithString:"
                              (concatenate 'string "file://" (namestring output-file))))))


(defun run-tests-with-coverage ()
  #+lispworks7
  (run-lw-test-coverage)
  #-lispworks7
  (error "Code coverage generation currently supported only on LispWorks 7 and above"))



Faré <fahree at gmail.com> writes:

> I recommend that you write run tests, etc., in a separate process, as
> orchestrated by a script that just after it loads ASDF
> 1- loads all the library code for which you do NOT want test coverage
> 2- turns on coverage
> 3- configures the asdf-output-translations to redirect object files
> for those systems that you DO want coverage (and only those) to an
> alternate location
> 4- load the rest of the code
> 5- runs the test
>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> Reasons for existence are usually provided for things that don't exist;
> they would be wasted on things which do. — Saul Gorn
>
>
> On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov
> <txm.fourier at gmail.com> wrote:
>> Anyone ? Is impossible to do or too hard?
>> I see it as following:
>> make a new operation which will:
>> 1) compile the test cases
>> 1) clean the test object (remove fasls of target system to test but not
>> dependencies)
>> 2) execute necessary startup code (set up the code coverage)
>> 3) run testcases
>> 4) perform teardown code (turn off the code coverage and collect stats etc)
>>
>> How could I achieve at least this?
>>
>> Alexey Veretennikov <txm.fourier at gmail.com> writes:
>>
>>> Hi all,
>>>
>>> Right now I'm running unit tests using Fukamachi's prove library:
>>> (asdf/operate:test-system 'my-system).
>>>
>>> I want to run my tests generating the tests coverage of my system.
>>>
>>> For this I would like to have similar operation, but which will:
>>> 1) turn on the code coverage in LispWorks (just call to some global
>>> function)
>>> 2) rebuild system which I want to test (in this case my-system), but
>>> only (!) this system (not dependencies)
>>> 3) run tests
>>> 4) call coverage results processing function.
>>> 5) on normal run (asdf/operate:test-system 'my-system) rebuild the
>>> system without coverage.
>>>
>>> How could I proceed with this task?
>>
>> --
>> Br,
>> /Alexey
>>

-- 
Br,
/Alexey