Running tests with test coverage enabled

[Faré]

1- Calling clear-system from within perform is wrong.
2- I still recommend redirecting the output-translations for that
particular tree around the load-system with coverage, and dropping the
:force t.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
"I believe that sex is one of the most beautiful, natural, wholesome things
that money can buy." — Steve Martin

On Tue, Oct 18, 2016 at 6:21 PM, Alexey Veretennikov
<txm.fourier at gmail.com> wrote:
> Hi,
>
> I've done something slightly more brute-force and probably not elegant.
> I've modified the test asd file for my library in the following way (see
> below)
>
> Basically I've tried to introduce the same operation as test-op. This
> helps me to keep dependencies loaded/compiled via quicklisp. But it will
> 1) Turn on the coverage
> 2) Force reload of test subject with load-op and :force t
> 3) Run tests
> 4) Dump coverage results and [optionally] open browser with them.
>
> Please let me know if it could be done in an easier/more elegant way or
> if there are any drawbacks of this approach.
>
> #|
>   Test package.
>   Usage:
>   (ql:quickload :mylib-api-test)
>   (asdf/operate:test-system :mylib-api)
>
>   In order to perform code coverage with these tests (currenty supported
>   only on LispWorks 7), run the following:
>   (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test)
> |#
>
>
>
> (in-package :cl-user)
> (defpackage mylib-api-test-asd
>   (:use :cl :asdf)
>   (:export coverage-op))
>
> (in-package :mylib-api-test-asd)
>
>
> (defclass coverage-op (selfward-operation)
>   ((selfward-operation :initform 'load-op :allocation :class))
>   (:documentation "Test coverage operation"))
>
>
> (defsystem mylib-api-test
>   :author "Alexey Veretennikov"
>   :license "BSD"
>   :depends-on (:mylib-api
>                :cl-fad
>                :flexi-streams
>                :prove)
>   :components ((:module "t"
>                 :components
>                 ((:file "base")
>                  (:test-file "pack-test")
>                  (:test-file "utils-test"))))
>   :description "Test system for mylib-api"
>
>   :defsystem-depends-on (:prove-asdf)
>   :perform (test-op :after (op c)
>                     (funcall (intern #.(string :run-test-system) :prove-asdf) c)
>                     (asdf:clear-system c))
>   :perform (coverage-op (op c)
>                         (run-tests-with-coverage)
>                         (asdf:clear-system c)))
>
>
> #+lispworks7
> (defun generate-coverage-output-path ()
>   (multiple-value-bind (second minute hour date month year);; day)
>       (get-decoded-time)
>     (let ((results-directory-name
>            (pathname
>             (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html"
>                     year month date hour minute second))))
>
>       (merge-pathnames results-directory-name (hcl:get-temp-directory)))))
>
>
> #+lispworks7
> (defun run-lw-test-coverage ()
>   (hcl:clear-code-coverage)
>   (hcl:with-code-coverage-generation ()
>     (asdf/operate:load-system :mylib-api :force t))
>   (asdf/operate:test-system :mylib-api-test)
>   (let ((output-file (generate-coverage-output-path)))
>     (hcl:code-coverage-data-generate-coloring-html output-file)
>     (format *standard-output* "Generated coverage report to ~a" output-file)
>     #+macosx
>     (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:"
>                  (objc:invoke "NSURL" "URLWithString:"
>                               (concatenate 'string "file://" (namestring output-file))))))
>
>
> (defun run-tests-with-coverage ()
>   #+lispworks7
>   (run-lw-test-coverage)
>   #-lispworks7
>   (error "Code coverage generation currently supported only on LispWorks 7 and above"))
>
>
>
> Faré <fahree at gmail.com> writes:
>
>> I recommend that you write run tests, etc., in a separate process, as
>> orchestrated by a script that just after it loads ASDF
>> 1- loads all the library code for which you do NOT want test coverage
>> 2- turns on coverage
>> 3- configures the asdf-output-translations to redirect object files
>> for those systems that you DO want coverage (and only those) to an
>> alternate location
>> 4- load the rest of the code
>> 5- runs the test
>>
>> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
>> Reasons for existence are usually provided for things that don't exist;
>> they would be wasted on things which do. — Saul Gorn
>>
>>
>> On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov
>> <txm.fourier at gmail.com> wrote:
>>> Anyone ? Is impossible to do or too hard?
>>> I see it as following:
>>> make a new operation which will:
>>> 1) compile the test cases
>>> 1) clean the test object (remove fasls of target system to test but not
>>> dependencies)
>>> 2) execute necessary startup code (set up the code coverage)
>>> 3) run testcases
>>> 4) perform teardown code (turn off the code coverage and collect stats etc)
>>>
>>> How could I achieve at least this?
>>>
>>> Alexey Veretennikov <txm.fourier at gmail.com> writes:
>>>
>>>> Hi all,
>>>>
>>>> Right now I'm running unit tests using Fukamachi's prove library:
>>>> (asdf/operate:test-system 'my-system).
>>>>
>>>> I want to run my tests generating the tests coverage of my system.
>>>>
>>>> For this I would like to have similar operation, but which will:
>>>> 1) turn on the code coverage in LispWorks (just call to some global
>>>> function)
>>>> 2) rebuild system which I want to test (in this case my-system), but
>>>> only (!) this system (not dependencies)
>>>> 3) run tests
>>>> 4) call coverage results processing function.
>>>> 5) on normal run (asdf/operate:test-system 'my-system) rebuild the
>>>> system without coverage.
>>>>
>>>> How could I proceed with this task?
>>>
>>> --
>>> Br,
>>> /Alexey
>>>
>
> --
> Br,
> /Alexey