Running tests with test coverage enabled

[Alexey Veretennikov]

Hi,

I'm trying to understand what you are saying :)
Right now I'm having a problem what

(let ((asdf/output-translations:*output-translations*
           '(((#1=#P"/Users/alexeyv/.cache/common-lisp/lw-7.0.0-macosx-x86/**/profile-*.*" T) (T #1#)))))
      (asdf/operate:load-system :mylibrary))

would like to recompile all the dependent libraries as well with the new
output translations (I'll fix paths later).
Is where a way to load exactly this system I want to load, without even
trying to load everything which it depends on?


Faré <fahree at gmail.com> writes:

> 1- Calling clear-system from within perform is wrong.
> 2- I still recommend redirecting the output-translations for that
> particular tree around the load-system with coverage, and dropping the
> :force t.
>
> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
> "I believe that sex is one of the most beautiful, natural, wholesome things
> that money can buy." — Steve Martin
>
> On Tue, Oct 18, 2016 at 6:21 PM, Alexey Veretennikov
> <txm.fourier at gmail.com> wrote:
>> Hi,
>>
>> I've done something slightly more brute-force and probably not elegant.
>> I've modified the test asd file for my library in the following way (see
>> below)
>>
>> Basically I've tried to introduce the same operation as test-op. This
>> helps me to keep dependencies loaded/compiled via quicklisp. But it will
>> 1) Turn on the coverage
>> 2) Force reload of test subject with load-op and :force t
>> 3) Run tests
>> 4) Dump coverage results and [optionally] open browser with them.
>>
>> Please let me know if it could be done in an easier/more elegant way or
>> if there are any drawbacks of this approach.
>>
>> #|
>>   Test package.
>>   Usage:
>>   (ql:quickload :mylib-api-test)
>>   (asdf/operate:test-system :mylib-api)
>>
>>   In order to perform code coverage with these tests (currenty supported
>>   only on LispWorks 7), run the following:
>>   (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test)
>> |#
>>
>>
>>
>> (in-package :cl-user)
>> (defpackage mylib-api-test-asd
>>   (:use :cl :asdf)
>>   (:export coverage-op))
>>
>> (in-package :mylib-api-test-asd)
>>
>>
>> (defclass coverage-op (selfward-operation)
>>   ((selfward-operation :initform 'load-op :allocation :class))
>>   (:documentation "Test coverage operation"))
>>
>>
>> (defsystem mylib-api-test
>>   :author "Alexey Veretennikov"
>>   :license "BSD"
>>   :depends-on (:mylib-api
>>                :cl-fad
>>                :flexi-streams
>>                :prove)
>>   :components ((:module "t"
>>                 :components
>>                 ((:file "base")
>>                  (:test-file "pack-test")
>>                  (:test-file "utils-test"))))
>>   :description "Test system for mylib-api"
>>
>>   :defsystem-depends-on (:prove-asdf)
>>   :perform (test-op :after (op c)
>>                     (funcall (intern #.(string :run-test-system) :prove-asdf) c)
>>                     (asdf:clear-system c))
>>   :perform (coverage-op (op c)
>>                         (run-tests-with-coverage)
>>                         (asdf:clear-system c)))
>>
>>
>> #+lispworks7
>> (defun generate-coverage-output-path ()
>>   (multiple-value-bind (second minute hour date month year);; day)
>>       (get-decoded-time)
>>     (let ((results-directory-name
>>            (pathname
>>             (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html"
>>                     year month date hour minute second))))
>>
>>       (merge-pathnames results-directory-name (hcl:get-temp-directory)))))
>>
>>
>> #+lispworks7
>> (defun run-lw-test-coverage ()
>>   (hcl:clear-code-coverage)
>>   (hcl:with-code-coverage-generation ()
>>     (asdf/operate:load-system :mylib-api :force t))
>>   (asdf/operate:test-system :mylib-api-test)
>>   (let ((output-file (generate-coverage-output-path)))
>>     (hcl:code-coverage-data-generate-coloring-html output-file)
>>     (format *standard-output* "Generated coverage report to ~a" output-file)
>>     #+macosx
>>     (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:"
>>                  (objc:invoke "NSURL" "URLWithString:"
>>                               (concatenate 'string "file://" (namestring output-file))))))
>>
>>
>> (defun run-tests-with-coverage ()
>>   #+lispworks7
>>   (run-lw-test-coverage)
>>   #-lispworks7
>>   (error "Code coverage generation currently supported only on LispWorks 7 and above"))
>>
>>
>>
>> Faré <fahree at gmail.com> writes:
>>
>>> I recommend that you write run tests, etc., in a separate process, as
>>> orchestrated by a script that just after it loads ASDF
>>> 1- loads all the library code for which you do NOT want test coverage
>>> 2- turns on coverage
>>> 3- configures the asdf-output-translations to redirect object files
>>> for those systems that you DO want coverage (and only those) to an
>>> alternate location
>>> 4- load the rest of the code
>>> 5- runs the test
>>>
>>> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
>>> Reasons for existence are usually provided for things that don't exist;
>>> they would be wasted on things which do. — Saul Gorn
>>>
>>>
>>> On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov
>>> <txm.fourier at gmail.com> wrote:
>>>> Anyone ? Is impossible to do or too hard?
>>>> I see it as following:
>>>> make a new operation which will:
>>>> 1) compile the test cases
>>>> 1) clean the test object (remove fasls of target system to test but not
>>>> dependencies)
>>>> 2) execute necessary startup code (set up the code coverage)
>>>> 3) run testcases
>>>> 4) perform teardown code (turn off the code coverage and collect stats etc)
>>>>
>>>> How could I achieve at least this?
>>>>
>>>> Alexey Veretennikov <txm.fourier at gmail.com> writes:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Right now I'm running unit tests using Fukamachi's prove library:
>>>>> (asdf/operate:test-system 'my-system).
>>>>>
>>>>> I want to run my tests generating the tests coverage of my system.
>>>>>
>>>>> For this I would like to have similar operation, but which will:
>>>>> 1) turn on the code coverage in LispWorks (just call to some global
>>>>> function)
>>>>> 2) rebuild system which I want to test (in this case my-system), but
>>>>> only (!) this system (not dependencies)
>>>>> 3) run tests
>>>>> 4) call coverage results processing function.
>>>>> 5) on normal run (asdf/operate:test-system 'my-system) rebuild the
>>>>> system without coverage.
>>>>>
>>>>> How could I proceed with this task?
>>>>
>>>> --
>>>> Br,
>>>> /Alexey
>>>>
>>
>> --
>> Br,
>> /Alexey

-- 
Br,
/Alexey