Running tests with test coverage enabled

[Faré]

Your output translations rule is non-sensical. Please read the documentation.
You need to only translate things under mylibrary/

Actually, redirecting using output-translations will make things work,
but is not the right thing to do. The right thing to do is to define
operations compile-coverage-op, load-coverage-op, prepare-coverage-op
and test-coverage-op that do the right thing, maybe inheriting from
compile-op, load-op, prepare-op, test-op (respectively). Then the
input-files and output-files for these actions can do the right thing
and add a suffix to the file name, etc.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Give up all hope for a better yesterday, even a better just now.
Never give up hope for a better tomorrow. — Patri Friedman


On Wed, Oct 19, 2016 at 5:34 PM, Alexey Veretennikov
<txm.fourier at gmail.com> wrote:
> Hi,
>
> I'm trying to understand what you are saying :)
> Right now I'm having a problem what
>
> (let ((asdf/output-translations:*output-translations*
>            '(((#1=#P"/Users/alexeyv/.cache/common-lisp/lw-7.0.0-macosx-x86/**/profile-*.*" T) (T #1#)))))
>       (asdf/operate:load-system :mylibrary))
>
> would like to recompile all the dependent libraries as well with the new
> output translations (I'll fix paths later).
> Is where a way to load exactly this system I want to load, without even
> trying to load everything which it depends on?
>
>
> Faré <fahree at gmail.com> writes:
>
>> 1- Calling clear-system from within perform is wrong.
>> 2- I still recommend redirecting the output-translations for that
>> particular tree around the load-system with coverage, and dropping the
>> :force t.
>>
>> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
>> "I believe that sex is one of the most beautiful, natural, wholesome things
>> that money can buy." — Steve Martin
>>
>> On Tue, Oct 18, 2016 at 6:21 PM, Alexey Veretennikov
>> <txm.fourier at gmail.com> wrote:
>>> Hi,
>>>
>>> I've done something slightly more brute-force and probably not elegant.
>>> I've modified the test asd file for my library in the following way (see
>>> below)
>>>
>>> Basically I've tried to introduce the same operation as test-op. This
>>> helps me to keep dependencies loaded/compiled via quicklisp. But it will
>>> 1) Turn on the coverage
>>> 2) Force reload of test subject with load-op and :force t
>>> 3) Run tests
>>> 4) Dump coverage results and [optionally] open browser with them.
>>>
>>> Please let me know if it could be done in an easier/more elegant way or
>>> if there are any drawbacks of this approach.
>>>
>>> #|
>>>   Test package.
>>>   Usage:
>>>   (ql:quickload :mylib-api-test)
>>>   (asdf/operate:test-system :mylib-api)
>>>
>>>   In order to perform code coverage with these tests (currenty supported
>>>   only on LispWorks 7), run the following:
>>>   (asdf/operate:operate 'mylib-api-test-asd::coverage-op :mylib-api-test)
>>> |#
>>>
>>>
>>>
>>> (in-package :cl-user)
>>> (defpackage mylib-api-test-asd
>>>   (:use :cl :asdf)
>>>   (:export coverage-op))
>>>
>>> (in-package :mylib-api-test-asd)
>>>
>>>
>>> (defclass coverage-op (selfward-operation)
>>>   ((selfward-operation :initform 'load-op :allocation :class))
>>>   (:documentation "Test coverage operation"))
>>>
>>>
>>> (defsystem mylib-api-test
>>>   :author "Alexey Veretennikov"
>>>   :license "BSD"
>>>   :depends-on (:mylib-api
>>>                :cl-fad
>>>                :flexi-streams
>>>                :prove)
>>>   :components ((:module "t"
>>>                 :components
>>>                 ((:file "base")
>>>                  (:test-file "pack-test")
>>>                  (:test-file "utils-test"))))
>>>   :description "Test system for mylib-api"
>>>
>>>   :defsystem-depends-on (:prove-asdf)
>>>   :perform (test-op :after (op c)
>>>                     (funcall (intern #.(string :run-test-system) :prove-asdf) c)
>>>                     (asdf:clear-system c))
>>>   :perform (coverage-op (op c)
>>>                         (run-tests-with-coverage)
>>>                         (asdf:clear-system c)))
>>>
>>>
>>> #+lispworks7
>>> (defun generate-coverage-output-path ()
>>>   (multiple-value-bind (second minute hour date month year);; day)
>>>       (get-decoded-time)
>>>     (let ((results-directory-name
>>>            (pathname
>>>             (format nil "mylib-api-coverage_~4,'0d-~2,'0d-~2,'0d_~2,'0d_~2,'0d_~2,'0d/index.html"
>>>                     year month date hour minute second))))
>>>
>>>       (merge-pathnames results-directory-name (hcl:get-temp-directory)))))
>>>
>>>
>>> #+lispworks7
>>> (defun run-lw-test-coverage ()
>>>   (hcl:clear-code-coverage)
>>>   (hcl:with-code-coverage-generation ()
>>>     (asdf/operate:load-system :mylib-api :force t))
>>>   (asdf/operate:test-system :mylib-api-test)
>>>   (let ((output-file (generate-coverage-output-path)))
>>>     (hcl:code-coverage-data-generate-coloring-html output-file)
>>>     (format *standard-output* "Generated coverage report to ~a" output-file)
>>>     #+macosx
>>>     (objc:invoke (objc:invoke "NSWorkspace" "sharedWorkspace") "openURL:"
>>>                  (objc:invoke "NSURL" "URLWithString:"
>>>                               (concatenate 'string "file://" (namestring output-file))))))
>>>
>>>
>>> (defun run-tests-with-coverage ()
>>>   #+lispworks7
>>>   (run-lw-test-coverage)
>>>   #-lispworks7
>>>   (error "Code coverage generation currently supported only on LispWorks 7 and above"))
>>>
>>>
>>>
>>> Faré <fahree at gmail.com> writes:
>>>
>>>> I recommend that you write run tests, etc., in a separate process, as
>>>> orchestrated by a script that just after it loads ASDF
>>>> 1- loads all the library code for which you do NOT want test coverage
>>>> 2- turns on coverage
>>>> 3- configures the asdf-output-translations to redirect object files
>>>> for those systems that you DO want coverage (and only those) to an
>>>> alternate location
>>>> 4- load the rest of the code
>>>> 5- runs the test
>>>>
>>>> —♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
>>>> Reasons for existence are usually provided for things that don't exist;
>>>> they would be wasted on things which do. — Saul Gorn
>>>>
>>>>
>>>> On Sat, Oct 15, 2016 at 2:52 AM, Alexey Veretennikov
>>>> <txm.fourier at gmail.com> wrote:
>>>>> Anyone ? Is impossible to do or too hard?
>>>>> I see it as following:
>>>>> make a new operation which will:
>>>>> 1) compile the test cases
>>>>> 1) clean the test object (remove fasls of target system to test but not
>>>>> dependencies)
>>>>> 2) execute necessary startup code (set up the code coverage)
>>>>> 3) run testcases
>>>>> 4) perform teardown code (turn off the code coverage and collect stats etc)
>>>>>
>>>>> How could I achieve at least this?
>>>>>
>>>>> Alexey Veretennikov <txm.fourier at gmail.com> writes:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> Right now I'm running unit tests using Fukamachi's prove library:
>>>>>> (asdf/operate:test-system 'my-system).
>>>>>>
>>>>>> I want to run my tests generating the tests coverage of my system.
>>>>>>
>>>>>> For this I would like to have similar operation, but which will:
>>>>>> 1) turn on the code coverage in LispWorks (just call to some global
>>>>>> function)
>>>>>> 2) rebuild system which I want to test (in this case my-system), but
>>>>>> only (!) this system (not dependencies)
>>>>>> 3) run tests
>>>>>> 4) call coverage results processing function.
>>>>>> 5) on normal run (asdf/operate:test-system 'my-system) rebuild the
>>>>>> system without coverage.
>>>>>>
>>>>>> How could I proceed with this task?
>>>>>
>>>>> --
>>>>> Br,
>>>>> /Alexey
>>>>>
>>>
>>> --
>>> Br,
>>> /Alexey
>
> --
> Br,
> /Alexey