massive-oss / mcover

A cross platform code coverage framework for Haxe with testing and profiling applications. Supports AVM1, AVM2, JavaScript, C++, PHP and Neko.

MCover is a collection of macro based cross-platform code coverage and quality tools. MCover runs on all main Haxe targets including js, flash, neko, cpp and php.

To install mcover:

haxelib install mcover

MCover supports Haxe 2.10 and Haxe 3

Features

Code Coverage

MCover can provide detailed coverage of executed code, including:

• code blocks (statements)
• code branches

Example macro usage:

--macro mcover.MCover.coverage([''], ['src'])

Function logging

MCover can generate timing metrics around function entry/exit times, including:

• function start/exit time
• function duration (both inclusive and exclusive of nested methods)
• call stack depths

Example macro usage:

--macro mcover.MCover.logger([''], ['src'])

Cross Platform

MCover has been designed to work with any Haxe target. Officially we support the following:

• ActionScript
• JavaScript
• Neko
• CPP
• PHP

Code Coverage

For detailed information see src/m/cover/coverage/README.md

Compiler args

Add the following to your hxml file:

-lib mcover
--macro mcover.MCover.coverage(['{package}'], {classPaths}, {ignoredClasses})

Where:

• package is an array of packages to filter on (e.g. 'com.example'). Default is all packages - e.g. ['']
• classPaths is an array of classpaths to include (e.g. ['src']). Default is local path - e.g ['']
• ignoredClasses is an array of specific classes to ignore (e,g, ['com.example.IgnoredClass']). Default is null.

Example:

--macro mcover.MCover.coverage(['com.example'], ['src'], null)

Note: Only use single quotation marks (' ') to avoid compiler issues on windows platforms

Runtime report

Add the followng code to your application after code has executed:

var logger = mcover.coverage.MCoverage.getLogger();
logger.report();

Function Logging

For detailed information see src/m/cover/logger/README.md

Compiler args

Add the following to your hxml file:

-lib mcover
--macro mcover.MCover.logger(['{package}'], {classPaths}, {ignoredClasses})

Where:

• package is an array of packages to filter on (e.g. 'com.example'). Default is all packages - e.g. ['']
• classPaths is an array of classpaths to include (e.g. ['src']). Default is local path - e.g ['']
• ignoredClasses is an array of specific classes to ignore (e,g, ['com.example.IgnoredClass']). Default is null.

Example:

--macro mcover.MCover.logger(['com.example'], ['src'], null)

Note: Only use single quotation marks (' ') to avoid compiler issues on windows platforms

Runtime Usage

Add the followng code to your application to start recording

var logger = mcover.coverage.MCoverLogger.getLogger();
logger.startRecording();

Add the followng code to your application to stop recording and print report

var logger = mcover.coverage.MCoverLogger.getLogger();
logger.stopRecording();
logger.report();

Changes

See CHANGES for full changes

New since 2.0.0

• Haxe 3 support, includes some breaking changes to APIs to better align with Haxe 3
• Updated for Munit 2.0
• Removed support for Haxe 2.08 and 2.09

New since 1.5.x

• Improved handling of macro generated code blocks

New since 1.4.x

• Changed top level package from m.cover to mcover

New since 1.3.x

• Added cpp target support
• Added php target support
• Updated for Haxe 2.10

New since 1.2.x

• Added function logging macro

Building from source

OSX: Use the build.sh bash script (osx only) Windows: Manually run steps within build.sh

Note: Make sure to set the dev path of mcover to /src before running tests (it's a bit tricky running code coverage on itself!)

mcover's Issues

Generating html based coverage report for haxe source code for as3

Hi,
I ran code coverage with munit and got a report in html format. This would do well for unit test code.
But for actual haxe source code ( output is as3 swf), when I do coverage using MCover ,it simply gives the output to some swf as plain text.
I noticed that when doing coverage for cpp, it gives a output in console and I can redirect it to a text file. In this case I can have a script to generate html report from that text file. But how to do that for a swf file?
Is there a way to generate html based report similar to what munit does while running mcover on the source code? Are you planning to have any MCover release supporting this in the future?

How to ignore compilation of some files inside a classpath.

My haxe code has multiple files within a single class path folder."myexample/hxproj/common/src".

Since MCOVER expects classPath and NOT individual file names, how do I exclude some of the files from being compiled ?I cannot move out any file..have to just ignore some files in that classPath.
In the normal build I can use a MAKEFILE to add list of files I compile in the folder.
I do not want MCover to compile few files which are not in the MAKEFILE.

Please note that ignoreClass option in the MCOVER macro will not help here as it appears to still compile these classes and only ignore it when reporting the coverage.

Support ignoring coverage for concrete expressions.

We use a build macro for some of our classes, that inject some runtime checking expressions into a class method.

We test all expressions that can be generated by that macro in a separate test and would like these expressions to be ignored by MCover in all other classes.

It would be nice if MCover could ignore coverage at expression level by checking expression metadata. Then our build macro can add that meta for its expressions to make sure they are ignored.

Implement support for static methods

Single-letter classes are not supported

Subj.

Allow package wildcards

Right now, we can either get the coverage on all packages, or on a list of packages given as a parameter to the macro. This is not efficient when we want to filter to, for example, only one parent package, but all its sub-packages.

It should be possible to do something like --macro m.cover.MCover.coverage(['myPackage.*'],['src']).

Thanks for the amazing tool :)

CPP: 0% branch coverage prints -21474836.48%

This was observed in munit's example app when running coverage:

OVERALL COVERAGE STATS:

packages    100%       1 / 1      
files       100%       1 / 1      
classes     100%       1 / 1      
methods     100%       3 / 3      
statements  100%       4 / 4      
branches    -21474836.48%0 / 0      
lines       100%       21 / 21    

Backslashes missing in paths on Windows

The generated coverage report shown in a browser after "munit t -coverage" gives file paths like c:UsersGimmickyDocumentscoveragetestsourceblabla.hx

Extract reporting into Client interface and implement basic print client

New release process?

What needs to be done to get a new release to haxelib?

mcover breaks null-safety

Compiling with the macro enabled, I get this error: Null safety: Cannot cast nullable value to not nullable type

from this expression:

function myfunc(key:String):String {
    var tmp = mydict[key];
    if (tmp != null)
       return (tmp:String); // here is the line it complains about
    //...
}

The only way I can find to solve this is to disable null safety for the offending class, run mcover, then re-enable null safety

MCover fails if package has a space at the end before ';'

If a package has a space at the end: eg:

package this.my.package ;

mcover fails to create the report and appears some erros in the console.

I opened this issue because i got stuck with this problem twice and took me some time to figure out the problem.

I think flashdevelop appends a space before the ';' when classes are renamed, thats why it happened to me again.

Track code coverage in individual classes

Add coverage stats for individual classes

Coverage doesn't include classes contained in packages with underscore (_) in names.

Subj. That is because regexp in mcover.macro.ClassPathFilter.getPackageDefinitionInFile doesn't allow underscore. Easy to fix, just add _ to the group.

Deadlocks with mcover in C++ code

Hi,

I'm wondering about something better than using mutex on c++... I am trying to integrate mcover to a big multi-threaded project (haxe 3.1.3) and I get deadlocks as soon as there's a garbage collection happening.

Anyone had similar issues?

Thanks!

Running tests stuck with -coverage flag with flixel

After adding the -coverage flag to test command the execution stops.
Browser does not get opened and nothing gets logged.

Issue can be reproduced with default flixel template and no added code.

Is there any workaround for this?

Ignored Mockitoo-Mocks

Hi everyone,
I´ve got a little bug. We use MCover to get our CodeCoverage done and now I tried to use Mockatoo to mock and stub methods. When I write tests without Mockatoo I get the percentage of the covered code, but when I use mocks, it doesn´t. I can see inside MCover how many tests are written, to be seen by the dots, but not the percentage. Am I missing something?
Looking forward to hear from you.
Greetings

Methods using return expression (i.e. without braces) are not covered

I'm testing a Node.js application, and generating a LCOV report from the tests.
See repo: https://github.com/cedx/setup-dart

If a class has methods without braces and covered by tests, these methods don't appear in the generated coverage report.
See the DartSdk.download() method from file:
https://github.com/cedx/setup-dart/blob/main/src/setup_dart/DartSdk.hx#L52

Without braces (i.e. method signature + return ...), Coveralls reports a code coverage of 44.44 %, and the LCOV report looks like this:

TN:setup_dart.DartSdk
SF:C:\Projets\haxe\setup-dart\src\setup_dart\DartSdk.hx

FN:37,new
FN:44,get_releaseUrl
FN:61,install
FN:76,__init__

FNDA:0,new
FNDA:0,get_releaseUrl
FNDA:1,install
FNDA:1,__init__

FNF:4
FNH:3

With braces (i.e. method signature + { return ...; }), Coveralls reports a code coverage of 45.65 %, and the LCOV report looks like this:

TN:setup_dart.DartSdk
SF:C:\Projets\haxe\setup-dart\src\setup_dart\DartSdk.hx

FN:37,new
FN:44,get_releaseUrl
FN:57,download
FN:62,install
FN:77,__init__

FNDA:0,new
FNDA:0,get_releaseUrl
FNDA:1,download
FNDA:1,install
FNDA:1,__init__

FNF:5
FNH:4

Haxe : 4.1.0 (x64) on Windows 10
MCover : 2.2.2

Off topic: is there a way to customize the path of the mcover.MCover.TEMP_DIR folder?

Memory tracking

By far the most useful for me would be to add some allocation / deallocation information so I could track down memory leaks. I don't know if this is possible to do through Haxe Macros, but having a dump of which objects were created, which were destroyed, and which are still actively tracked by the GC would be a big help when tracking down memory leaks.

Top level `m` package clashes with local varables

The injected reference to m.cover.coverage.MCoverage causes compilation conflicts with local vars called m.

Solution is to refactor 'm.cover' to 'mcover' however this will be a breaking changes and will require an update to Munit as well.

Coverage Option Disables Expanding Classes, Icons

I've noticed that when I run:

haxelib run munit test -as3 -coverage

If I have failed tests, I can't click on the test classes to expand them and see the failure methods and line numbers. The javascript never activates; also, the pass and fail icons never appear.

If I remove the -coverage command, then I can click on failed classes to see method and line numbers, and I can see the pass/fail icons next to class names.

This is with Haxe 2.09, NME 3.3.3, and the latest MUnit as of yesterday.

0% coverage

Running munit with -coverage, I get an empty coverage report:

------------------------------
COVERAGE REPORT
------------------------------

OVERALL COVERAGE STATS:

    packages    0%         0 / 0
    files       0%         0 / 0
    classes     0%         0 / 0
    methods     0%         0 / 0
    statements  0%         0 / 0
    branches    0%         0 / 0
    lines       0%         0 / 0

COVERAGE RESULT     0%
==============================

My .munit settings:

classPaths=..
coveragePackages=com

Have also tried an absolute path for classPaths.

Doesn't seem to work?

How does it work? There's no code?! Are macros THAT magical?

Generating HTML Coverage for Neko

TLDR: Is there a way to generate an HTML report against Neko-built code, and see which lines of my classes aren't covered by any tests?

I'm working on Butterfly, which is a static website generator; I build against Neko (and only Neko) because I read/write a lot of files from the file system.

I'm at a loss about how to generate a proper HTML coverage report, like EMMA for Java (there's a sample report here).

What I really want is some command-line option (or client option) that generates a bunch of HTML files which allows me to drill down class by class and see what lines are not covered at all by any tests.

It doesn't seem like this is possible -- am I mistaken? The current RichPrintClient is the best thing so far, and doesn't generate this level of granularity. There's some sort of test-runner/index.html file generated, but it doesn't give me anything with my builds (presumably because I'm using Neko, and I may be mixing up MUnit and MCover reports here).

Circular dependencies with munit

It feels wrong that munit and mcover list each other as dependencies. If they are so tightly coupled, they should be a single library.
If those dependencies are optional, then it shouldn't be required to have one when requesting to install only the other one.

Coverage on abstract functions?

Hello, I have a project where the only Haxe type under src/ is an abstract.

However it appears that mcover does not count abstracts as types when calculating coverage?
Because even though all tests run (or fail) as they should, in the output I get the following:

----------------------------------------------------------------
OVERALL COVERAGE STATS:
    packages    0%         0 / 0      
    files       0%         0 / 0      
    classes     0%         0 / 0      
    methods     0%         0 / 0      
    statements  0%         0 / 0      
    branches    0%         0 / 0      
    lines       0%         0 / 0      
----------------------------------------------------------------
COVERAGE RESULT     0%         
----------------------------------------------------------------
utest/ui/text/PrintReport.hx:52: 
assertations: 20
successes: 20
errors: 0
failures: 0
warnings: 0
execution time: 0.004
results: ALL TESTS OK (success: true)
(unknown) : Warning : No classes match criteria in MCover macro:
(unknown) : ... 	packages: [],
(unknown) : ... 	classPaths: [/builds/haath/bytesize/src],
(unknown) : ... 	exclusions: []

Is this intended? Should abstract type functions not count for coverage?

• Repo
• The type
• Test main

Customizing coverage result client?

Hi there.

First of all thank you for munit and mcoverage ; I'm really glad that you guys brought CI tools for HaXe.

I have a question though ; I don't really understand how to add/customize the way coverage results output. I can see the details in the AS3 runner, but report.txt file only provides projet level information.

Also, I would like to try the EMMAPrintClient that you provide ; how to you add it (I tried MCoverage.getLogger().addClient but no luck). Is it functional?

Eventually I might need to write a Cobertura client, so clarifying how you can swap implementations in this particular area of the test runner settings would be nice.

Thanks

Always reporting 0% on cpp target

I am using a CPP target with the latest haxe version, munit and mcover form haxelib, I set up everything like the doc says and the coverage always shows 0% event if I know I am testing code from the package specified in mcover's config.

I can provide more info/sample code in case it is not a known bug or something easily reproducible on your side.

Include coverage breakdown in report

Possibly based on format from EMMA (Java code coverage tool)

http://emma.sourceforge.net/samples.html

E.g.

[EMMA v2.0.4015 (stable) report, generated Sat May 15 12:02:28 CDT 2004]

OVERALL COVERAGE SUMMARY:

[class, %] [method, %] [block, %] [line, %] [name]
85% (157/185)! 65% (1345/2061)! 60% (44997/74846)! 64% (8346.3/13135)! all classes

OVERALL STATS SUMMARY:

total packages: 8
total classes: 185
total methods: 2061
total executable files: 62
total executable lines: 13135

COVERAGE BREAKDOWN BY PACKAGE:

[class, %] [method, %] [block, %] [line, %] [name]
25% (1/4)! 25% (3/12)! 40% (3012/7446)! 25% (3/12)! com.sun.tools.javac.v8.resources
94% (16/17)! 49% (41/83)! 48% (1111/2292)! 45% (201.1/450)! com.sun.tools.javac.v8
88% (45/51)! 61% (242/397)! 54% (3070/5729)! 52% (809.6/1563)! com.sun.tools.javac.v8.tree
83% (19/23)! 60% (134/224)! 54% (2746/5063)! 56% (580.1/1041)! com.sun.tools.javac.v8.util
100% (1/1) 40% (2/5)! 58% (25/43)! 49% (5.9/12)! com.sun.tools.javac
77% (33/43)! 59% (310/529)! 60% (10584/17674)! 61% (2077.2/3396)! com.sun.tools.javac.v8.code
91% (39/43)! 75% (521/698) 66% (19701/29863)! 70% (3606.9/5138)! com.sun.tools.javac.v8.comp

100% (3/3) 81% (92/113) 70% (4748/6736)! 70% (1062.4/1523)! com.sun.tools.javac.v8.parser

Primitive not found std@mutex_create

This is my first time integrating mcover. I was able to get it running successfully in PHP and JS. But Neko throws me an error. This is with Haxe 3.2.

/usr/lib/haxe/std/neko/vm/Mutex.hx:27: characters 6-20 : Primitive not found std@mutex_create:0
<builtin>:1: character 0 : Called from
/usr/lib/haxe/std/neko/vm/Mutex.hx:24: lines 24-42 : Called from
/usr/lib/haxe/lib/mcover/2,1,1/mcover/coverage/MCoverage.hx:59: characters 29-40 : Called from
src/project/test/TestRunner.hx:9: characters 21-58 : Called from
src/project/test/TestRunner.hx:33: characters 8-18 : Called from
?:1: characters 1048575-8796094070782 : Called from
Aborted

Build instructions are:

-cp src
-lib mcover
--macro mcover.MCover.coverage(['project.view'], ['src'])
-main project.test.TestRunner
--interp

Test runner main function looks like:

        var logger = mcover.coverage.MCoverage.getLogger();  // Line: 9 mentioned in the stack trace

        var r = new haxe.unit.TestRunner();

        r.add(new project.view.test.TestVirtualizationTools());
        r.run();

        logger.report();

Issue migrating from NME to OpenFL

Hi guys, great work on the massive libs!

I have a build system (https://github.com/crayfellow/haxe-maven-plugin) that supports openfl and uses munit/mcover for testing. It insulates developers from having to install haxe, neko, nme, openfl, or any haxelibs into their local and allows them to centrally manage repositories using Maven.

I had testing and coverage working with NME, but now that I have migrated to openfl, when I enable code coverage I get the following error

if I dismiss or continue, the browser just hangs and never shows the test results. I do get test results when I disable coverage and just use munit.

This is using the haxelib version of mcover (2.0.3).

[Feature request] Add a LCOV coverage reporter

Currently MCoverage.getLogger().report() prints a coverage report on the standard output.

What I want is to be able to collect coverage data and send it to the Coveralls service.
So I need a way to get the coverage data instead of printing on the command line, and ideally have that coverage data formatted to LCOV (as this format is very easy to parse : see the FILES section of the provided link).

Thanks in advance.

Module sub-types are not supported.

Subj.

Listing Method Coverage Per Method on Neko

Hi,

On Neko, when I run haxelib run munit test -coverage, I get summary info such as:

OVERALL COVERAGE STATS:

    packages    100%       5 / 5      
Tests PASSED under neko using summary client
    files       100%       11 / 11    
    classes     100%       11 / 11    
    methods     82%        41 / 50    
    statements  64.74%     101 / 156  
    branches    32.76%     19 / 58    
    lines       65.97%     661 / 1002 

According to a README file buried deep in the tree, I can add coverage reporting for Neko by adding the following to TestMain.hx:

client.includeMissingBlocks = true;
client.includeExecutionFrequency = true;
client.includeClassAndPackageBreakdowns = true;

It's not clear to me if this change will be obliterated automatically or not. TestMain.hx contains this comment block:

/**
 * Auto generated Test Application.
 * Refer to munit command line tool for more information (haxelib run munit)
 */
class TestMain

Is this the correct way to generate detailed coverage on Neko?

Detailed reporting output

Disclaimer: I have seen a closed issue on milestone 0.2 talking about something similar, but I don't see that feature in the version currently available on Haxelib.

I'd like to get a detailed output of coverage, per package, class and method. The best possible output would be something like Atlassian Clover, where you see your code with covered lines highlighted in green (and can even see which tests covered the case).

Is that something already supported, and I've missed it? Or is this planned in the future?

Thanks for all the awesome work.

Line number missing from coverage block output (if single line)

At some point pre 1.2.4 the line number got dropped from block locations.

Currently in munit the output looks like this:

someMethod (/path/to/src/package/Foo.hx chars 3-35)

should be like this:

someMethod (/path/to/src/package/Foo.hx:126: chars 3-35)

Cause: In CoverageExpressionParser the block.location is droping the line number from the originating PosInfos

Allow CoverageLogger to start/stop/resume recording coverage for currentTest

Currently CoverageLogger.currentTest provides a mechanism for recording a temporary set of statement and branch results. This is used in MUnit to calculate an accurate percentage for the current class irrespective while it's tests are running.

MUnit 2 will support running tests out of order (i.e. not run all tests in a single class sequentially). Proposal is to extend the 'currentTest' to cache each filter set and resume recording to it each time currentTest is set to the same value.

logger.currentTest = "Foo"
//Foo coverage now at 0%
//run FooTest.doSomething()
//Foo coverage now at 50%
logger.currentTest = "Bar";
//run BarTest.doSomething()
logger.currentTest = "Foo"
//run FooTest.doSomethingElse()
//Foo coverage now at 75%