Developing Code Coverage Support. about karma-parallel

[mkarthik90]

Works fine for me. I was using an older version.

[joeljeske]

I have not attempted to run with code-coverage enabled. It is possible that it is already working, although I highly doubt it.

The https://github.com/rschuft/karma-sharding project was the primary inspiration behind this project. He added code coverage support. You could look at his project and see if it would be possible to implement code coverage in the same way.

You could also try running karma-parallel with code coverage on and post back here the environment / config you used and any errors / inconsistencies in reports generated.

[PikaChokeMe]

I'm actually looking at it right now and trying to figure out how to fuse the coverage reporter into this project now.
Right now I'm using this with the angular CLI

Everything seems to work surprisingly well except it seems like, the code coverage is under-reporting. It's not completely broken it still reporting back something, just likely not the correct numbers. I thought maybe the values were quartered since I'm using 4 executors, but it doesn't seem to be the case? Branch coverage appears to be 0 in a lot of cases. I'm not really sure what that is about.

In karma.conf.js I have

frameworks: ['jasmine', '@angular/cli'],
        plugins: [
            require('karma-parallel'),
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
            require('karma-jasmine-html-reporter'),
            require('karma-coverage-istanbul-reporter'),
            require('karma-istanbul-threshold'),
            require('@angular/cli/plugins/karma'),
            require('karma-spec-reporter')
        ],
        parallelOptions:{
            executors: 4,
            shardStrategy: 'round-robin'
        },
        reporters: config.angularCli && config.angularCli.codeCoverage
            ? ['progress', 'coverage-istanbul', 'istanbul-threshold', 'spec']
            : ['progress', 'kjhtml'],

I forked the repo and started making changes here https://github.com/BustyLoli-Chan/karma-parallel
I tried to mimick the code coverage set up from karma sharding in here
https://github.com/BustyLoli-Chan/karma-parallel/blob/master/lib/framework.js
but it seems like the way the browsers are stored/tracked differs between the projects?

I'm also currently using istanbul-threshold to report coverage. I don't know if the coverage isn't working or if I have istanbul-threshold set up in a way that is incompatible.
Thank you for your help!

[PikaChokeMe]

So when I look at the karma debug output I do see that with karma-parallel

24 01 2018 17:18:15.995:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'lcovonly', 'json' ]
24 01 2018 17:18:23.920:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'lcovonly', 'json' ]
24 01 2018 17:18:37.717:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'lcovonly', 'json' ]
24 01 2018 17:18:50.867:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'lcovonly', 'json' ]

Happens four times whereas without it only happens once without running karma-parallel

24 01 2018 17:25:03.320:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'lcovonly', 'json' ]

I probably need a way to get them to only write the combined data once.

[joeljeske]

Hmm, interesting. I merged your changes into a test branch and added the karma-coverage reporter and it seemed to work. This notably has a lot less moving parts since its not using angular / webpack. Do you want to try it out?
https://github.com/joeljeske/karma-parallel/commits/example/coverage-impl

It looks like the copied reporter from karma-sharding is hard-coded to use karma-coverage and not anything else like karma-istanbul-threshold or what-have-you. (https://github.com/BustyLoli-Chan/karma-parallel/blob/master/lib/reporter.js#L31) We might need to make this configurable somehow. Or perhaps even better, we can "merge" the coverage results before they ever get to the standard karma coverage interface so we can be compatible with any karma coverage reporter

[PikaChokeMe]

Sure, I'd love to pull down the test branch and look at it!

I kind of figured something like that just from the way it's included in index.js, but I wasn't 100% sure
https://github.com/BustyLoli-Chan/karma-parallel/blob/master/lib/index.js#L5

Yeah, the last suggestion definitely sounds ideal, although to be honest I'm not entirely sure how one would do that. I don't know a lot about karma or it's moving parts (maybe this would take place in fullConfig somewhere?)

[PikaChokeMe]

Alright so I gave it a shot but it seems like it's still failing.
From everything I looked into it seems like I guess it looks like coverage-istanbul is the default coverage reporter for @angular/cli projects

There's actually even an option

coverageIstanbulReporter: {
            combineBrowserReports: true,

But it doesn't seem to have any effect as I'm still seeing the files get written four times and now I'm seeing four separate blocks of low coverage errors for each of them.

I'll probably try messing around and seeing if it's as easy as targeting coverage-istanbul instead of just coverage

[KhalipskiSiarhei]

We have already integrated karma-parallel into our build process and lost coverage support as well, but we check the coverage locally by running tests without karma-parallel. I hope it will be added soon and it is nice to see active discussion here.

karma-sharding coverage doesn't work with CLI as was discussed here rschuft/karma-sharding#1. All the hopes for karma-parallel :-).

[PikaChokeMe]

Okay I've worked on this a lot recently

I've made some changes to how code coverage is exported so that it basically replaces not just the typical karma coverage but will also replace the coverage-istanbul reporter or can be referenced with coverage-parallel. It would be nice if there was a way to check if we are referencing coverage-parallel, because then we could just use that and not overwrite coverage or coverage-istanbul if for some reason users want (potentially messed up?) coverage results not from the parallel framework. I tried to do this, but I guess we don't have access to rootConfig from inside index.js. If you know a way around this I'd like to know it.

I kind of simplified the output coverage dir (it no longer includes the browser name) and I changed it so that I can specify an output dis as follows (I didn't really understand what it was doing before, so maybe this was a change for the worst).

coverageReporter: {
            reporters: [
                { type: 'html' },
                { type: 'json' }
            ],
            dir: './coverage'
        },

If you understand what it was doing before, or if it's needed for compatibility with certain karma configurations it would be good to revert it.

Like this I have it writing reports and almost properly working (it seems four of my files no longer meet code coverage that they seem to have met with istanbul-coverage before?) but more than anything I'm just happy it's writing html and json reports that require('karma-istanbul-threshold'), can read.

I also have an issue on output where it says TypeError: Cannot read property 'text' of undefined which traces back to istanbul at node_modules\istanbul\lib\report\html.js:242:52

if (type === 'no') {
            if (endLine !== startLine) {
                endLine = startLine;
                endCol = structuredText[startLine].text.originalLength();

Where I've noticed that structuredText[startLine] is undefined because structuredText is an array with only 10 elements. So there might be some mis-match with how our reporter is writing and how instanbul is interpreting this...

Some thoughts:

I don't know the differences between how karma-coverage and coverage-istanbul write reports... I wonder if there's a way to pull the source reporter from istanbul, and modify it like the karma-coverage one we have and then choose either or depending on the situation. Maybe that will make everything play nicely 100%.

I made the changes in a branch here. Sorry for the novel. Please let me know your thoughts.
https://github.com/BustyLoli-Chan/karma-parallel/tree/example/coverage-impl

[joeljeske]

Thanks for all your help on this! I've update my branch to reflect a couple of changes in how the coverages are aggregated. Currently, I am aggregating them at the karma-reporter api level. This means I am not changing the coverage reporter code at all, or relying on them to implement the aggregation. Instead, I am wrapping their reporter instance in a fake reporter. This aggregate fake reporter takes the events from karma regarding each browser instance, and reports events to the real reporter under a fake browser id. After very limited testing, it appears to work. I would like to test it out more heavily in different configurations and with different coverage reporters, but I am pretty confident in this approach and think it should work.

If you have time, please go through and give me your thoughts on the changes, and test it out in various configurations and let me know how it works.

[PikaChokeMe]

I looked at it and gave it a shot... but I'm currently just getting the error

30 01 2018 11:45:35.692:ERROR [reporter]: Can not load "parallel-coverage"!
  Error: No provider for "baseReporterDecorator"! (Resolving: webServer -> reporter -> reporter:coverage-istanbul -> baseReporterDecorator)

I don't know if this a side effect of require('karma-coverage-istanbul-reporter'), even if I exclude it from there and exclude coverage-istanbul from the reporters list in the CLI the angular CLI still requires it and gets upset

[joeljeske]

Ah yes, i've been doing the bulk of my testing with karma-coverage instead of karma-coverage-istanbul-reporter. I wasn't fully complying to the karma-reporter api. I fixed that and it appears to work on a vanilla ng-cli project now. Hopefully it works for you too

[PikaChokeMe]

I've tried again and now I'm getting the error

31 01 2018 10:07:54.709:ERROR [reporter]: Can not load "parallel-coverage"!
  RangeError: Maximum call stack size exceeded
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:25:21)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)

that's if I include 'parallel-coverage' as a reporter. If I don't include it I don't really get any errors, but it is clear that code coverage still gets written 5 times and looking through the HTML shows that a lot of files are missing coverage (it seems the same as if the reporters were never combined)

[joeljeske]

Ahh yes, I probably have a bug in the reporter filter thats causing the max call stack. But the reporter is automatically added to the list and should not need to be explicitly specified.

As for the writing out 5 times, I'm not sure. It would really help if I could have access to your project or a sample project that experiences this issue. Do you have it pubic somewhere?

[PikaChokeMe]

Sadly I don't :(

I'm working for a company releasing commercial software so I do my best to obfuscate even all posted code samples. I can try to create a dummy project with a few tests to see if it works the same.

looking at the HTML I can still see that a lot of files look like they are missing coverage...
This is from the latest code on the example/coverage-impl branch, should I be using master?

31 01 2018 11:16:49.185:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
31 01 2018 11:16:54.375:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
31 01 2018 11:16:58.117:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
31 01 2018 11:17:01.807:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
31 01 2018 11:17:04.649:DEBUG [karma]: Run complete, exiting.
31 01 2018 11:17:04.650:DEBUG [launcher]: Disconnecting all browsers
31 01 2018 11:17:04.756:WARN [launcher]: ChromeHeadless was not killed in 15000 ms, sending SIGKILL.

Waiting for remapped coverage source timed out…
31 01 2018 11:17:04.760:DEBUG [launcher]: Process ChromeHeadless exited with code 0
31 01 2018 11:17:04.761:DEBUG [temp-dir]: Cleaning temp dir Angular-App\AppData\Local\Temp\karma-25698132
31 01 2018 11:17:04.799:DEBUG [launcher]: Finished all browsers

it seems like it should also be throwing errors... based on line coverage and my settings

istanbulThresholdReporter: {
            src: './coverage/coverage-final.json',
            reporters: ['text'],
            thresholds: {
                global: {
                    statements: 0,
                    branches: 0,
                    lines: 75,
                    functions: 0
                },
                each: {
                    statements: 0,
                    branches: 0,
                    lines: 75,
                    functions: 0
                }
            }
        },

although if the coverage source is timing out, I guess it explains why this isn't exploding.

[joeljeske]

Can you post a gist of the karma config perhaps?

[PikaChokeMe]

// Karma configuration file, see link for more information
// https://karma-runner.github.io/0.13/config/configuration-file.html

module.exports = function (config) {
    config.set({
        basePath: '',
        frameworks: ['parallel', 'jasmine', '@angular/cli'],
        plugins: [
            require('karma-parallel'),
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
            require('karma-jasmine-html-reporter'),
            require('karma-istanbul-threshold'),
            require('karma-coverage-istanbul-reporter'),
            require('@angular/cli/plugins/karma'),
            require('karma-spec-reporter')
        ],
        parallelOptions:{
            executors: 4,
            shardStrategy: 'round-robin'
        },
        client:{
            clearContext: false // leave Jasmine Spec Runner output visible in browser
        },
        files: [
            { pattern: './src/test.ts', watched: false },
            './node_modules/jquery/dist/jquery.min.js'
        ],
        preprocessors: {
            './src/test.ts': ['@angular/cli']
        },
        mime: {
            'text/x-typescript': ['ts','tsx']
        },
        istanbulThresholdReporter: {
            src: './coverage/coverage-final.json',
            reporters: ['text'],
            excludes: [],
            thresholds: {
                global: {
                    statements: 0,
                    branches: 0,
                    lines: 75,
                    functions: 0
                },
                each: {
                    statements: 0,
                    branches: 0,
                    lines: 75,
                    functions: 0
                }
            }
        },
        coverageIstanbulReporter: {
            reports: [ 'html', 'json' ],
            fixWebpackSourcePaths: true
        },
        angularCli: {
            environment: 'dev'
        },
        specReporter: {
            maxLogLines: 9999,
            suppressErrorSummary: false,
            suppressFailed: false,
            suppressPassed: false,
            suppressSkipped: true,
            showSpecTiming: false
        },
        reporters: ['progress', 'kjhtml', 'istanbul-threshold', 'spec'],
        port: 9876,
        colors: true,
        logLevel: config.LOG_DEBUG,
        autoWatch: true,
        browsers: ['ChromeHeadless'],
        singleRun: false,
        captureTimeout: 210000, // it was already there
        browserDisconnectTimeout: 210000,
        browserDisconnectTolerance: 3,
        browserNoActivityTimeout: 210000,//by default 10000
        processKillTimeout: 15000
    });
};

[joeljeske]

What version of the ng-cli did you use to create your project?

[joeljeske]

Ok here are some things I have found:

1. The msg "Waiting for remapped coverage source timed out…" is from istanbul threshold. Its strategy is to watch the output file from istanbulThresholdReporter.src. After the run completes, it watches the file and reads it to check coverage levels. I am not sure why it cannot detect the file change event after the reporter seems to write it multiple times. But that is also why it does not throw on your coverage thresholds.

2. I cannot reproduce multiple write statements from reporter.coverage-istanbul. I'm wondering if we are on different versions of the ng-cli. I started a new project and changed the karma-config to match yours and do not see multiple writes. I did notice that 'coverage-istanbul' is not listed as a reporter in your karma conf. Does the ng cli add it dynamically? It is possible that on your version of the cli, it happens to add it after I look for it in my framework.js files and thus my reporter is not correctly aggregating it. Not sure..

Thoughts?

[PikaChokeMe]

Right now I'm on version 1.6
This is a giant nightmare of packages from package.json

  "dependencies": {
    "@angular/animations": "^5.1.0",
    "@angular/common": "^5.1.0",
    "@angular/compiler": "^5.1.0",
    "@angular/core": "^5.1.0",
    "@angular/forms": "^5.1.0",
    "@angular/http": "^5.1.0",
    "@angular/platform-browser": "^5.1.0",
    "@angular/platform-browser-dynamic": "^5.1.0",
    "@angular/router": "^5.1.0",
    "@angular/upgrade": "^5.1.0",
    "@ngrx/core": "~1.2.0",
    "@ngrx/store": "~4.1.1",
    "chart.js": "~2.6.0",
    "core-js": "^2.4.1",
    "enhanced-resolve": "^3.4.1",
    "font-awesome": "~4.7.0",
    "immutable": "~3.8.1",
    "moment": "2.18.1",
    "moment-timezone": "0.5.13",
    "mydatepicker": "2.0.0",
    "net": "^1.0.2",
    "ngx-bootstrap": "~2.0.0-beta.2",
    "ngx-filter-pipe": "1.0.0",
    "ngx-infinite-scroll": "~0.5.1",
    "ngx-pagination": "~3.0.0",
    "primeng": "^5.0.2",
    "quill": "^1.3.0",
    "rxjs": "^5.5.5",
    "stompjs": "^2.3.3",
    "web-animations-js": "^2.3.1",
    "zone.js": "^0.8.18"
  },
  "devDependencies": {
    "@angular/cli": "^1.6.0",
    "@angular/compiler-cli": "^5.0.0",
    "@angular/language-service": "^5.0.0",
    "@ngrx/store-devtools": "~4.1.1",
    "@types/bootstrap": "3.3.33",
    "@types/jasmine": "^2.5.54",
    "@types/jasminewd2": "~2.0.2",
    "@types/jquery": "2.0.41",
    "@types/node": "~6.0.60",
    "@types/stompjs": "~2.3.0",
    "@types/webpack": "^3.0.13",
    "@types/webpack-env": "^1.13.0",
    "angular2-template-loader": "^0.6.2",
    "awesome-typescript-loader": "^3.1.3",
    "codelyzer": "^4.0.1",
    "concurrently": "^3.4.0",
    "gulp": "^3.9.1",
    "gulp-inline-ng2-template": "^4.0.0",
    "istanbul-instrumenter-loader": "^3.0.0",
    "jasmine-core": "^2.8.0",
    "jasmine-spec-reporter": "^4.2.1",
    "jquery": "3.2.1",
    "json-loader": "^0.5.4",
    "karma": "^2.0.0",
    "karma-chrome-launcher": "^2.2.0",
    "karma-cli": "^1.0.1",
    "karma-coverage-istanbul-reporter": "^1.3.3",
    "karma-istanbul-threshold": "^1.2.2",
    "karma-jasmine": "^1.1.0",
    "karma-jasmine-html-reporter": "^0.2.2",
    "karma-parallel": "^0.1.2",
    "karma-spec-reporter": "^0.0.31",
    "ngrx-store-freeze": "~0.2.0",
    "raw-loader": "^0.5.1",
    "rimraf": "^2.6.1",
    "source-map-explorer": "^1.3.3",
    "to-string-loader": "^1.1.5",
    "ts-node": "~3.2.0",
    "tslint": "~5.7.0",
    "typescript": "~2.4.2",
    "webpack-angular-externals": "^1.0.2",
    "webpack-rxjs-externals": "^1.1.0"
  }

I think the ng cli is hard coded to require coverage-istanbul but I did have it included before explicitly (I think I forgot to re-add it when trying different configurations). I'll re-add it and see what kind of results I get.

[PikaChokeMe]

Making this change I have still noticed no difference...
I did notice my stuff didn't seem up to date though (might have been me being dumb the whole time)

I did update and now I'm seeing this

01 02 2018 10:12:34.015:ERROR [karma]: TypeError: Cannot read property 'name' of undefined
    at handleBrowserRegister (Angular-App\Repos\karma-parallel\lib\framework.js:45:33)
    at Server.emitter.on (Angular-App\Repos\karma-parallel\lib\framework.js:85:52)
    at emitOne (events.js:121:20)
    at Server.emit (events.js:211:7)
    at Browser.init (Angular-App\Repos\second-angular\node_modules\karma\lib\browser.js:79:13)
    at Socket.<anonymous> (Angular-App\Repos\second-angular\node_modules\karma\lib\server.js:309:20)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at Angular-App\Repos\second-angular\node_modules\karma\node_modules\socket.io\lib\socket.js:513:12
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[joeljeske]

Shoot. I fixed that last one but forgot to push it up. It fixes "Cannot read property 'name'..."
Just pushed.

[PikaChokeMe]

Alright, I'll try it
I'm still ending up with this as my end result... even when I completely remove the coverage folder and can still clearly see that it is being written from scratch/nothing

01 02 2018 16:53:41.428:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
TOTAL: 1273 SUCCESS
01 02 2018 16:53:45.771:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
01 02 2018 16:53:49.746:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
01 02 2018 16:53:54.280:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
01 02 2018 16:53:58.746:DEBUG [reporter.coverage-istanbul]: Writing coverage reports: [ 'html', 'json' ]
01 02 2018 16:54:02.194:DEBUG [karma]: Run complete, exiting.
01 02 2018 16:54:02.195:DEBUG [launcher]: Disconnecting all browsers
01 02 2018 16:54:02.297:WARN [launcher]: ChromeHeadless was not killed in 15000 ms, sending SIGKILL.

Waiting for remapped coverage source timed out…

this is still without including 'parallel-coverage'
reporters: ['progress', 'kjhtml', 'coverage-istanbul', 'istanbul-threshold', 'spec'],
in karma.conf.js

trying to include it still gives me the old

01 02 2018 16:56:22.591:ERROR [reporter]: Can not load "parallel-coverage"!
  RangeError: Maximum call stack size exceeded
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:25:21)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
    at Injector.get (Angular-App\node_modules\di\lib\injector.js:54:19)
01 02 2018 16:56:22.593:DEBUG [karma]: Registered a load error of type reporter with name parallel-coverage

[joeljeske]

Ok, I was finally able to reproduce your issue and I think I've tracked it down:

As I had suspected, ng-cli was adding the 'coverage-istanbul' reporter after my plugin was wrapping the reporters. This led to the coverage reporter not getting properly wrapped. I have moved the logic to remove the reporters from the framework to the middleware. This executes a bit later and is able to intercept reporters added by other frameworks.

Also, I fixed the max call stack issue.

At your leisure, please give it another shot. Thanks for your help in testing this!

[PikaChokeMe]

Should I still explicitly include parallel-coverage I have included it, but it doesn't seem to break anything. It seems like this is working.

I've also tested that it passes and fails, in several cases where it should. It looks like it's working great! This is awesome. Thank you for your hard work in putting all of this together!

Do you think you'll release a beta version of this soon or anything? I would love to actually start making use of this in our CLI system

[joeljeske]

It does not matter if you include it or not. It will automatically add it as needed. (For that reason, it will not be in the docs. Most people won't ever know about it.)

I'm glad its finally working! I've benefited a lot from the community and I've finally had a good chance to try to give back.

Yes, I will cleanup and release a version here soon.

[PikaChokeMe]

Awesome! I'm glad everything is working too.
Thank you so much again! I look forward to the release version!

[joeljeske]

Fixed in 0.2.0

[KhalipskiSiarhei]

@joeljeske in yarn it is still available 3 prev versions only, 0.2.0 is not available yet. It is issue in my local environment only or you have the same?

`>yarn upgrade [email protected]
yarn upgrade v1.3.2
[1/4] Resolving packages...
Couldn't find any versions for "karma-parallel" that matches "0.2.0"
? Please choose a version of "karma-parallel" from this list: (Use arrow keys)

0.1.2
0.1.1
0.1.0`

[joeljeske]

That’s odd. I was under the impression that yarn was nothing more than a npm client; meaning there is not an additional registry in which I must publish. Maybe try blowing the cache and/or doing a clean yarn add?

[KhalipskiSiarhei]

@joeljeske yes, today morning it started working :-). It looks like it was some local cache issue or smth like this. Thanks for the reply :-).

I am confirming it just works, CC report is good one! But what I noticed during my testing is the following: when ng test is finished the process doesn't return control, it is still hanging and as a result next steps of the build process can't start; after waiting of 1 hour it is cancelled. If I run ng test command not as a part of build process, but manually the result is the same: tests finished, CC report is created but the console is still opened and control is still not returned; I need to close console manually. It is not a working solution for automated build process where after successful tests should be run next steps.

In the prev version of the package 0.1.2 it did worked as expected with the same configuration: tests run, CC report was created but for one browser instance only, control was returned.

Here is how I run ng test manually:
ng test --watch=false --code-coverage=true --single-run=true --progress=true --colors=true --log-level=debug --reporters=progress,junit,coverage-istanbul --config=karma.conf.headless.js

Here is my karma.conf.headless.js config:

// Karma configuration file, see link for more information
// https://karma-runner.github.io/1.0/config/configuration-file.html

module.exports = function (config) {
    config.set({
        basePath: '',
        frameworks: ['parallel', 'jasmine', '@angular/cli'],
        parallelOptions: {
            executors: 4, // Defaults to cpu-count - 1
            shardStrategy: 'round-robin'
            // shardStrategy: 'description-length'  
        },
        plugins: [
            require('karma-jasmine'),
            require('karma-chrome-launcher'),
            require('karma-jasmine-html-reporter'),
            require('karma-coverage-istanbul-reporter'),
            require('karma-junit-reporter'),
            require('@angular/cli/plugins/karma'),
            require('karma-parallel')
        ],
        client: {
            clearContext: false // leave Jasmine Spec Runner output visible in browser
        },
        coverageIstanbulReporter: {
            reports: ['html', 'lcovonly', 'cobertura'],
            fixWebpackSourcePaths: true,
            // Most reporters accept additional config options. You can pass these through the `report-config` option
            'report-config': {
                // all options available at: https://github.com/istanbuljs/istanbuljs/blob/aae256fb8b9a3d19414dcf069c592e88712c32c6/packages/istanbul-reports/lib/html/index.js#L135-L137
                html: {
                    // outputs the report in ./coverage/html
                    subdir: 'html'
                }
            },
            thresholds: {
                // thresholds for all files
                global: {
                    statements: 95,
                    lines: 95,
                    branches: 73,
                    functions: 95
                },
            }
        },
        angularCli: {
            environment: 'dev'
        },
        files: [
        ],
        reporters: ['progress', 'kjhtml'],
        port: Math.floor((Math.random() * 500) + 9500),
        colors: true,
        logLevel: config.LOG_INFO,
        autoWatch: false,
        browsers: ['ChromeHeadless'],
        singleRun: true,
        concurrency: Infinity,
        customLaunchers: {
            ChromeHeadless: {
                base: 'Chrome',
                flags: [
                    '--headless',
                    '--disable-gpu',
                    // Without a remote debugging port, Google Chrome exits immediately.
                    `--remote-debugging-port=${Math.floor((Math.random() * 500) + 9000)}`,
                    '--no-sandbox'
                    // '--max_old_space_size=4096'
                ],
                debug: false
            }
        },
        // How long does Karma wait for a browser to reconnect (in ms).
        // With a flaky connection, it is pretty common that the browser disconnects, but the actual test execution is still running without any problems.
        // Karma does not treat a disconnection as an immediate failure and will wait for browserDisconnectTimeout (ms).If the browser reconnects during that time, everything is fine.
        // Default: 2000
        browserDisconnectTimeout: 6000,
        // The number of disconnections tolerated.
        // The disconnectTolerance value represents the maximum number of tries a browser will attempt in the case of a disconnection.
        // Usually, any disconnection is considered a failure, but this option allows you to define a tolerance level when there is a flaky network link between the Karma server and the browsers.
        // Default: 0
        browserDisconnectTolerance: 3,
        // How long will Karma wait for a message from a browser before disconnecting from it (in ms).
        // If, during test execution, Karma does not receive any message from a browser within browserNoActivityTimeout(ms), it will disconnect from the browser.
        // Default: 10000
        browserNoActivityTimeout: 60000,
        // Timeout for capturing a browser (in ms).
        // The captureTimeout value represents the maximum boot-up time allowed for a browser to start and connect to Karma.
        // If any browser does not get captured within the timeout, Karma will kill it and try to launch it again and, after three attempts to capture it, Karma will give up.
        // Default: 60000
        captureTimeout: 120000,
        // Karma will report all the tests that are slower than given time limit (in ms).This is disabled by default (since the default value is 0).
        reportSlowerThan: 1000
        // More info about disconnected issue is here:
        // https://github.com/karma-runner/karma-chrome-launcher/issues/154
        // https://github.com/karma-runner/karma/issues/2652
    });
};

[joeljeske]

Could you open a new issue?

[KhalipskiSiarhei]

@joeljeske done #5

[alok183]

I was trying to run my test parallel karma-parallel but not getting the consolidated junit and code coverage report.
My karma.config.js has

parallelOptions: {
executors: 4, // Defaults to cpu-count - 1
shardStrategy: 'round-robin',
// shardStrategy: 'description-length'
aggregatedReporterTest: '/junit/i'
}
Can anyone help me with sample karma.config.js code to get the consolidated reports.

[joeljeske]

It looks like you are specifying a string instead of a regex here
aggregatedReporterTest: '/junit/i'
instead of
aggregatedReporterTest: /junit/i

[alok183]

@joeljeske Thanks, now i am getting consolidated junit report.

Similar way i tried to get code coverage report but noticed its not executing all the test .Also not able to identify whether code coverage report is consolidated or not.
Karma.config:
parallelOptions: {
executors: 4, // Defaults to cpu-count - 1
shardStrategy: 'round-robin',
// shardStrategy: 'description-length'
aggregatedReporterTest: /coverage/i
},
plugins: [
require('karma-jasmine'),
require('karma-chrome-launcher'),
require('karma-jasmine-html-reporter'),
require('karma-coverage-istanbul-reporter'),
require('@angular/cli/plugins/karma'),
require('karma-htmlfile-reporter'),
require('karma-junit-reporter'),
require('karma-phantomjs-launcher'),
require('karma-parallel')
],
reporters: config.angularCli && config.angularCli.codeCoverage
? ['progress', 'coverage-istanbul']
: ['progress', 'kjhtml', 'html', 'junit'],

run ng test
ng test --single-run=true --browser=ChromeHeadless --code-coverage=true

output:

junit: here all test cases were executed
Executed 122 of 122 SUCCESS (11 mins 1.151 secs / 11 mins 1.016 secs)
HeadlessChrome 63.0.3239 (Windows 7.0.0): Executed 101 of 101 (1 FAILED) (8 mins 53.288 secs / 8 mins 53.141 secs)
HeadlessChrome 63.0.3239 (Windows 7.0.0): Executed 121 of 121 (1 FAILED) (11 mins 1.914 secs / 11 mins 1.771 secs)
HeadlessChrome 63.0.3239 (Windows 7.0.0): Executed 77 of 77 (6 FAILED) (6 mins 33.557 secs / 6 mins 33.016 secs)
TOTAL: 8 FAILED, 413 SUCCESS
TOTAL: 8 FAILED, 413 SUCCESS
TOTAL: 8 FAILED, 413 SUCCESS

code-coverage: not ran all the test and got exit

Executed 67 of 122 SUCCESS (11 mins 21.314 secs / 5 mins 34.993 secs)
HeadlessChrome 63.0.3239 (Windows 7.0.0): Executed 37 of 77 SUCCESS (6 mins 49.287 secs / 2 mins 41.514 secs)
HeadlessChrome 63.0.3239 (Windows 7.0.0): Executed 95 of 121 SUCCESS (11 mins 20.396 secs / 8 mins 29.122 secs)
HeadlessChrome 63.0.3239 (Windows 7.0.0): Executed 64 of 101 SUCCESS (8 mins 42.517 secs / 4 mins 41.518 secs)
TOTAL: 263 SUCCESS
TOTAL: 263 SUCCESS

[Raiune]

On version 0.2.9, I'm having the issue that my tests are meeting their required global coverage in normal, non parallel runs, but when I switch to karma-parallel, they are slightly below. Is it possible that I have something configured incorrectly?

[mkarthik90]

+1

[joeljeske]

If you are having an issue, please create a new bug instead of using a old issue. This ticket was for the initial development of code coverage support.