GithubHelp home page GithubHelp logo

danger-junit's Introduction

danger-junit

This Danger Plugin allows you to standardise the output for all of your testing runs. Most test runners include an ability to have a reporter that conforms to the JUnit XML standard. This plugin will understand that file and offer a way to introspect it, and to report on it.

Installation

$ gem install danger-junit

Usage

Ruby

For Rspec, add the gem rspec_junit_formatter to your project. Then use a .rspec file to configure your tests to have multiple reporters. This file looks like:

...
--format documentation
--format RspecJunitFormatter --out junit-results.xml
...

Then you can pass the junit-results.xml file to the plugin in your Dangerfile.

JS

In a Jasmine, or Jest testing project, you want to install the module jasmine-reporters. Then as you are setting up your Jasmine runner, add the following:

var junitReporter = new jasmineReporters.JUnitXmlReporter({
    savePath: 'junit-results.xml',
    consolidateAll: false
});
jasmine.getEnv().addReporter(junitReporter);

Then you can pass the junit-results.xml file to the plugin in your Dangerfile.

iOS

You have a lot of options:

  • trainer is the fastest way to generate the junit file.
  • xcpretty is used in both fastlane, and a lot of standard setups.
  • xctool includes it's own reporter for creating the file.

JUnit

Report, or inspect any JUnit XML formatted test suite report.

Testing frameworks have standardized on the JUnit XML format for reporting results, this means that projects using Rspec, Jasmine, Mocha, XCTest and more - can all use the same Danger error reporting. Perfect.

You can see some examples on this page from Circle CI and on this project's README about how you can add JUnit XML output for your testing projects.

Parse the XML file, and let the plugin do your reporting
junit.parse "/path/to/output.xml"
junit.report
Let the plugin parse the XML file, and report yourself
junit.parse "/path/to/output.xml"
fail("Tests failed") unless junit.failures.empty?
Warn on a report about skipped tests
junit.parse "/path/to/output.xml"
junit.show_skipped_tests = true
junit.skipped_headers = [:name, :file]
junit.report
Only show specific parts of your results
junit.parse "/path/to/output.xml"
junit.headers = [:name, :file]
junit.report
Only show specific parts of your results
junit.parse "/path/to/output.xml"
all_test = junit.tests.map(&:attributes)
slowest_test = sort_by { |attributes| attributes[:time].to_f }.last
message "#{slowest_test[:time]} took #{slowest_test[:time]} seconds"

Attributes

tests - All the tests for introspection

passes - An array of XML elements that represent passed tests.

failures - An array of XML elements that represent failed tests.

errors - An array of XML elements that represent passed tests.

skipped - An array of XML elements that represent skipped tests.

show_skipped_tests - An attribute to make the plugin show a warning on skipped tests.

headers - An array of symbols that become the columns of your tests, if nil, the default, it will be all of the attributes.

skipped_headers - An array of symbols that become the columns of your skipped tests, if nil, the default, it will be all of the attributes for a single parse or all of the common attributes between multiple files

Methods

parse - Parses an XML file, which fills all the attributes, will raise for errors

report - Causes a build fail if there are test failures, and outputs a markdown table of the results.

Development

  1. Clone this repo
  2. Run bundle install to setup dependencies.
  3. Run bundle exec rake spec to run the tests.
  4. Use bundle exec guard to automatically have tests run as you make changes.
  5. Make your changes.

danger-junit's People

Contributors

abhirao avatar aikrice avatar bonkey avatar hanneskaeufler avatar krausefx avatar maxfriedrich avatar orta avatar readmecritic avatar vkt0r avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

danger-junit's Issues

Support multiple result files

Our CI setup leads to multiple junit format files. The simplest reason is that our backend generates results from rspec whereas the frontend has jest tests.

The current version of this plugin supports only one file. Calling parse multiple times means that only the last one will get reported.

I'm going to make a PR to support another method: junit.parse_files

Junit retry tests formatting

We have a small issue wherein since we retry our tests for failure, how do we incorporate that with danger-junit, because junit.report would fail even for single test ๐Ÿ˜“

Ability to parse all file matching a pattern

We are using the ci_reporter gem ( https://github.com/ci-reporter/ci_reporter_rspec ) with parrelle:spec (https://github.com/grosser/parallel_tests) which generate individual JUnit result file under the spec/reports directory.

The list of files is dynamic, based on the test name (i.e. when someone add a new spec file a new xml file pops up).

Currently there is no way to easily as for all file matching a pattern to be processed. File need to be listed out explicitly.

Invalid `Dangerfile` file: undefined method `nodes'

My Danger file include this lines

junit_tests_dir = "**/build/**/reports/jacoco/report.xml"
Dir[junit_tests_dir].each do |file_name|
    junit.parse file_name
    junit.report
end

I have exception

[!] Invalid `Dangerfile` file: undefined method `nodes' for #<Ox::DocType:0x000055b4b0207848>
#  from Dangerfile:14
#  -------------------------------------------
#  Dir[junit_tests_dir].each do |file_name|
>      junit.parse file_name
#      junit.report 
#  -------------------------------------------

Do you have any ideas?

Also, XML file is really large. ~30Mb

Add Danger screenshot

I'd be cool to add a screenshot illustrating the beauty of danger-junit. Something like this:
screen shot 2016-07-30 at 12 40 08 pm

[!] Invalid `Dangerfile` file: undefined method `header'

Starting today (with version 1.0.1) I get the following error when running Danger with a junit report on an iOS project.

bundler: failed to load command: danger (/Users/vagrant/.rbenv/versions/2.6.5/bin/danger)
Danger::DSLError: 
[!] Invalid `Dangerfile` file: undefined method `header' for #<Danger::Dangerfile:0x00007fba2c314c08>
 #  from Dangerfile:12
 #  -------------------------------------------
 #          markdown(message)
 >          junit.report        
 #      else 
 #  -------------------------------------------
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:70:in `method_missing'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/plugin_support/plugin.rb:23:in `method_missing'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-junit-1.0.1/lib/junit/plugin.rb:174:in `report'
  Dangerfile:12:in `processTestFiles'
  Dangerfile:32:in `block in eval_file'
  Dangerfile:31:in `each'
  Dangerfile:31:in `eval_file'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:307:in `eval'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:307:in `eval_file'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:200:in `block in parse'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:197:in `instance_eval'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:197:in `parse'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/dangerfile.rb:283:in `run'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/danger_core/executor.rb:29:in `run'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/lib/danger/commands/runner.rb:73:in `run'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/claide-1.0.3/lib/claide/command.rb:334:in `run'
  /Users/vagrant/.rbenv/versions/2.6.5/lib/ruby/gems/2.6.0/gems/danger-8.0.2/bin/danger:5:in `<top (required)>'
  /Users/vagrant/.rbenv/versions/2.6.5/bin/danger:23:in `load'
  /Users/vagrant/.rbenv/versions/2.6.5/bin/danger:23:in `<top (required)>'
[!] The exception involves the following plugins:
 -  danger-junit

I am running Danger as part of a Bitrise step as follows:

def processTestFiles(path, name) 
    if !Dir.glob(path).empty? then
        junit.parse path
        message = "## #{name}\n" <<
            "Status | Number of tests |\n" <<
            "--- | --- | \n" <<
            " โœ… | #{junit.passes.count} tests passed |\n"
        if junit.failures.count > 0 then 
            message = message << " ๐Ÿšซ | #{junit.failures.count} tests failed |\n"
        end
        markdown(message)
        junit.report        # <--- Fails here
    else 
        markdown("## #{name}\nโ‰๏ธ: No tests run")    
    end
end

......

Dir["build/split-results/*"].each do |f| 
    processTestFiles(f, File.basename(f, ".*"))    
end

It was working OK in version 1.0.0

Issue running danger-junit on M1 CI machines

Hello @orta due to a dependency to the Ox gem, it appears the junit report functionality is broken on M1 machines, have you run into this issue? Do you have any advice or could you offer any help to work around this problem with danger-junit?

Raised it with the Ox project too in order to see if a fix could be developed there (bit sad as we do not quite know what it entails at the moment so not able to commit a PR to fix this issue at the source sadly): ohler55/ox#297

/CI/Jenkins/workspace/tests-NA-all-commits-ios-Lobby-pr/ruby_path/ruby/3.1.0/gems/ox-2.14.11/lib/ox.rb:79:in `require': dlopen(/CI/Jenkins/workspace/tests-NA-all-commits-ios-Lobby-pr/ruby_path/ruby/3.1.0/gems/ox-2.14.11/ext/ox/ox.bundle, 0x0009): tried: '/CI/Jenkins/workspace/tests-NA-all-commits-ios-Lobby-pr/ruby_path/ruby/3.1.0/gems/ox-2.14.11/ext/ox/ox.bundle' (mach-o file, but is an incompatible architecture (have (x86_64), need (arm64e))) - 

We researched the issue and tried the commonly suggested workarounds like:

Ensuring all gems would be installed with the native extensions recompiled for the host machine:

bundle config set --local force_ruby_platform true

and/or

bundle config set force_ruby_platform true

We do have ruby as one of the supported platforms in our Gemfile.lock btw:

PLATFORMS
  arm64-darwin-21
  ruby
  x86_64-darwin-19
  x86_64-darwin-20
  x86_64-darwin-21

Adding the following platform restriction in the Gemfile:

# HACK(bouk): Overwrite Bundler's platform matcher to ignore universal CPU
# The protobuf and gRPC 'universal' macOS gems break on M1
module Bundler::MatchPlatform
    def match_platform(p)
      return false if ::Gem::Platform === platform && platform.cpu == "universal"
      Bundler::MatchPlatform.platforms_match?(platform, p)
    end
end

But nothing seems to do the trick :/...


References: protocolbuffers/protobuf#8199 (and similar ones)

Crash when parsing valid JUnit files

Seems like this plugin doesn't support all kinds of JUnit files yet

TestFile.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="2" failures="1">
    <testsuite name="ThemojiUITests" tests="2" failures="1">
        <testcase classname="ThemojiUITests" name="testExample()">
        </testcase>
        <testcase classname="ThemojiUITests" name="testIsBroken()">
            <failure message="XCTAssertTrue failed - /Users/fkrause/Developer/themoji/ThemojiUITests/ThemojiUITests.swift:24">
            </failure>
        </testcase>
    </testsuite>
</testsuites>
=> [#<Ox::Element:0x007ffed5b24928 @attributes={:classname=>"ThemojiUITests", :name=>"testExample()"}, @nodes=["\n        "], @value="testcase">,
 #<Ox::Element:0x007ffed5b24798
  @attributes={:classname=>"ThemojiUITests", :name=>"testIsBroken()"},
  @nodes=
   [#<Ox::Element:0x007ffed5b246a8
     @attributes={:message=>"XCTAssertTrue failed - /Users/fkrause/Developer/themoji/ThemojiUITests/ThemojiUITests.swift:24"},
     @nodes=["\n            "],
     @value="failure">],
  @value="testcase">]
[5] pry(#<Danger::DangerJunit>)> failed_tests.select { |test| test.nodes.count > 0 }
=> [#<Ox::Element:0x007ffed5b24928 @attributes={:classname=>"ThemojiUITests", :name=>"testExample()"}, @nodes=["\n        "], @value="testcase">,
 #<Ox::Element:0x007ffed5b24798
  @attributes={:classname=>"ThemojiUITests", :name=>"testIsBroken()"},
  @nodes=
   [#<Ox::Element:0x007ffed5b246a8
     @attributes={:message=>"XCTAssertTrue failed - /Users/fkrause/Developer/themoji/ThemojiUITests/ThemojiUITests.swift:24"},
     @nodes=["\n            "],
     @value="failure">],
  @value="testcase">]

Stack Trace

[!] Invalid `Dangerfile` file: undefined method `value' for "\n        ":String. Updating the Danger gem might fix the issue.

 #  from Dangerfile:4
 #  -------------------------------------------
 #    puts "Parsing JUnit file at path '#{path}'"
 >    junit.parse(path)
 #    junit.report
 #  -------------------------------------------

  /Users/fkrause/Developer/hacking/danger-junit/lib/junit/plugin.rb:100:in `block in parse'
  /Users/fkrause/Developer/hacking/danger-junit/lib/junit/plugin.rb:100:in `select'
  /Users/fkrause/Developer/hacking/danger-junit/lib/junit/plugin.rb:100:in `parse'
  Dangerfile:4:in `block (2 levels) in parse'
  Dangerfile:2:in `each'
  Dangerfile:2:in `block in parse'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bundler/gems/danger-a67bc71710d9/lib/danger/danger_core/dangerfile.rb:193:in `eval'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bundler/gems/danger-a67bc71710d9/lib/danger/danger_core/dangerfile.rb:193:in `block in parse'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bundler/gems/danger-a67bc71710d9/lib/danger/danger_core/dangerfile.rb:189:in `instance_eval'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bundler/gems/danger-a67bc71710d9/lib/danger/danger_core/dangerfile.rb:189:in `parse'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bundler/gems/danger-a67bc71710d9/lib/danger/commands/local.rb:109:in `run'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/gems/claide-1.0.0/lib/claide/command.rb:334:in `run'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bundler/gems/danger-a67bc71710d9/bin/danger:5:in `<top (required)>'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bin/danger:22:in `load'
  /Users/fkrause/.rbenv/versions/2.2.1/lib/ruby/gems/2.2.0/bin/danger:22:in `<top (required)>'

Before working on a fix I'd just like to know if I'm missing something, or if I misunderstood how the JUnit format works.

You used `puts` in your Dangerfile. To print out text to GitHub use `message` instead

I'm trying to run danger-junit as one of the steps in my CI setup. This is the contents of my danger/UnitTestDangerfile:

junit.parse "app/build/outputs/junit/merged-test-results.xml"
junit.show_skipped_tests = true
junit.report

But I get the complaint that I am using puts instead of message which is confusing for couple of reasons:
(1) I could not find any puts command in this plugin
(2) That error message assumes I am running on GitHub

This is the raw output from my CI server:

[15:17:14]: $ bundle exec danger --danger_id=unit-tests --dangerfile=danger/UnitTestDangerfile --new-comment
[15:17:15]: โ–ธ You used `puts` in your Dangerfile. To print out text to GitHub use `message` instead

Regarding (2) - although I can see from the code it should support GitLab too and there are else-clauses if we are running in an environment other than GitHub.

What can be wrong?

Customize fail message

It would be nice if we could customize failure message.
For example I'd like to show number of succeeded/failed tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.