junit reporter format? about newman

[Ernesto-Maserati-CH]

I don't understand the latest state of this issue.
Is it now built into Newman?
Or does it to be converted with a custom script by each guy who wants a xUnit report?

[prakhar1989]

Hi @mattaylor, this sounds like a great idea. We'll work on this and release an update. Will keep you posted! 👍

[patimen]

+1 on this!

[mattaylor]

FYI heres a brain dead livescript module to convert the json output to xunit...

report = require './testReport.json'
xml = '<?xml version="1.0" encoding="UTF-8"?>'
for suite in report.results
   xml += "<testsuite name='#{suite.name}'>"
   for test, count of suite.testPassFailCounts then xml += "<testcase label='#test' failures=#{count.fail} />"
   xml += "</testsuite>"
console.log xml

[morozgrafix]

+1 on junit/xunit reporter to be built into newman

[nbalsamo]

This would be awesome to have junit/xunit built in.

[jimthedev]

This would be fantastic since most CI tools can easily parse xunit style reporting.

[jaetask]

+1 one this. Any update? would be very nice to integrate with jenkins

[jxf86]

+1

[abhijitkane]

Newman 1.2.0 now supports junit-style output for tests. (Thanks @pal-thomassen)

Sample response:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="TestRequest" id="609a5233-6542-c2ed-b308-446101ac693d" timestamp="2014-12-02T07:06:21.186Z" time="51.50 ms" totalTime="103 ms" tests="4" iterations="2">
        <testcase name="SampleTest1" successes="2" failures="0" />
        <testcase name="SampleTest2" successes="2" failures="0" />
        <testcase name="SampleTest3" successes="1" failures="1" />
        <testcase name="SampleTest4" successes="0" failures="2" />
    </testsuite>
</testsuites>

[sciutand]

The syntax of the report file does not seem to correspond to the junit supported syntax found at http://llg.cubic.org/docs/junit/

successes is not recognised.
failures property should live in the testsuite not testcase. Just tested with Jenkins junit plugin and success es/failures are not reported correctly because of malformed syntax

[Torquem]

Jenkins seems to expect the optional failure inside the testcase.

<testcase ...>
    <failure message="I so fail" type="epic" />
</testcase>

[abhijitkane]

A <testsuite> element corresponds to a request.
A single request can have multiple tests (represented by the tests[] array).
Each test corresponds to a <testcase>.

In case of multiple iterations, each test is run multiple times. So having a single <failure> element inside the <testcase> isn't suitable.

The next newman version (v1.2.5) will have successes and failures attributes in the <testsuite> element, that'll aggregate the counts in the constituent <testcase> elements

[sbutlerjr]

just tested 1.2.6. successes and failures are in the testsute and testcase node. The testsuite node is aggregating the test case node. However Jenkins still isn't parsing that format it seems. Jenkins reads that junit output, it shows them all as successful, even when there are failures in the junit xml.

[sbutlerjr]

To add some additional detail to this, the Jenkins xsd for jUnit is located here: https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd

So, an example file would like like below, if there were an error. Seems like the current version that newman creates is missing the failures sumed up in the testsuites node, and its also missing the failure node within the testcase (if it had an error). For example:

<testsuite failures="1" tests="3">
    <testcase classname="foo1" name="ASuccessfulTest"/>
    <testcase classname="foo2" name="AnotherSuccessfulTest"/>
    <testcase classname="foo3" name="AFailingTest" failures="1">
        <failure type="NotEnoughFoo"> details about failure </failure>
    </testcase>
</testsuite>

[rclayton-the-terrible]

+10

[sbutlerjr]

Tested the patch from commit #184 and it worked. Thanks!