collective / collective.xmltestreport

Test runner which can output an XML report compatible with JUnit and Jenkins

Introduction

This package provides an extension to the test runner to the one that ships with zope.testrunner, as well as a buildout recipe based on zc.recipe.testrunner to install a test script for this test runner.

The test runner is identical to the one in zope.testrunner, but it is capable of writing test reports in the XML format output by JUnit/Ant. This allows the test results to be analysed by tools such as the Jenkins continuous integration server.

Code repository

https://github.com/collective/collective.xmltestreport

Usage

In your buildout, add a part like this:

[buildout]
parts =
    ...
    test

...

[test]
recipe = collective.xmltestreport
eggs =
    my.package
defaults = ['--auto-color', '--auto-progress']

The recipe accepts the same options as zc.recipe.testrunner, so look at its documentation for details.

When buildout is run, you should have a script in bin/test and a directory parts/test.

To run the tests, use the bin/test script. If you pass the --xml option, test reports will be written to parts/test/testreports directory:

$ bin/test --xml -s my.package

Use bin/test --help for a full list of options.

If you are using Jenkins, you can now configure the build to publish JUnit test reports for <buildoutdir>/parts/test/testreports/*.xml.

collective.xmltestreport's Issues

collective.xmltestreport 1.2.3 does not include CHANGES.txt

http://pypi.python.org/pypi/collective.xmltestreport/1.2.3
-> CHANGES.txt is missing in the dist, but included in setup.py

Getting distribution for 'collective.xmltestreport'.
error: CHANGES.txt: No such file or directory
An error occured when trying to install collective.xmltestreport 1.2.3. Look above this message for any errors that were output by easy_install.
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe collective.xmltestreport.
  Getting distribution for 'collective.xmltestreport'.
Error: Couldn't install: collective.xmltestreport 1.2.3

Traceback with newer zope (4.5.1)

This traceback happens right after printing the test totals, after that, it tries to write the XML report that jenkins, or other CI, can use to make visualizations and graphs out of it.

Traceback (most recent call last):
  File "bin/jenkins-test", line 350, in <module>
    '--test-path', '/home/jenkins/workspace/plone-next/src/freitag.util.validators/src',
  File "/home/jenkins/.buildout/eggs/cp27m/collective.xmltestreport-2.0.2-py2.7.egg/collective/xmltestreport/runner.py", line 66, in run
    failed = run_internal(defaults, args, script_parts=script_parts)
  File "/home/jenkins/.buildout/eggs/cp27m/collective.xmltestreport-2.0.2-py2.7.egg/collective/xmltestreport/runner.py", line 83, in run_internal
    runner.options.output.writeXMLReports()
  File "/home/jenkins/.buildout/eggs/cp27m/collective.xmltestreport-2.0.2-py2.7.egg/collective/xmltestreport/formatter.py", line 258, in writeXMLReports
    errorMessage = str(excInstance)
  File "/home/jenkins/.buildout/eggs/cp27m/Chameleon-3.8.1-py2.7.egg/chameleon/exc.py", line 275, in __call__
    for name, value in self._kwargs.items()
  File "/home/jenkins/.buildout/eggs/cp27m/Zope-4.5.1-py2.7.egg/OFS/SimpleItem.py", line 87, in __repr__
    return super(PathReprProvider, self).__repr__()
TypeError: descriptor '__repr__' for 'persistent.Persistent' objects doesn't apply to 'Acquisition.ImplicitAcquisitionWrapper' object

No idea what Persistent object can be 🤷

Does not support buildout 2

collective.xmltestreport requires z3c.rescipe.scripts, which does not work on buildout 2.

Exception: Gaaa!

https://pypi.python.org/packages/source/c/collective.xmltestreport/collective.xmltestreport-1.3.1.zip contains a file not present in the source repository: collective/xmltestreport/tests/test_teardown.py, with the following contents:

from unittest import TestCase


class Layer(object):

    __name__ = 'Layer'
    __bases__ = ()

    def setUp(self):
        pass

    def tearDown(self):
        raise Exception('Gaaa!')


class TestTearDown(TestCase):

    layer = Layer()

    def testFoo(self):
        pass

This "test file" gets included in my test suite for silly reasons having to do with buildout and zc.recipe.testrunner, and causes a failure when I forget to filter it out.

Exit codes

When a test is failing (an assertion within the test), xmltestreport exits with a code != 0 and writes a .xml file declaring the failure.
When a test-setup does not work, xmltestreport exists too with a exit code != 0 but does not write an xml.

When using it with Jenkins CI the first scenario should result in yellow balls but the second should result in red balls.
The problem is that it is not possible to have yellow balls on test failures but detect test-setup failures too.

For having yellow balls it is possible to just ignore the exitcode. But when a test-setup fails there is no xml file and therefore the job will have green balls instead of red balls.

Possible fix:
I'd like to change the recipe so that the command calls runner.run_internal instead of runner.run at
https://github.com/collective/collective.xmltestreport/blob/master/collective/xmltestreport/recipe.py#L83 .

This will result in having an exit code 0 when tests are failing (but the failure is declared in the xml) but having an exit code != 0 when the test-setup is failing.

I'm not sure how good this idea is - does someone have another idea or does not agree with my solution?

Skipped tests do not seem to be reported

Tests skipped by zope.testunner currently seem to be ignored in the XML result file.

See https://stackoverflow.com/questions/4922867/what-is-the-junit-xml-format-specification-that-hudson-supports for implementation hints.

XML test reports not published for jenkins since 1.2.7

Since updating from 1.2.6 to 1.2.8, XML test reports are not generated on our Jenkins server.

Configuration has not changed between these updates, and all tests continue to run and pass.

Configuration in buildout:

[test]
recipe = collective.xmltestreport
eggs = 
    my_egg [test]
defaults = ['--xml', '--auto-color', '--auto-progress']

Downgrading to 1.2.6 meant reports were generated again.

Problems with zope.testrunner 6.3

Some of the collective.xmltestreport code was merged into zope.testrunner 6.3. When I upgrade the Plone 6.0 core development buildout to use that version, I get an error:

$ bin/test -u
Traceback (most recent call last):
  File "/Users/maurits/community/plone-coredev/6.0/bin/test", line 297, in <module>
    import collective.xmltestreport.runner
  File "/Users/maurits/shared-eggs/cp311/collective.xmltestreport-2.0.2-py3.11.egg/collective/xmltestreport/runner.py", line 36, in <module>
    group.add_argument(
  File "/Users/maurits/.pyenv/versions/3.11.7/lib/python3.11/argparse.py", line 1468, in add_argument
    return self._add_action(action)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/.pyenv/versions/3.11.7/lib/python3.11/argparse.py", line 1670, in _add_action
    action = super(_ArgumentGroup, self)._add_action(action)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/maurits/.pyenv/versions/3.11.7/lib/python3.11/argparse.py", line 1482, in _add_action
    self._check_conflict(action)
  File "/Users/maurits/.pyenv/versions/3.11.7/lib/python3.11/argparse.py", line 1619, in _check_conflict
    conflict_handler(action, confl_optionals)
  File "/Users/maurits/.pyenv/versions/3.11.7/lib/python3.11/argparse.py", line 1628, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
argparse.ArgumentError: argument --xml: conflicting option string: --xml

The changelog for zope.testrunner provides a hint:

Add new –xml path option to write JUnit-like XML reports. Code comes from collective.xmltestreport, but be aware that here –xml is not a boolean, but expects a path!

It is not clear to me what to do though. Does collective.xmltestreport need an update? Should we stop using it, and use zope.testrunner directly? Do we need a change in tests.cfg?

Depending on the answer, it might be better to only do this in Plone 6.1, not 6.0.

No argparse support

The most up-to-date version of zope.testrunner on github has removed optparse and instead replaced it with argparse, as recommended by official documentation. This means that when zope.testrunner cuts a new release, this package will crash with an AttributeError thanks to code in runner.py that's expecting optparse code. I suggest placing in support for both optparse and argparse to 'futureproof'.

environment option broken

Since version 1.2.8 (fix of issue #5) the environment option is broken in the same way as it is broken in zc.recipe.testrunner: The contents of the environment option are not rendered in the generated script when using a virtualenv Python.

Example: See the following buildout.cfg:

[buildout]
parts = test
versions = versions

[versions]
collective.xmltestreport = 1.2.6

[test-env]
foo = bar

[test]
recipe = collective.xmltestreport
eggs = collective.xmltestreport
environment = test-env

When used with a Python interpreter from a virtualenv it renders bin/test like this:

#!/opt/2.7/bin/python

import sys
sys.path[0:0] = [...]

import os
...
os.environ['foo'] = 'bar'


import collective.xmltestreport.runner

if __name__ == '__main__':
    ...

After upgrading to 1.2.8 the block beginning with import os is missing.

errors during layer teardown prevent the report from getting written

Here's a sample of what happens...the second exception prevents any report from getting written for plone.app.contenttypestest.test_content_profile.PloneAppContenttypesContent.

Perhaps we should fix the layer teardown, but the most important thing to fix is making sure that problems with layer teardown don't prevent the report from getting written.

Running plone.app.contenttypes.tests.test_content_profile.PloneAppContenttypesContent:Integration tests:
  Tear down plone.app.contenttypes.testing.PloneAppContenttypes:Robot in 0.000 seconds.
  Tear down plone.testing.z2.ZServer in 4.133 seconds.
  Set up plone.app.contenttypes.tests.test_content_profile.PloneAppContenttypesContent Traceback (most recent call last):
  File "/Users/davisagli/.buildout/eggs/zope.testrunner-4.1.1-py2.7.egg/zope/testrunner/runner.py", line 380, in run_layer
    setup_layer(options, layer, setup_layers)
  File "/Users/davisagli/.buildout/eggs/zope.testrunner-4.1.1-py2.7.egg/zope/testrunner/runner.py", line 667, in setup_layer
    setup_layer(options, base, setup_layers)
  File "/Users/davisagli/.buildout/eggs/zope.testrunner-4.1.1-py2.7.egg/zope/testrunner/runner.py", line 672, in setup_layer
    layer.setUp()
  File "/Users/davisagli/Plone/5.0/src/plone.app.testing/plone/app/testing/helpers.py", line 343, in setUp
    self.setUpPloneSite(portal)
  File "/Users/davisagli/Plone/5.0/src/plone.app.contenttypes/plone/app/contenttypes/tests/test_content_profile.py", line 32, in setUpPloneSite
    self.applyProfile(portal, 'plone.app.contenttypes:plone-content')
  File "/Users/davisagli/Plone/5.0/src/plone.app.testing/plone/app/testing/helpers.py", line 389, in applyProfile
    return applyProfile(portal, profileName)
  File "/Users/davisagli/Plone/5.0/src/plone.app.testing/plone/app/testing/helpers.py", line 113, in applyProfile
    setupTool.runAllImportStepsFromProfile(profileId)
  File "/Users/davisagli/.buildout/eggs/Products.GenericSetup-1.7.3-py2.7.egg/Products/GenericSetup/tool.py", line 350, in runAllImportStepsFromProfile
    ignore_dependencies=ignore_dependencies)
   - __traceback_info__: profile-plone.app.contenttypes:plone-content
  File "/Users/davisagli/.buildout/eggs/Products.GenericSetup-1.7.3-py2.7.egg/Products/GenericSetup/tool.py", line 1100, in _runImportStepsFromContext
    message = self._doRunImportStep(step, context)
  File "/Users/davisagli/.buildout/eggs/Products.GenericSetup-1.7.3-py2.7.egg/Products/GenericSetup/tool.py", line 1015, in _doRunImportStep
    return handler(context)
   - __traceback_info__: portlets
  File "/Users/davisagli/Plone/5.0/src/plone.app.portlets/plone/app/portlets/exportimport/portlets.py", line 727, in importPortlets
    importer.body = body
  File "/Users/davisagli/.buildout/eggs/Products.GenericSetup-1.7.3-py2.7.egg/Products/GenericSetup/utils.py", line 509, in _importBody
    self._importNode(dom.documentElement)
  File "/Users/davisagli/Plone/5.0/src/plone.app.portlets/plone/app/portlets/exportimport/portlets.py", line 249, in _importNode
    self._initProvider(node)
  File "/Users/davisagli/Plone/5.0/src/plone.app.portlets/plone/app/portlets/exportimport/portlets.py", line 258, in _initProvider
    self._initPortlets(node)
  File "/Users/davisagli/Plone/5.0/src/plone.app.portlets/plone/app/portlets/exportimport/portlets.py", line 312, in _initPortlets
    self._initAssignmentNode(child)
  File "/Users/davisagli/Plone/5.0/src/plone.app.portlets/plone/app/portlets/exportimport/portlets.py", line 459, in _initAssignmentNode
    portlet_factory = getUtility(IFactory, name=type_)
  File "/Users/davisagli/.buildout/eggs/zope.component-3.9.5-py2.7.egg/zope/component/_api.py", line 169, in getUtility
    raise ComponentLookupError(interface, name)
ComponentLookupError: (<InterfaceClass zope.component.interfaces.IFactory>, 'portlets.Events')

Tearing down left over layers:
  Tear down plone.app.contenttypes.testing.PloneAppContenttypes in 0.109 seconds.
  Tear down plone.app.testing.layers.PloneFixture
Traceback (most recent call last):
  File "/Users/davisagli/Plone/5.0/bin/jenkins-test", line 469, in <module>
    '--test-path', '/Users/davisagli/.buildout/eggs/zope.globalrequest-1.0-py2.7.egg',
  File "/Users/davisagli/.buildout/eggs/collective.xmltestreport-1.3.0-py2.7.egg/collective/xmltestreport/runner.py", line 60, in run
    failed = run_internal(defaults, args, script_parts=script_parts)
  File "/Users/davisagli/.buildout/eggs/collective.xmltestreport-1.3.0-py2.7.egg/collective/xmltestreport/runner.py", line 72, in run_internal
    runner.run()
  File "/Users/davisagli/.buildout/eggs/zope.testrunner-4.1.1-py2.7.egg/zope/testrunner/runner.py", line 148, in run
    self.run_tests()
  File "/Users/davisagli/.buildout/eggs/zope.testrunner-4.1.1-py2.7.egg/zope/testrunner/runner.py", line 252, in run_tests
    tear_down_unneeded(self.options, (), setup_layers, True)
  File "/Users/davisagli/.buildout/eggs/zope.testrunner-4.1.1-py2.7.egg/zope/testrunner/runner.py", line 644, in tear_down_unneeded
    l.tearDown()
  File "/Users/davisagli/Plone/5.0/src/plone.app.testing/plone/app/testing/layers.py", line 102, in tearDown
    with z2.zopeApp() as app:
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/Users/davisagli/.buildout/eggs/plone.testing-4.0.8-py2.7.egg/plone/testing/z2.py", line 250, in zopeApp
    app = addRequestContainer(Zope2.app(connection), environ=environ)
  File "/Users/davisagli/.buildout/eggs/Zope2-2.13.21-py2.7.egg/Zope2/__init__.py", line 52, in app
    return bobo_application(*args, **kw)
  File "/Users/davisagli/.buildout/eggs/Zope2-2.13.21-py2.7.egg/App/ZApplication.py", line 75, in __call__
    return connection.root()[aname]
  File "/Users/davisagli/.buildout/eggs/ZODB3-3.10.5-py2.7-macosx-10.8-x86_64.egg/ZODB/Connection.py", line 366, in root
    return RootConvenience(self.get(z64))
  File "/Users/davisagli/.buildout/eggs/ZODB3-3.10.5-py2.7-macosx-10.8-x86_64.egg/ZODB/Connection.py", line 248, in get
    p, serial = self._storage.load(oid, '')
AttributeError: 'NoneType' object has no attribute 'load'

Release please

Can someone grant the 'plone' user on PyPI rights for this package? Then I can make a release.

PyPI owners:
optilude, jfroche, gotcha, davisagli, icemac, hannosch, wosc, iElectric, dokai, jone, timo

Error in writing test report if it contains non ascii.

We see this in Plone coredev 5.1 with plip-zope4.cfg because there is a failing test which contains non ascii (note the Pictüre 1.png):

Error in test test_image_id_from_unicode_title (Products.ATContentTypes.tests.test_atimage.TestIDFromTitle)
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/Users/maurits/community/plone-coredev/5.1/src/Products.ATContentTypes/Products/ATContentTypes/tests/test_atimage.py", line 96, in test_image_id_from_unicode_title
    self._make_image('', filename=u'Pictüre 1.png'.encode('utf-8'))
  File "/Users/maurits/community/plone-coredev/5.1/src/Products.ATContentTypes/Products/ATContentTypes/tests/test_atimage.py", line 75, in _make_image
    '/createObject?type_name=Image')
  File "/Users/maurits/community/plone-coredev/5.1/src/zope.testbrowser/src/zope/testbrowser/browser.py", line 255, in open
    self._processRequest(url, make_request)
  File "/Users/maurits/community/plone-coredev/5.1/src/zope.testbrowser/src/zope/testbrowser/browser.py", line 288, in _processRequest
    self._checkStatus()
  File "/Users/maurits/community/plone-coredev/5.1/src/zope.testbrowser/src/zope/testbrowser/browser.py", line 296, in _checkStatus
    raise HTTPError(self.url, code, msg, [], None)
HTTPError: HTTP Error 500: Internal Server Error

On standard Plone 5.1 we do not see this, simply because the test does not fail there. When I deliberately add a failure in this same test, I get the same result there.

Traceback from collective.xmltestreport is this (taken from Jenkins):

Total: 622 tests, 18 failures, 5 errors and 0 skipped in 1 minutes 9.991 seconds.
Traceback (most recent call last):
  File "/home/jenkins/workspace/plip-zope4/bin/test", line 462, in <module>
    '--test-path', '/home/jenkins/.buildout/eggs/interlude-1.3.1-py2.7.egg',
  File "/home/jenkins/workspace/plip-zope4/src/collective.xmltestreport/collective/xmltestreport/runner.py", line 60, in run
    failed = run_internal(defaults, args, script_parts=script_parts)
  File "/home/jenkins/workspace/plip-zope4/src/collective.xmltestreport/collective/xmltestreport/runner.py", line 77, in run_internal
    runner.options.output.writeXMLReports()
  File "/home/jenkins/workspace/plip-zope4/src/collective.xmltestreport/collective/xmltestreport/formatter.py", line 287, in writeXMLReports
    outputFile.write(prettyXML(testSuiteNode).decode('utf-8'))
  File "/home/jenkins/workspace/plip-zope4/src/collective.xmltestreport/collective/xmltestreport/utils.py", line 42, in prettyXML
    return ElementTree.tostring(tree)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1126, in tostring
    ElementTree(element).write(file, encoding, method=method)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 820, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
    _serialize_xml(write, e, encoding, qnames, None)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 939, in _serialize_xml
    _serialize_xml(write, e, encoding, qnames, None)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 937, in _serialize_xml
    write(_escape_cdata(text, encoding))
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1073, in _escape_cdata
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 323: ordinal not in range(128)

What basically happens, can be tried in a standard Python prompt:

>>> from xml.etree import ElementTree
>>> failureNode = ElementTree.Element('failure')
>>> failureNode.text = 'H\xc3\xac'  # 'Hi' with an accent, encoded as 'utf-8'.
>>> ElementTree.tostring(failureNode)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1126, in tostring
    ElementTree(element).write(file, encoding, method=method)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 820, in write
    serialize(write, self._root, encoding, qnames, namespaces)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 937, in _serialize_xml
    write(_escape_cdata(text, encoding))
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1073, in _escape_cdata
    return text.encode(encoding, "xmlcharrefreplace")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Using ElementTree.tostring(failureNode, encoding='utf-8') does not help.

Wait, I thought the next failed, but it works now in a Python prompt:

>>> failureNode.text = u'H\xec'  # Same as above, but as unicode.
>>> ElementTree.tostring(failureNode)
'<failure>H&#236;</failure>'

I tried changing our formatter.py accordingly, but it did not help. Maybe I need to do decoding in a few more places.

I am working on this currently, at the Innsbruck Sprint.

Not callable as a module in other programs (eg Coverage.py)

There are countless cases where I've needed to call collective.xmltestreport as a module, namely in coverage.py like so:

coverage run -m collective.xmltestreport --test-path=. --auto-color --auto-progress --xml

I can do this with zope.testrunner as-is. If this package is supposed to be identical to zope.testrunner, then I heavily recommend a __main__.py file and __init__.py file get mirrored from zope.testrunner.