scalative / haas Goto Github PK

View Code? Open in Web Editor NEW

8.0 5.0 6.0 540 KB

Python Test Runner

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

haas's Introduction

haas: Extensible Python Test Runner

haas is a Python test runner that is backward-compatible with Python's built-in unittest Test Cases, but supports more advanced features, such as project-specific plugins.

Features

Runs unittest.TestCase based tests!
Plugin system, based on stevedore.
- Still subject to change and revision to clean the plugin APIs.
- Plugins for whole-test-run environment configuration (run before any client-code is imported).
- Plugins for test result output formatting.
- Plugins for test runner scheme (e.g. parallel runner)
Generic test result collection, presentation & output handled by plugins.
(Very) Basic parallel test run support.

Missing (unittest) Features

Does not support the unittest.load_tests protocol.
Does not support subtests.

Future Features

Per-project config file
Improve parallel test runner to allow conditional splitting of tests, or allow tests to be run in the main process.
Improve plugin system
More result output plugins/options (xunit, result summary)
... ?

Copyright

The name haas is taken from the animal on the cover of O'Reilly's Python Cookbook, a springhaas or Pedetes capensis.

haas's People

Contributors

Stargazers

Watchers

Forkers

itziakos mdickinson jvkersch shurph pzahemszky jayvdb

haas's Issues

Show ten slowest tests report

it would be useful for haas to show the 10 slowest tests in the summary report.

Haas does not package or install tests

Allow for tests to designate single/multi processing execution

While a number of tests can be run on a multiprocessing pool, some (e.g. tests that use the subprocess module) are not really suitable or cannot be executed properly. There should be a way, probably a decorator that could designate that a test class/method needs to be executed by the main process

Support the `load_tests` protocol

Python 2.7 introduced a load_tests protocol. While, as far as I can tell it is not commonly used, I think at some point we need to support it.

Support multiple start dirtectories given on command line

Add interface to run subprocess setup & teardown functions

A method of adding custom setup & teardown of the parallel runner's subprocesses.

Haas does not show warnings produced by code under test.

$ python -m unittest discover
..............xx.....x..<path>:47: DeprecationWarning: Please use assertRegex instead.
  self.assertRegexpMatches(request.url, r'^.*?regex$')
......x.....x..................................<path>:341: DeprecationWarning: Please use assertRegex instead.
  result.output, r'.*?regex$')
.......................................................
----------------------------------------------------------------------
Ran 126 tests in 0.207s

OK (expected failures=5)
$ haas
........................................................................................x.....x...............xx.....x........
----------------------------------------------------------------------
Ran 126 tests in 0.224s

OK (expected failures=5)

Support subtests as used in unittest.TestCase

ImportError from package under test results in 0 tests discovered

Adding raise ImportError() to the top-level __init__.py of a project yields this behaviour:

$ haas

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Allow using multiple test result output handlers

Release v0.5.0 decouples test result collection from result presentation. Currently only one presentation plugin can be enabled at a time.

It would be useful to be able to enable multiple. For example, with a test run occurring on Jenkins, it might be useful to display test progress to stdout/stderr as well as write an xunit XML file. These two very different output mechanisms would be handled by separate result presentation plugins.

Haas crashes on discovery if there is an emacs temporary file from an unsaved buffer

If, while editing a file in emacs, there is a recovery file of the form /path/.../.#some_file.py, haas will crash trying to import it as a python module with an error stating that the file does not exist.

Add an option to run each individual test in a new process

It would be useful to be able to have each test run in a clean process to ensure isolation between all test cases.

Replace the unittest TextTestResult

I find that the output of TextTestResult is lacking some useful information in verbose mode:

Timestamp
Current test number and total number of tests

Haas crashes when trying to import subdirectories which are not packages

For example, I have some tmp directory with some python packages junk as follows:

$root/tmp/some_directory/package/
$root/tmp/some_directory/package/__init__.py

but there is no __init__ above package. Haas traceback:

Traceback (most recent call last):
  File "/home/davidc/src/enthought/enstaller/.env/bin/haas", line 9, in <module>
    load_entry_point('haas==0.2.0', 'console_scripts', 'haas')()
  File "/home/davidc/src/enthought/enstaller/.env/local/lib/python2.7/site-packages/haas/main.py", line 24, in main
    return application.run()
  File "/home/davidc/src/enthought/enstaller/.env/local/lib/python2.7/site-packages/haas/haas_application.py", line 81, in run
    pattern=args.pattern,
  File "/home/davidc/src/enthought/enstaller/.env/local/lib/python2.7/site-packages/haas/discoverer.py", line 165, in discover
    pattern=pattern)
  File "/home/davidc/src/enthought/enstaller/.env/local/lib/python2.7/site-packages/haas/discoverer.py", line 276, in discover_by_directory
    return self._loader.create_suite(list(tests))
  File "/home/davidc/src/enthought/enstaller/.env/local/lib/python2.7/site-packages/haas/discoverer.py", line 286, in _discover_tests
    yield load_module(get_module_by_name(module_name))
  File "/home/davidc/src/enthought/enstaller/.env/local/lib/python2.7/site-packages/haas/utils.py", line 42, in get_module_by_name
    __import__(name)
ImportError: No module named tmpstore_new.zope.interface.common.tests.test_idatetime

Add very basic parallel test runner

Basic idea:

A test runner that will:

Assume no interaction between tests
Flatten all test suites into a single list of tests to be executed.
Feed tests into a queue
Create a multiprocessing process pool (or similar), each worker of which will act as a consumer of the queue of tests to be executed
The test results are fed into a return queue, consumed by the master to construct the test result

Support customized result output

Update documentation for 0.5.0 release

This includes documenting #16 and #78 (with #88)

Add tox.ini for running tests under all targets Python versions

When filtering tests, always include module import failures

Given this package structure:

package/
|- __init__.py
`- test_something.py

If haas is run as haas -v test_something and there is an import error in test_something, then haas reports that there are no tests.

Add decorator to run individual tests or TestCases in a subprocess.

For example, the code below would run each test method in TestClassInSubprocess in a new subprocess.

from haas.testing import subprocess_test

@subprocess_test
class TestClassInSubprocess(unittest.TestCase):

    def test_foo_in_subprocess(self):
        pass

    def test_bar_in_subprocess(self):
        pass

Design plugin architecture and plugin manager

Add failfast support to parallel runner

Currently if the parallel test runner is used, there is no failfast support; the entire test suite must be run and can't be terminated early.

Add a --version option

It is not possible to discover the haas version from the command line.

Make py2.x discovery find tests that are in non-package subdirectories

Should this be optional?

Haas ignores unittest.TestCase when unittest2 is installed.

Add real plugin documentation

There is no documentation for extending haas.

Installing haas in Python 2.6 does not install the required unittest2

Pin stevedore version for Python 2.6.

stevedore >= 1.10.0 is not compatible with Python 2.6; for Python 2.6 compatibility we need to pin the version.

Should module import failures be sorted to the start of the test suite?

Add inital documentation

haas fails while unittest works on running the traits tests

To reproduce using python 2.7.10:

Checkout the current traits master
Install traits in developer mode (i.e. python setup develop)

run tests using unittest

PS C:\Users\itziakos\Projects\traits> python -m unittest discover
.....................................s.s.........''
...................................................ssssssssssss..................................................................................................sssssssss.......................................................................x.......s................................................................s.............................................................s........................................................s.....ssss...................................s
----------------------------------------------------------------------
Ran 528 tests in 19.106s

OK (skipped=32, expected failures=1)

run tests using haas

PS C:\Users\itziakos\Projects\traits> haas --version
haas 0.8.0.dev486
PS C:\Users\itziakos\Projects\traits> haas
Traceback (most recent call last):
  File "c:\pythons\envs\ets\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\pythons\envs\ets\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "c:\pythons\envs\ets\scripts\haas.exe\__main__.py", line 9, in <module>
  File "c:\pythons\envs\ets\lib\site-packages\haas\main.py", line 24, in main
    return application.run()
  File "c:\pythons\envs\ets\lib\site-packages\haas\haas_application.py", line 115, in run
    for start in args.start
  File "c:\pythons\envs\ets\lib\site-packages\haas\plugins\discoverer.py", line 210, in discover
    pattern=pattern)
  File "c:\pythons\envs\ets\lib\site-packages\haas\plugins\discoverer.py", line 332, in discover_by_directory
    return self._loader.create_suite(list(tests))
  File "c:\pythons\envs\ets\lib\site-packages\haas\plugins\discoverer.py", line 392, in _discover_tests
    yield self._load_from_file(filepath, top_level_directory)
  File "c:\pythons\envs\ets\lib\site-packages\haas\plugins\discoverer.py", line 374, in _load_from_file
    return self._loader.load_module(module)
  File "c:\pythons\envs\ets\lib\site-packages\haas\loader.py", line 122, in load_module
    suites = [self.load_case(case) for case in cases]
  File "c:\pythons\envs\ets\lib\site-packages\haas\loader.py", line 93, in load_case
    for name in self.find_test_method_names(testcase)]
  File "c:\pythons\envs\ets\lib\site-packages\haas\loader.py", line 80, in load_test
    return testcase(methodName=method_name)
TypeError: __init__() got an unexpected keyword argument 'methodName'

Normally traits are tested using nosetests. I was just testing out haas and seen the traceback and then checked with unittest. I have not investigated more.

haas should sort sort tests by name by default for compatibility with unittest

Add an option for random ordering.

Create TestSuite replacement

The unittest.TestSuite class directly manipulates non-public state of the TestResult. This makes it difficult to replace the TestResult.

Create a replacement for the TestSuite to facilitate easier substitution of a different result collector.

The new test suite will make use of external helpers to manage state of the suite (e.g. which test case class is being executed) rather than bundling this onto the result.

TestSuite state is not maintained across suites

If a custom constructed test suite is run where tests from a single class are in seperate sub-suites, then any setUpClass, tearDownClass, setUpModule,tearDownModule methods will be run multiple times even if the actual cases are executed sequentially.

Unable to execute tests in a single file by path

There are two issues here. If there is a leading dot, a ValueError is raised. If there is no leading dot, then no tests are detected.

$ haas ./package/tests/test_case.py
Traceback (most recent call last):
  File "/home/simon/.virtualenvs/venv/bin/haas", line 9, in <module>
    load_entry_point('haas==0.3.1', 'console_scripts', 'haas')()
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/main.py", line 24, in main
    return application.run()
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/haas_application.py", line 87, in run
    for start in args.start
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/haas_application.py", line 87, in <listcomp>
    for start in args.start
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/discoverer.py", line 201, in discover
    pattern=pattern)
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/discoverer.py", line 233, in discover_by_module
    module, case_attributes = find_module_by_name(module_name)
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/discoverer.py", line 81, in find_module_by_name
    module = get_module_by_name(module_name)
  File "/home/simon/.virtualenvs/venv/lib/python3.4/site-packages/haas/utils.py", line 44, in get_module_by_name
    __import__(name)
ValueError: Empty module name

$ haas package/tests/test_case.py

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Disallow test discovery in packages without init.py in Python >= 3.3

As a result of attempting to allow missing __init__.py in Python 3.3 and greater, issue #98 was introduced. We should consider reverting this feature.

Implement failfast option

Crash when there is a directory with a dot in the name.

Discover tests to run only by function name

Given the following dotted path to a test method
haas.tests.test_loader.TestLoadTest.test_creates_instance_for_valid_test
it would be nice to run only this test by simply specifying the following:
python -m haas test_creates_instance_for_valid_test

This would try to discover all tests called test_creates_instance_for_valid_test starting from the current directory.

Support collecting DocTests

Make test discovery pluggable.

unicode error in `_format_exception`

[...]ts\xxx\.devenv\lib\site-packages\haas\result.py", line 198, in inner
    return method(self, *args, **kw)
  File "C:\projects\xxx\.devenv\lib\site-packages\haas\result.py", line 377, in addError
    test, TestCompletionStatus.error, exception=exception)
  File "C:\projects\xxx\.devenv\lib\site-packages\haas\result.py", line 359, in _handle_result
    stderr=stderr,
  File "C:\projects\xxx\.devenv\lib\site-packages\haas\result.py", line 148, in from_test_case
    exception, is_failure, stdout, stderr)
  File "C:\projects\xxx\.devenv\lib\site-packages\haas\result.py", line 84, in _format_exception
    return ''.join(msgLines)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 158: ordinal not in range(128)
[````

parallel test runner doesn't fail on import errors.

If there are module import errors that are converted to error test-cases, the parallel runner simply ignores them, and may "succeed" despite many tests not being run.

Crash if there is a non-import error on test module import

For example, if there is a RuntimeError when importing a test module, the haas test discovery will crash.

$ haas
Traceback (most recent call last):
  File "/home/simon/.virtualenvs/project34/bin/haas", line 11, in <module>
    sys.exit(main())
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/main.py", line 24, in main
    return application.run()
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/haas_application.py", line 87, in run
    for start in args.start
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/haas_application.py", line 87, in <listcomp>
    for start in args.start
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/discoverer.py", line 204, in discover
    pattern=pattern)
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/discoverer.py", line 326, in discover_by_directory
    return self._loader.create_suite(list(tests))
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/discoverer.py", line 381, in _discover_tests
    yield self._load_from_file(filepath, top_level_directory)
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/discoverer.py", line 362, in _load_from_file
    module = get_module_by_name(module_name)
  File "/home/simon/.virtualenvs/project34/lib/python3.4/site-packages/haas/utils.py", line 44, in get_module_by_name
    __import__(name)
  File "/home/simon/workspace/enthought/source/project/project/tests/test_main.py", line 1, in <module>
    raise RuntimeError()
RuntimeError

Support starting coverage internally

Run haas --with-coverage

Discover and load function-based tests

@itziakos pointed out that there is a facility in unittest to generate a test-case based on a top-level function using unittest.FunctionTestCase(callable).

This is something that could easily be added to the haas test discovery system. This will probably start by matching functions names to the regex ^test_.*.

Error running haas on Windows

ERROR:stevedore.extension:error calling 'quiet': 'NoneType' object has no attribute 'enabled'
ERROR:stevedore.extension:'NoneType' object has no attribute 'enabled'
Traceback (most recent call last):
  File "C:\dev\devenv\lib\site-packages\stevedore\extension.py", line 255, in _invoke_one_plugin
    response_callback(func(e, *args, **kwds))
  File "C:\dev\devenv\lib\site-packages\haas\plugin_manager.py", line 96, in _create_hook_plugin
    if plugin.enabled:
AttributeError: 'NoneType' object has no attribute 'enabled'
ERROR:stevedore.extension:error calling 'verbose': 'NoneType' object has no attribute 'enabled'
ERROR:stevedore.extension:'NoneType' object has no attribute 'enabled'
Traceback (most recent call last):
  File "C:\dev\devenv\lib\site-packages\stevedore\extension.py", line 255, in _invoke_one_plugin
    response_callback(func(e, *args, **kwds))
  File "C:\dev\devenv\lib\site-packages\haas\plugin_manager.py", line 96, in _create_hook_plugin
    if plugin.enabled:
AttributeError: 'NoneType' object has no attribute 'enabled'

Haas does not support Python3.4

$ haas -v
Traceback (most recent call last):
  File "/home/vagrant/install/venv/bin/haas", line 9, in <module>
    load_entry_point('haas==0.2.0', 'console_scripts', 'haas')()
  File "/home/vagrant/install/venv/local/lib/python3.4/dist-packages/pkg_resources.py", line 353, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/vagrant/install/venv/local/lib/python3.4/dist-packages/pkg_resources.py", line 2302, in load_entry_point
    return ep.load()
  File "/home/vagrant/install/venv/local/lib/python3.4/dist-packages/pkg_resources.py", line 2029, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/home/vagrant/install/venv/lib/python3.4/site-packages/haas/main.py", line 11, in <module>
    from .haas_application import HaasApplication  # pragma: no cover
  File "/home/vagrant/install/venv/lib/python3.4/site-packages/haas/haas_application.py", line 13, in <module>
    from .loader import Loader
  File "/home/vagrant/install/venv/lib/python3.4/site-packages/haas/loader.py", line 9, in <module>
    from .testing import unittest
  File "/home/vagrant/install/venv/lib/python3.4/site-packages/haas/testing.py", line 21, in <module>
    from unittest.case import _ExpectedFailure, _UnexpectedSuccess
ImportError: cannot import name '_ExpectedFailure'

Add --exclude option to exclude discovered tests from the suite

It would be useful to be able to filter tests out of the discovered suite rather than only having an inclusive filter (i.e. the positional arguments to haas).

The semantics would be the same, filtering on partial names. Multiple --exclude flags are allowed to mirror the multiple positional arguments for include filter.