chrisbeaumont / smother

coverage, just moreso

Smother

Smother is a wrapper utility around coverage.py that measures code coverage separately for each test in a test suite. Its main features include:

• Fast and reliable coverage tracking using coverage.py.
• Ability to lookup which tests visit an arbitrary section of your application code.
• Ability to convert version control diffs into a subset of affected tests to rerun.

Demo

Smother contains plugins for nose and pytest, and behaves similarly to coverage.py:

py.test --smother=my_module test_suite/
or
nosetests --with-smother --smother-package=my_module test_suite/

These commands create a .smother file that can be queried by the smother CLI

smother lookup foo.bar          # which tests visited module foo.bar?
smother lookup foo.bar:120-122  # or just some lines in that file?
smother lookup foo.bar:baz      # or just the `baz` function?

smother diff                    # given local modifications to my repo, what tests might have broken?
smother diff | xargs py.test    # just run them!

smother to_coverage             # build a vanilla .coverage file from a .smother file
smother csv test.csv            # dump all (application, test) pairs to a file

Why?

Smother was designed to make it easier to work with legacy codebases. Such codebases often have several properties that make rapid iteration difficult:

• Long-running test suites. The initial codebase that smother was designed for took nearly 24 hours of CPU time to run its 11K tests. smother diff makes it easier to select a (hopefully much) smaller subset of tests to re-run to quickly identify possible regressions.
• Many different subsystems -- most of which any single developer is unfamiliar with. smother lookup can be used to explore how and where particular modules are invoked in a test suite. These tests often reveal implicit contracts about code behavior that are not obvious from documentation alone.
• Scope creep. Over time the abstractions in codebases become leakier, and logic between different subsystems becomes more heavily coupled. smother csv catalogs the coupling between source code units (lines, functions, or classes) and tests. Exploring this data often yields insights about which subsystems are well-encapsulated, and which would benefit from refactoring.

Full Documentation

Read the Docs

smother's Issues

`smother` is not compatible with `coverage>=5.0.0`

Unfortunately the 5.0 version of coverage breaks smother. Running without coverage enabled, this exception is shown:

../../.pyenv/versions/3.7.9/envs/backend/lib/python3.7/site-packages/smother/pytest_plugin.py:63: in pytest_runtest_teardown
    self.smother.save_context(item.nodeid)
../../.pyenv/versions/3.7.9/envs/backend/lib/python3.7/site-packages/smother/control.py:70: in save_context
    for key, val in self.coverage.collector.data.items()
E   AttributeError: 'Coverage' object has no attribute 'collector'

It seems that they added "Who tests what" in 5.0 of Coverage, which might replace some of the logic required by smother. But either way sounds like it's likely to be a big refactor 😞 .

https://nedbatchelder.com/blog/201810/who_tests_what_is_here.html

Inconsistency between `smother diff` and `smother lookup`

Summary:

Testing a trivial single-line change to a branch that's covered by a few tests, smother lookup correctly spots that the line is only covered by 5 tests, but smother diff returns a much larger set of tests.

Expected behavior:

if smother lookup can detect that a single line is only covered by 5 tests, then if only that line has changed, smother diff should return those 5 tests.

Details:

(backend) ➜  backend git:(test-impact-analysis-smother) ✗ git diff income
diff --git a/income/models/income.py b/income/models/income.py
index 1fced01ac..01e770439 100644
--- a/income/models/income.py
+++ b/income/models/income.py
@@ -399,7 +399,7 @@ class Income(AuditedModel, ExternalIdFieldABC):
         returns a date in Pacific Time.
         """
         if self.is_historical:
-            return convert_to_pacific_timezone(self.paid_timestamp).date()     <--- this is line 402, the only line that has changed.
+            assert False

(backend) ➜  backend git:(test-impact-analysis-smother) ✗ smother lookup income.models.income:402|wc -l
       5
(backend) ➜  backend git:(test-impact-analysis-smother) ✗ smother lookup income.models.income:401|wc -l
     113
(backend) ➜  backend git:(test-impact-analysis-smother) ✗ smother diff|wc -l
     113

As you can see smother diff seems to be "off by one" on mapping the git diff output into smother lookup.

Any further diags I can take here to help narrow this down? (Closed-source repo so unfortunately I can't share the source here, but I could take a few mins to try to repro it on an open-source project like Django).

Support fetching files over URLs

Smother should be able to refer to data files with URLs. These would be downloaded and cached locally.

As a possible extension: smother could have a "URL pattern" config value that describes what URL the smother report for a given source code revision lives at. It would then be smart about obtaining the appropriate report file depending on the context (ie smother diff origin/foo_branch would grab the smother report for foo-branch so that the diff arithmetic is correct).

Support mercurial repositories

smother diff is git-specific. It would be nice to add a DiffReporter class that supports mercurial repositories as well.

portalocker::Lock unexpected keyword argument 'truncate'

$ virtualenv --python=python3 venv
$ source venv/bin/activate
$ pip install -Ur requirements_test.txt
$ make test
python setup.py clean
running clean
removed './smother.egg-info/PKG-INFO'
removed './smother.egg-info/SOURCES.txt'
removed './smother.egg-info/requires.txt'
removed './smother.egg-info/top_level.txt'
removed './smother.egg-info/dependency_links.txt'
removed './smother.egg-info/entry_points.txt'
removed directory './smother.egg-info'
find smother -type d -name "__pycache__" -exec rm -fr "{}" +
find smother -type d -name ".cache" -exec rm -fr "{}" +
find smother -type f -name '*.pyc' -delete
find smother -type f -name '*y.class' -delete
rm -f nosetests.xml
rm -f memory_usage.txt
python setup.py develop
running develop
running egg_info
creating smother.egg-info
writing requirements to smother.egg-info/requires.txt
writing smother.egg-info/PKG-INFO
writing top-level names to smother.egg-info/top_level.txt
writing entry points to smother.egg-info/entry_points.txt
writing dependency_links to smother.egg-info/dependency_links.txt
writing manifest file 'smother.egg-info/SOURCES.txt'
reading manifest file 'smother.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'smother.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/smother/venv/lib/python3.5/site-packages/smother.egg-link (link to .)
Adding smother 0.2 to easy-install.pth file
Installing smother script to /tmp/smother/venv/bin

Installed /tmp/smother
Processing dependencies for smother==0.2
Searching for unidiff
Reading https://pypi.python.org/simple/unidiff/
Downloading https://pypi.python.org/packages/c5/a0/0e0edabce6abece92bc9eb20dd7b60c0dea8d86c6ab6a7a4359b20c517e5/unidiff-0.5.2.tar.gz#md5=20dd70ba5a35bc95bf869322d6852227
Best match: unidiff 0.5.2
Processing unidiff-0.5.2.tar.gz
Writing /tmp/easy_install-dfkb1min/unidiff-0.5.2/setup.cfg
Running unidiff-0.5.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-dfkb1min/unidiff-0.5.2/egg-dist-tmp-gw9tylvp
zip_safe flag not set; analyzing archive contents...
tests.__pycache__.test_parser.cpython-35: module references __file__
creating /tmp/smother/venv/lib/python3.5/site-packages/unidiff-0.5.2-py3.5.egg
Extracting unidiff-0.5.2-py3.5.egg to /tmp/smother/venv/lib/python3.5/site-packages
Adding unidiff 0.5.2 to easy-install.pth file
Installing unidiff script to /tmp/smother/venv/bin

Installed /tmp/smother/venv/lib/python3.5/site-packages/unidiff-0.5.2-py3.5.egg
Searching for portalocker>=0.4
Reading https://pypi.python.org/simple/portalocker/
Downloading https://pypi.python.org/packages/a5/8e/558a0c16fb766b79b122e608481be3037827766f8a54b9eebcfc43d4500e/portalocker-1.0.0.tar.gz#md5=638d163b17a506b951ff2cfaf12fcdc3
Best match: portalocker 1.0.0
Processing portalocker-1.0.0.tar.gz
Writing /tmp/easy_install-om4v9pbb/portalocker-1.0.0/setup.cfg
Running portalocker-1.0.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-om4v9pbb/portalocker-1.0.0/egg-dist-tmp-_ljxgnrj
zip_safe flag not set; analyzing archive contents...

Installed /tmp/easy_install-om4v9pbb/portalocker-1.0.0/.eggs/pytest_runner-2.9-py3.5.egg
warning: no files found matching 'CHANGELOG'
zip_safe flag not set; analyzing archive contents...
Moving portalocker-1.0.0-py3.5.egg to /tmp/smother/venv/lib/python3.5/site-packages
Adding portalocker 1.0.0 to easy-install.pth file

Installed /tmp/smother/venv/lib/python3.5/site-packages/portalocker-1.0.0-py3.5.egg
Searching for more_itertools
Reading https://pypi.python.org/simple/more_itertools/
Downloading https://pypi.python.org/packages/11/f0/7dc530a028b2b18366974960b26c7125872bdba65e703164617003bbf9fc/more-itertools-2.4.1.tar.gz#md5=f6bbfe38a6d0ee1a743adb2c1a4fb6fc
Best match: more-itertools 2.4.1
Processing more-itertools-2.4.1.tar.gz
Writing /tmp/easy_install-gimci_hv/more-itertools-2.4.1/setup.cfg
Running more-itertools-2.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gimci_hv/more-itertools-2.4.1/egg-dist-tmp-ljx6ghie
warning: no files found matching 'fabfile.py'
zip_safe flag not set; analyzing archive contents...
Moving more_itertools-2.4.1-py3.5.egg to /tmp/smother/venv/lib/python3.5/site-packages
Adding more-itertools 2.4.1 to easy-install.pth file

Installed /tmp/smother/venv/lib/python3.5/site-packages/more_itertools-2.4.1-py3.5.egg
Searching for click
Reading https://pypi.python.org/simple/click/
Downloading https://pypi.python.org/packages/7a/00/c14926d8232b36b08218067bcd5853caefb4737cda3f0a47437151344792/click-6.6.tar.gz#md5=d0b09582123605220ad6977175f3e51d
Best match: click 6.6
Processing click-6.6.tar.gz
Writing /tmp/easy_install-i_9eoawj/click-6.6/setup.cfg
Running click-6.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-i_9eoawj/click-6.6/egg-dist-tmp-pqwz8mn4
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
zip_safe flag not set; analyzing archive contents...
click.__pycache__.core.cpython-35: module references __file__
creating /tmp/smother/venv/lib/python3.5/site-packages/click-6.6-py3.5.egg
Extracting click-6.6-py3.5.egg to /tmp/smother/venv/lib/python3.5/site-packages
Adding click 6.6 to easy-install.pth file

Installed /tmp/smother/venv/lib/python3.5/site-packages/click-6.6-py3.5.egg
Searching for six==1.10.0
Best match: six 1.10.0
Adding six 1.10.0 to easy-install.pth file

Using /tmp/smother/venv/lib/python3.5/site-packages
Searching for coverage==4.0.3
Best match: coverage 4.0.3
Adding coverage 4.0.3 to easy-install.pth file
Installing coverage script to /tmp/smother/venv/bin
Installing coverage-3.5 script to /tmp/smother/venv/bin
Installing coverage3 script to /tmp/smother/venv/bin

Using /tmp/smother/venv/lib/python3.5/site-packages
Finished processing dependencies for smother==0.2
smother erase
py.test --smother=smother --smother-cover --smother-append smother
================================================= test session starts ==================================================
platform linux -- Python 3.5.2, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /tmp/smother, inifile: 
plugins: cov-2.4.0, smother-0.2
...
  File "/tmp/smother/smother/control.py", line 120, in write
    fail_when_locked=False
TypeError: __init__() got an unexpected keyword argument 'truncate'
Makefile:47: recipe for target 'test' failed
make: *** [test] Error 1

test_combine_different_root fails with coverage 4.0.3

Not sure why make test installs coverage 4.0.3, but it causes test_combine_different_root to fail:

    def test_combine_different_root():
    
        expected = {
            "test1": {
                "/test-root/smother/tests/demo.py": [8]
            },
            "test2": {
                "/test-root/smother/tests/demo.py": [11, 12]
            },
            "test3": {
                "/test-root/smother/tests/demo.py": [1, 2, 3]
            },
            "test4": {
                "/test-root/smother/tests/demo.py": [4]
            }
        }
    
        runner = CliRunner()
    
        with NamedTemporaryFile(mode='w+') as tf:
            result = runner.invoke(
                cli,
                ['--rcfile', '.parallel_coveragerc',
                 'combine',
                 'smother/tests/.smother',
                 'smother/tests/.smother_3',
                 tf.name
                 ]
            )
    
            assert result.exit_code == 0
            tf.seek(0)
>           assert json.load(tf) == expected
E           AssertionError: assert {'test1': {'s...emo.py': [4]}} == {'test1': {'/t...emo.py': [4]}}
E             Differing items:
E             {'test1': {'smother/tests/demo.py': [8]}} != {'test1': {'/test-root/smother/tests/demo.py': [8]}}
E             {'test2': {'/home-2/smother/tests/demo.py': [11], 'smother/tests/demo.py': [12]}} != {'test2': {'/test-root/smother/tests/demo.py': [11, 12]}}
E             {'test4': {'/home/smother/tests/demo.py': [4]}} != {'test4': {'/test-root/smother/tests/demo.py': [4]}}
E             {'test3': {'smother/tests/demo.py': [1, 2, 3]}} != {'test3': {'/test-root/smother/tests/demo.py': [1, 2, 3]}}
E             Full diff:
E             - {'test1': {'smother/tests/demo.py': [8]},
E             + {'test1': {'/test-root/smother/tests/demo.py': [8]},
E             ?             +++++++++++
E             -  'test2': {'/home-2/smother/tests/demo.py': [11],
E             ?              ^ ^^^^
E             +  'test2': {'/test-root/smother/tests/demo.py': [11, 12]},
E             ?              ^^^^^^ ^^                            ++++ +
E             -            'smother/tests/demo.py': [12]},
E             -  'test3': {'smother/tests/demo.py': [1, 2, 3]},
E             +  'test3': {'/test-root/smother/tests/demo.py': [1, 2, 3]},
E             ?             +++++++++++
E             -  'test4': {'/home/smother/tests/demo.py': [4]}}
E             ?              ^ ^^
E             +  'test4': {'/test-root/smother/tests/demo.py': [4]}}
E             ?              ^^^^^^ ^^

smother/tests/test_cli.py:98: AssertionError

Using coverage 4.4.2 fixes it.

[Docs] Docs in pdf format?

I like to read docs while sitting in the train, etc. Could you generate the docs in pdf format as well?

Support branch coverage

smother will choke on it's own data if coverage is run with branch-tracking enabled

Support combining smother files

cc @AlejandroFrias

Smother should have a combine command similar to coverage combine that can merge several coverage reports from different machines. It should support (or ideally directly reuse) coverage's mechanism for normalizing file paths that change across different machines

Use tox for tests?

Have you considered using https://tox.readthedocs.org/ for tests?

Support for nose2?

I wonder if smother can be made to support nose2, there's a project I'm involved where the testing is performed by nose2.

Tests fail on Windows

When I clone the repo and pip install -e . it, then install the test requirements, then run plain pytest on Windows (the makefile will not work, I expect), in an anaconda install, about half the tests fail.
Some are due to OS-dependent path handling in the expected data:

_______________________ test_convert_to_relative_paths ________________________

    def test_convert_to_relative_paths():
        smother = Smother()
        smother.data = {
            'test1': {os.path.abspath('smother/tests/demo.py'): [8]}
        }

        expected_data = {
            'test1': {'smother/tests/demo.py': [8]}
        }

>       assert Smother.convert_to_relative_paths(smother).data == expected_data
E       AssertionError: assert {'test1': {'s...emo.py': [8]}} == {'test1': {'sm...emo.py': [8]}}
E         Differing items:
E         {'test1': {'smother\\tests\\demo.py': [8]}} != {'test1': {'smother/tests/demo.py': [8]}}
E         Use -v to get the full diff

smother\tests\test_controller.py:107: AssertionError

some are parsing problems, I suspect because of a similar reason (note the additional \\):

____________________________ test_parse[deletion] _____________________________

new = '\ndef foo():\n    def bar():\n        pass\n    pass\n'
expected = [ContextInterval(filename='test.py', context='test:foo')]

    @pytest.mark.parametrize('new,expected', CASES, ids=IDS)
    def test_parse(new, expected):
        diff = TestDiffReporter(old, new)
>       assert list(diff.changed_intervals()) == expected
E       AssertionError: assert [ContextInter...'\\test:foo')] == [ContextInterv...t='test:foo')]
E         At index 0 diff: ContextInterval(filename='test.py', context='\\test:foo') != ContextInterval(filename='test.py', context='test:foo')
E         Use -v to get the full diff

smother\tests\test_diff.py:89: AssertionError

and some tests involving calling external processes fail with FileNotFoundError or CalledProcessError.

Incompatibility with flaky?

Hi!

I am currently playing around with testing hypothesis with smother (see HypothesisWorks/hypothesis#852), and seem to (apparently) run into problems because hypothesis uses flaky, many tests fail with "TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple'" in control.py:70, e.g.

___________ ERROR at teardown of test_errors_when_asked_for_example ___________
C:\Continuum\Anaconda3\envs\hypothesis_test\lib\site-packages\flaky\flaky_pytest_plugin.py:271: in <lambda>
    lambda: ihook(item=item, **kwds),
..\smother\smother\pytest_plugin.py:63: in pytest_runtest_teardown
    self.smother.save_context(item.nodeid)
..\smother\smother\control.py:70: in save_context
    for key, val in self.coverage.collector.data.items()
..\smother\smother\control.py:70: in <dictcomp>
    for key, val in self.coverage.collector.data.items()
E   TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple'

Do you have an idea what's going wrong here?

pytest: error with default (True) for smother_source

When running pytest --smother it will result in an error:

Traceback (most recent call last):
  File "…/Vcs/pytest/_pytest/main.py", line 101, in wrap_session
  File "…/Vcs/pytest/_pytest/config.py", line 912, in _do_configure
  File "…/Vcs/pytest/_pytest/vendored_packages/pluggy.py", line 750, in call_historic
  File "…/Vcs/pytest/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
  File "…/Vcs/pytest/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
  File "…/Vcs/pytest/_pytest/vendored_packages/pluggy.py", line 614, in execute
  File "…/Vcs/smother/smother/pytest_plugin.py", line 30, in pytest_configure
  File "…/Vcs/smother/smother/pytest_plugin.py", line 45, in __init__
  File "…/project/.venv/lib/python3.6/site-packages/coverage/control.py", line 692, in start
  File "…/project/.venv/lib/python3.6/site-packages/coverage/collector.py", line 313, in start
  File "…/project/.venv/lib/python3.6/site-packages/coverage/control.py", line 585, in _should_trace
  File "…/project/.venv/lib/python3.6/site-packages/coverage/control.py", line 530, in _should_trace_internal
  File "…/project/.venv/lib/python3.6/site-packages/coverage/control.py", line 551, in _check_include_omit_etc_internal
  File "…/project/.venv/lib/python3.6/site-packages/coverage/files.py", line 242, in match
TypeError: startswith first arg must be str or a tuple of str, not bool

I think the default for smother_source (or at least what gets passed to coverage.py) should be the current directory, and not a boolean?!