boxed / mutmut Goto Github PK
View Code? Open in Web Editor NEWMutation testing system
Home Page: https://mutmut.readthedocs.io
License: BSD 3-Clause "New" or "Revised" License
mutmut's Issues
Confusing output with Ctrl-C
% mutmut
^C
Aborted!
Running tests without mutations...
It should either display the "Running tests without mutations..." before (assuming it was captured?!), or without any capturing in the first place, or not at all.
Maybe that's confusing me with #31 - that I do not see the "Running tests without mutations..." in the first place?
mutmut==0.0.24 - there is no --version flag?!
AttributeError in argument_mutation
Traceback
Running tests without mutations... Done
Traceback (most recent call last):
File "env/bin/mutmut", line 5, in <module>
main()
File "env/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "env/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "env/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "env/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "env/lib/python3.6/site-packages/mutmut/__main__.py", line 56, in wrapper
f(*args, **kwargs)
File "env/lib/python3.6/site-packages/mutmut/__main__.py", line 317, in main
add_mutations_by_file(mutations_by_file, filename, _exclude, dict_synonyms)
File "env/lib/python3.6/site-packages/mutmut/__main__.py", line 519, in add_mutations_by_file
dict_synonyms=dict_synonyms,
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 363, in list_mutations
mutate(context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 273, in mutate
mutate_list_of_nodes(result, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 320, in mutate_node
children=getattr(i, 'children', None),
File "env/lib/python3.6/site-packages/tri/declarative/__init__.py", line 340, in evaluate
return func_or_value(**kwargs)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 91, in argument_mutation
children[0] = Name(c.value + 'XX', start_pos=c.start_pos, prefix=c.prefix)
AttributeError: 'PythonNode' object has no attribute 'value'
Value of c:
PythonNode(atom, [<Operator: (>, PythonNode(testlist_comp, [PythonNode(atom_expr, [<Name: key@145,24>, PythonNode(trailer, [<Operator: (>, <Name: field@145,28>, <Operator: )>])]), <Operator: ,>, <Keyword: True>]), <Operator: )>])
Constants mutation
Hello!
Can you mutate constants, for example in
for _ in range(1, 10):
to
for _ in range(1, 11):
?
this can help to find bugs.
thank you.
`mutmut --version` does not work (required argument before)
% mutmut --version
Usage: mutmut [OPTIONS] COMMAND [ARGUMENT]
Try "mutmut -h" for help.
Error: Missing argument "COMMAND".
See #36 (comment) for some suggestion to make the click integration nicer in general.. --version would be an option for the main "command" then.
Top-down approach / applying multiple mutations at once
I wonder if it is a good idea to apply mutants top-down, i.e. initially apply all mutants and then remove them similar to bisecting.
This would only require a single test run with all mutants enabled for a project that has no surviving mutants anymore (best case), and would continue by removing half of the mutations etc afterwards, remembering which sets of mutants were caught etc.
The idea is to have less test runs in total.
Just thinking out loud..
More powerful whitelisting system
A lot of surviving mutants in a project are due to changing constants, e.g. FOO = 1, where FOO itself is used in tests then also.
Killing those would need a test where the value itself is used, which does not make much sense, does it?
I think it would be good if those mutations could be skipped (e.g. by not mutating anything that matches [A-Z_]+ maybe?
It would be good if those could be deselected either by name of the mutation (which would need to be something like mutate-constant-X), or by specifying a pattern of identifiers not to mutate.
(it is not possible to deselect mutations currently, is it?)
--use-coverage: No such file or directory: '.coverage'
With pytest:
% mutmut --use-coverage --runner 'python -m pytest testing/test_collection.py' src/_pytest/nodes.py
Running tests without mutations... Done
Using coverage data from .coverage file
Traceback (most recent call last):
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/coverage/data.py", line 292, in read_file
with self._open_for_reading(filename) as f:
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/coverage/data.py", line 306, in _open_for_reading
return open(filename, "r")
FileNotFoundError: [Errno 2] No such file or directory: '.coverage'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "…/Vcs/pytest/.venv/bin/mutmut", line 5, in <module>
main()
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 55, in wrapper
f(*args, **kwargs)
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 215, in main
coverage_data = read_coverage_data(use_coverage)
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 402, in read_coverage_data
coverage_data.read_file('.coverage')
File "…/Vcs/pytest/.venv/lib/python3.6/site-packages/coverage/data.py", line 297, in read_file
filename, exc.__class__.__name__, exc,
coverage.misc.CoverageException: Couldn't read data from '.coverage': FileNotFoundError: [Errno 2] No such file or directory: '.coverage'
mutmut doesn't seem to work on windows
It would appear that there are hard coded paths that prevent mutmut from running on windows. Would the accepted solution here be an acceptable fix?
>mutmut-cmd --tests-dir execution\tests execution Traceback (most recent call last):
File "C:\Users\fmoor\.venv\barometer\Scripts\mutmut-cmd.py", line 185, in <module>
main()
File "c:\users\fmoor\.venv\barometer\lib\site-packages\click\core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "c:\users\fmoor\.venv\barometer\lib\site-packages\click\core.py", line 697, in main
rv = self.invoke(ctx)
File "c:\users\fmoor\.venv\barometer\lib\site-packages\click\core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\fmoor\.venv\barometer\lib\site-packages\click\core.py", line 535, in invoke
return callback(*args, **kwargs)
File "c:\users\fmoor\.venv\barometer\lib\site-packages\mutmut\__init__.py", line 35, in wrapper
f(*args, **kwargs)
File "C:\Users\fmoor\.venv\barometer\Scripts\mutmut-cmd.py", line 73, in main
null_stdout = open('/dev/null', 'w') if not s else None
FileNotFoundError: [Errno 2] No such file or directory: '/dev/null'Windows 10 Enterprise
Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
mutmut (0.0.3)
Running with --use-coverage fails
Hello! I am trying out mutmut today, having tried out cosmic-ray recently, and mutmut appears to be a simpler implementation, so thanks for making this!
My project uses pytest, and leverages coverage, and during my initial mutmut run, it found mutants in a function that I have a pragma: no cover set, so I wanted to use the flag to leverage only covered lines for mutation (there's also a couple of private methods that aren't exercised).
When invoking:
mutmut run --use-coverage --paths-to-mutate unsilencer.py
I get this traceback:
1. Using cached time for baseline tests, to run baseline again delete the cache file
Using coverage data from .coverage file
Traceback (most recent call last):
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/bin/mutmut", line 5, in <module>
main()
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/mutmut/__main__.py", line 59, in wrapper
f(*args, **kwargs)
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/mutmut/__main__.py", line 285, in main
coverage_data = read_coverage_data(use_coverage)
File "/Users/miketheman/.local/share/virtualenvs/unsilencer-YmPJJ04A/lib/python3.7/site-packages/mutmut/__main__.py", line 484, in read_coverage_data
coverage_data.read_file('.coverage')
AttributeError: 'CoverageSqliteData' object has no attribute 'read_file'Reference:
Lines 482 to 484 in a3c5efa
It appears that the read_file may have been removed upstream? I'm using pytest-cov which installs anything above 4.4, and using 5.0a4 the method is removed.
Manually back-tracking to coverage 4.5.2 (latest stable) provides a non-traceback execution, however it doesn't appear to respect the coverage rules, as it's mutating a no cover line, like so:
$ mutmut show 26
--- unsilencer.py
+++ unsilencer.py
@@ -77,7 +77,7 @@
remove_from_list(suppression_list, email_address)
-if __name__ == "__main__": # pragma: no cover
+if __name__ == "XX__main__XX": # pragma: no cover
try:
input = sys.argv[1]
except IndexError:is this the desired behavior? I couldn't find any reference to coverage in the test/s directory, hoping to see how this is meant to work. Let me know if there's anything else I can do to help!
OSError raised when waiting for pytest to finish
I checked out master/commit 63089e9 to try the new UI (pretty nice!) but ran into trouble:
mutmut appears to be waiting for a spawned process's returncode to be set, continually asking for a line from stdout until this condition holds. However, it appears that, at some point:
- p.returncode is still None
- stdout.readline() raises an OSError
Wrapping the readline() call in a try/except pass seems to fix the issue. It let the initial test run complete, and then started counting up the killed/survived totals.
Stacktrace:
⠙ Running...Traceback (most recent call last):
File "/home/flux/anaconda3/bin/mutmut", line 5, in <module>
main()
File "/home/flux/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/home/flux/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/flux/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/flux/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/flux/anaconda3/lib/python3.6/site-packages/mutmut/__main__.py", line 57, in wrapper
f(*args, **kwargs)
File "/home/flux/anaconda3/lib/python3.6/site-packages/mutmut/__main__.py", line 236, in main
baseline_time_elapsed = time_test_suite(swallow_output=not s, test_command=runner, using_testmon=using_testmon)
File "/home/flux/anaconda3/lib/python3.6/site-packages/mutmut/__main__.py", line 431, in time_test_suite
returncode = popen_streaming_output(test_command, feedback)
File "/home/flux/anaconda3/lib/python3.6/site-packages/mutmut/__main__.py", line 306, in popen_streaming_output
line = stdout.readline()[:-1] # -1 to remove the newline at the end
OSError: [Errno 5] Input/output error
The bandaid I applied change line 306 to look like this:
try:
line = stdout.readline()[:-1] # -1 to remove the newline at the end
except OSError:
pass
The except block is only hit once. p.returncode appears to be set properly afterwards, and the program carries on.
My Python version is Python 3.6.5 :: Anaconda custom (64-bit), if that's relevant.
"Surviving" notation backwards?
Hello. First let me say I'm super excited for this repo! I literally learned about mutation testing today and immediately pip installed your package.
What's confusing me is what you call a "surviving mutant." My expectation was that surviving mutants would be those that still pass the test suite. However, line 388 defines surviving mutants as those that do not pass the test suite. This means mutations that aren't a problem wind up being reported as "FAILED," and I haven't found a way to show the mutations that have left the test in a passing state.
TL;DR I want to know which mutants passed my test suite, not which ones don't pass my test suite.
My proposal is to remove not from line 388, but this drastically changes the behavior of this code base. Like I said, I'm a mutation testing noob, so it's entirely possible I'm thinking about this the wrong way.
Thanks again for the awesome repo!
`mutations show <id>` results in `No mutation performed`
Thesis
Some mutations are marked as failed. However, they are not making any changes to the source code.
And the tests are working. So, there's an issue somewhere.
How to reproduce
Steps to reproduce (however, I am not sure that this can be reproduced with 100%):
- Follow the installation steps of https://github.com/wemake-services/wemake-django-template
- Install dependencies + install
mutmut(it is not included) - Add this lines to the
setup.cfg:
[mutmut]
paths_to_mutate = server/main_app
backup = False
runner = pytest
tests_dir = tests/- Run
mutmut run
Related output
(.venv) ~/Desktop/test_project
» mutmut run
- Mutation testing starting -
These are the steps:
1. A full test suite run will be made to make sure we
can run the tests successfully and we know how long
it takes (to detect infinite loops for example)
2. Mutants will be generated and checked
Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut results`.
Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.
1. Running tests without mutations
⠙ Running... Done
2. Checking mutants
⠦ 6/6 🎉 2 ⏰ 0 🤔 0 🙁 4
(.venv) ~/Desktop/test_project
» mutmut results
Timed out ⏰
Suspicious 🤔
Survived 🙁
mutmut apply 1
mutmut apply 2
mutmut apply 3
mutmut apply 4
(.venv) ~/Desktop/test_project
» mutmut show 1
No mutation performed
(.venv) ~/Desktop/test_project
» mutmut show 2
No mutation performed
(.venv) ~/Desktop/test_project
» mutmut show 3
--- server/main_app/urls.py
+++ server/main_app/urls.py
@@ -7,6 +7,6 @@
# Place your URLs here:
urlpatterns = [
- url(r'^hello/$', index, name='hello'),
+ url(r'XX^hello/$XX', index, name='hello'),
]
(.venv) ~/Desktop/test_project
» mutmut show 4
--- server/main_app/urls.py
+++ server/main_app/urls.py
@@ -7,6 +7,6 @@
# Place your URLs here:
urlpatterns = [
- url(r'^hello/$', index, name='hello'),
+ url(r'^hello/$', index, name='XXhelloXX'),
]
Originally posted as #31 (comment)
Whitelist string split/rsplit optimizations
foo = 'a/b/c'.split('/', 1)[0]the parameter 1 to split can be mutated to 2. But this is an optimization so it's very annoying to have to whitelist this. Same goes for rsplit.
Mutate only changes from a patch
I think a very useful feature would be to only mutate code that has been changed in a patch (e.g. what is in the Git index etc).
This way you could run mutmut before/after committing changes to check if everything is covered with regard to mutations in a new commit (which not enforcing checks on the whole code base).
Add support for unittest-based Django tests
I'm attempting to run mutmut inside a Django project. There are certain assumptions in the codebase that I need to change in order to achieve that.
I will use this place to report my progress and welcome any ideas.
Applied mutation causes syntax error
Hi,
mutmut version: 1.1.0
I'm using Python 3.7 so in my code base I'm using this syntax:
def except_to_bool(_func=None, *, exc=Exception, to=False):
When I apply mutmut it generates this mutant:
def except_to_bool(_func=None, /, exc=Exception, to=False):
which should not be generated as it produces syntax error.
If mutant should be generated here I imagine it should look like this:
def except_to_bool(_func=None, exc=Exception, to=False):
(simply remove star that enforces keyword arguments)
I even have test for this code (which is checking whether it is not possible to pass positional arguments to this function), but mutant still survives it. I think this is because mutant causes syntax error.
Either fix mutant generation with star, that enforces keyword arguments, or skip generating mutant at all in such case, please :)
Current state of tests is not great, let's fix them
As the title says current state of tests is not where it should be especially considering the purpose of the mutmut itself is to improve unit tests. I feel that we need to address these issues sooner rather than later since it will make maintenance much easier in the future.
- Current tests are more integration / end-to-end tests than unit tests
- Big chunk of code is not properly unit tested
mutmutis not used onmutmut
Current tests are more integration / end-to-end tests than unit tests
Here is the article by Michael Feathers that pretty much explains this point. Each unit test should test only one unit of the code, also no spinning disks for unit tests and no IO at all. Some parts of the code will, for sure, require refactoring to make them testable. There is a lot of IO in functions that should be moved up a level to make the code easy to test. Here is a great talk by Brandon Rhodes on how to "hoist IO".
Big chunk of code is not properly unit tested
It is hard to determine actual unit test coverage since coverage is calculated while running all tests. Currently it looks good, but there is a decent amount of edge cases that are not tested because it is hard to reproduce edge cases by writing integration tests. Coverage should be calculated when running only true unit tests.
mutmut is not used on mutmut
This is the best way to make our unit tests better, that's what mutmut does. Also this would make it easier for us to spot new use cases and potential improvements. But first, things need to be refactored.
Steps to mitigate:
- Making a clear separation between unit tests and integration / end-to-end tests. (different dirs/files and
pytestmark) - Stop merging pull requests that aren't properly covered with unit tests.
- Chunk by chunk refactor code to be testable and add appropriate unit tests for it.
- After the old code is refactored, switch coverage to work only with unit tests.
- Start using
mutmuton unit tests
Good thing is that code base is not that big, it is a great time to do something like this. Also current tests are good for what they do, we just need to add proper unit tests.
Please let me know what you think. :)
Install error
I was trying to do an initial install of mutmut this morning, and I failed with an error. Here is the entire stream:
Admins-MacBook-Air:~ shammond$ sudo -H pip install mutmut
Collecting mutmut
Downloading https://files.pythonhosted.org/packages/a4/68/fe0bec09d1685a58eb5b84c4caf75a0b25d16ac2355479ed14e639e7671a/mutmut-0.0.20.tar.gz
Requirement already satisfied: parso in /Library/Python/2.7/site-packages (from mutmut) (0.3.1)
Requirement already satisfied: tri.declarative in /Library/Python/2.7/site-packages (from mutmut) (1.0.3)
Requirement already satisfied: click in /Library/Python/2.7/site-packages (from mutmut) (6.7)
Requirement already satisfied: tri.struct in /Library/Python/2.7/site-packages (from tri.declarative->mutmut) (2.5.5)
Installing collected packages: mutmut
Running setup.py install for mutmut ... error
Complete output from command /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -u -c "import setuptools, tokenize;file='/private/tmp/pip-install-K6Bg3Q/mutmut/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/tmp/pip-record-EfYAk8/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/mutmut
copying ./mutmut/init.py -> build/lib/mutmut
copying ./mutmut/main.py -> build/lib/mutmut
running egg_info
writing requirements to mutmut.egg-info/requires.txt
writing mutmut.egg-info/PKG-INFO
writing top-level names to mutmut.egg-info/top_level.txt
writing dependency_links to mutmut.egg-info/dependency_links.txt
reading manifest file 'mutmut.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'mutmut.egg-info/SOURCES.txt'
running build_scripts
creating build/scripts-2.7
copying and adjusting bin/mutmut -> build/scripts-2.7
changing mode of build/scripts-2.7/mutmut from 644 to 755
running install_lib
running install_data
warning: install_data: setup script did not provide a directory for 'README.rst' -- installing right in '/System/Library/Frameworks/Python.framework/Versions/2.7'
copying README.rst -> /System/Library/Frameworks/Python.framework/Versions/2.7
error: [Errno 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/README.rst'
----------------------------------------
Command "/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -u -c "import setuptools, tokenize;file='/private/tmp/pip-install-K6Bg3Q/mutmut/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/tmp/pip-record-EfYAk8/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/tmp/pip-install-K6Bg3Q/mutmut/
Indicate result via exit code
Currently mutmut run and result always exits with code 0.
It would be very useful (e.g. for inclusion in CI systems) if both could give a return code depending on the type of found issues.
Option to skip initial test run
This is useful for testing scenarios and to speed up re-running stuff if you're pretty sure this step will succeed. The downside is of course that mutmut can't detect excessively long test runs.
Be more verbose
There should be some output when running mutmut by default.
I am just trying it out (seen it via tarpas/pytest-testmon#76), but it seems to be very silent during the initial run for example.
It would be nice to just get the normal output there - which should include wrapping in pytest-sugar if it is installed/enabled.
Mutating strings in type annotations
This is a special case from #28
I have some functions that use strings to annotate types. And this is a valid use case.
Like this example: https://github.com/wemake-services/wemake-python-styleguide/blob/master/wemake_python_styleguide/visitors/base.py#L65
But, mutmut tries to mutate this string. And tests do not fail.
Once again, I guess I have to run mypy as well. But in this case it will take too long.
Maybe we can have an option to switch off mutating type annotations?
Support for HTML reports
While looking at mutation libraries for other languages I found out that PIT (Java) has a really nice HTML report output. I think that feature like this would make mutmut easier to grasp especially for people new to mutation testing.
Also mutmut is difficult to work with when you have a lot of mutations, like:
mutmut show 58
mutmut show 59
It seems like a lot of work, but I want to hear opinions do you guys think this would be worth doing.
Any other ideas that would improve result reading experience are welcome.
the halting problem: when a mutation triggers an infinite loop
This is the simplest case I could come up with, some timeout handling is needed it seems?
t.py
def t(x):
r = 0
while x:
r += 2 * x
x -= 1
return rtests/test_t.py
from t import t
def test():
assert t(3) == 12output
$ mutmut t.py
Running tests without mutations... Done
--- starting mutation ---
6 out of 7 (t.py x -= 1⤑0)
(and then a (presumably) indefinite hang)
Output: final newline / "old line"
With the following output there are some problems:
- what does
old linemean? Should be removed or replaced with some more descriptive message (in verbose mode only?) - it is missing a final newline - as you can see since Zsh prints timing information in the same line
- it displays "1484 out of 133", i.e. the count / total appears to be off. It would be good to see what it refers to (mutations? files?)
% mutmut --use-coverage --runner 'python -m pytest testing/test_collection.py' src/_pytest/nodes.py
Running tests without mutations... Done
Using coverage data from .coverage file
--- starting mutation ---
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " return self.fspath, None, \"\"⤑0"
…
old line
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " cache = self.config.__dict__.setdefault(\"_bestrelpathcache\", {})⤑0"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " fspath = cache[location[0]]⤑0"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " fspath = cache[location[0]]⤑1"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " fspath = self.session.fspath.bestrelpath(location[0])⤑0"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " fspath = self.session.fspath.bestrelpath(location[0])⤑1"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " cache[location[0]] = fspath⤑0"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " cache[location[0]] = fspath⤑1"
FAILED: mutmut src/_pytest/nodes.py --apply --mutation " location = (fspath, location[1], str(location[2]))⤑0"
1484 out of 133 (src/_pytest/nodes.py self._location = location⤑0) mutmut --use-coverage --runner 'python -m pytest testing/test_collection.py' 341.91s user 16.20s system 98% cpu 6:02.87 total
Wrong "No mutation performed" message while another mutation is applied
While mutation 6107 was running I've tried mutmut show 6106 (the last failed one), and it said "No mutation performed".
However that was likely only due to the next mutation being applied already, which touched the same code.
The below commands were run later (after 6107 finished):
% mutmut show 6106
--- project/app/models.py
+++ project/app/models.py
@@ -113,7 +113,7 @@
…
-FOO = 'BAR' # legacy
+FOO = 'XXBARXX' # legacy
…
% mutmut show 6107
--- project/app/models.py
+++ project/app/models.py
@@ -113,7 +113,7 @@
…
-FOO = 'BAR' # legacy
+FOO = None # legacy
…
This indicates that mutmut show goes to the actual files contents or something like that?!
Mutating type aliases
We use mypy and type aliases a lot in my company.
And it works perfectly.
However, on lines like, (full source):
AnyImport = Union[ast.Import, ast.ImportFrom]I see a failing mutation test. And that's kind of correct, since types do not affect runtime.
FAILED: mutmut wemake_python_styleguide/types.py --apply --mutation "AnyImport = Union[ast.Import, ast.ImportFrom]⤑0"
What can be done to fix it?
# pragma: no mutateto ignore a single item. But, I guess it kind of user-unfriendly to force this pragma on all type aliases- also run
mypyafter (or before) the actual tests - automatically detect these aliases and ignore them (unsupported)
What do you think / recommend? Thanks!
Can't run mutmut
I have this error.
command:
mutmut run --paths-to-mutate=$PROJECT --tests-dir=$TESTS_DIR
Traceback (most recent call last):
File "/usr/local/bin/mutmut", line 5, in <module>
climain()
File "/usr/local/lib/python3.5/site-packages/click/core.py", line 695, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/click/core.py", line 654, in main
_verify_python3_env()
File "/usr/local/lib/python3.5/site-packages/click/_unicodefun.py", line 69, in _verify_python3_env
if locale.lower().endswith(('.utf-8', '.utf8')):
TypeError: a bytes-like object is required, not 'str'
Docker:
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64
Experimental: false
Docker Image:
amazonlinux:2017.09
requirements.txt:
...
mutmut==1.3.1
pytest==4.2.0
Cache updated incorrectly when running mutmut for single mutant, and maybe when rerunning generally
If you have an existing .mutmut-cache directory and retest just one mutant, this mutant is never removed from the cached mutants list. I think it's also incorrect when rerunning generally.
Handle errors from baron better
I've seen an error via baron when running mutmut, which seems to display the
same exception twice, and provides no reference to the source:
% mutmut project --tests-dir project/app/tests
Traceback (most recent call last):
File "…/project/.venv/lib/python3.6/site-packages/baron/baron.py", line 20, in _parse
return parser(tokens)
File "…/project/.venv/lib/python3.6/site-packages/baron/grammator.py", line 695, in parse
return parser.parse(iter(tokens))
File "…/project/.venv/lib/python3.6/site-packages/baron/parser.py", line 168, in parse
raise ParsingError(debug_output)
baron.parser.ParsingError: Error, got an unexpected token DOUBLE_STAR here:
1381 …
1382 …
1383 …
1384 …
1385 …
1386 …
1387
1388 foo_kwargs = {**<---- here
The token DOUBLE_STAR should be one of those: BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, COMPLEX, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, HEXA, INT, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SQUARE_BRACKET, LONG, MINUS, NAME, NOT, OCTA, PLUS, RAW_STRING, RIGHT_BRACKET, STRING, TILDE, UNICODE_RAW_STRING, UNICODE_STRING
It is not normal that you see this error, it means that Baron has failed to parse valid Python code. It would be kind if you can extract the snippet of your code that makes Baron fail and open a bug here: https://github.com/Psycojoker/baron/issues
Sorry for the inconvenience.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "…/project/.venv/bin/mutmut", line 5, in <module>
main()
File "…/project/.venv/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "…/project/.venv/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "…/project/.venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "…/project/.venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "…/project/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 35, in wrapper
f(*args, **kwargs)
File "…/project/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 135, in main
mutations_by_file[filename] = count_mutations(open(filename).read(), context__filename=filename, context__exclude=exclude)
File "…/project/.venv/lib/python3.6/site-packages/tri/declarative/__init__.py", line 587, in wrapper
return f(*args, **setdefaults_path(Namespace(), kwargs, defaults))
File "…/project/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 365, in count_mutations
return mutate(source, ALL, context=context)[1]
File "…/project/.venv/lib/python3.6/site-packages/tri/declarative/__init__.py", line 587, in wrapper
return f(*args, **setdefaults_path(Namespace(), kwargs, defaults))
File "…/project/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 282, in mutate
result = parse(source)
File "…/project/.venv/lib/python3.6/site-packages/baron/baron.py", line 66, in parse
return _parse(tokens, print_function)
File "…/project/.venv/lib/python3.6/site-packages/baron/baron.py", line 26, in _parse
raise e
File "…/project/.venv/lib/python3.6/site-packages/baron/baron.py", line 24, in _parse
return parser(tokens)
File "…/project/.venv/lib/python3.6/site-packages/baron/grammator.py", line 695, in parse
return parser.parse(iter(tokens))
File "…/project/.venv/lib/python3.6/site-packages/baron/parser.py", line 168, in parse
raise ParsingError(debug_output)
baron.parser.ParsingError: Error, got an unexpected token DOUBLE_STAR here:
1381 …
1382 …
1383 …
1384 …
1385 …
1386 …
1387
1388 foo_kwargs = {**<---- here
The token DOUBLE_STAR should be one of those: BACKQUOTE, BINARY, BINARY_RAW_STRING, BINARY_STRING, COMPLEX, FLOAT, FLOAT_EXPONANT, FLOAT_EXPONANT_COMPLEX, HEXA, INT, LAMBDA, LEFT_BRACKET, LEFT_PARENTHESIS, LEFT_SQUARE_BRACKET, LONG, MINUS, NAME, NOT, OCTA, PLUS, RAW_STRING, RIGHT_BRACKET, STRING, TILDE, UNICODE_RAW_STRING, UNICODE_STRING
It is not normal that you see this error, it means that Baron has failed to parse valid Python code. It would be kind if you can extract the snippet of your code that makes Baron fail and open a bug here: https://github.com/Psycojoker/baron/issues
Sorry for the inconvenience.
cache: take runner into account? / use args for "run"
I've used mutmut run --use-coverage --runner 'python -m pytest -x testing/test_collection.py' --paths-to-mutate src/_pytest/main.py first.
The first survived mutant was:
% mutmut show 1
EXIT_OK = 0
|
V
EXIT_OK = 1
I've then created a test_mutmut.py file, where I've checked that EXIT_OK == 0, but it appears like everything was handled through the cache (according to the speed, there is no actual feedback/count for that).
I assume that using a test file through the --runner argument is considered to be a hack, but I think it is rather important which runner (options) were used, and therefore it should be taken into account for the cache key.
I've seen that there is --tests-dir - maybe that should become --tests-arg?
I've tried using --tests-dir testing/test_mutmut.py, but it appears to run the whole tests?!
(the actual command being used would be useful to see with "1. Running tests without mutations" here)
When using run as a click sub-command, it could take any non-option arguments and append them to the --runner command maybe, making --tests-dir obsolete?!
mutmut cli doesn't work on windows
It appears that the mutmut command line script is just called mutmut. On windows this doesn't work. The script has to have an extension i.e. *.py. I renamed the cli to mutmut.py and this didn't work either because the cli name was now colliding with the mutmut library. Changing the name again to mutmut-cmd.py fixed the collision.
Windows 10 Enterprise
Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
mutmut (0.0.3)
`IndexError` inside `expression_mutation`
Hi, thanks for this module! I am trying to use it with my app: https://github.com/wemake-services/wemake-python-styleguide
But, it is failing. Output:
» mutmut
Running tests without mutations... Done
Traceback (most recent call last):
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/bin/mutmut", line 5, in <module>
main()
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 55, in wrapper
f(*args, **kwargs)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 225, in main
add_mutations_by_file(mutations_by_file, filename, _exclude, dict_synonyms)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__main__.py", line 417, in add_mutations_by_file
dict_synonyms=dict_synonyms,
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 373, in list_mutations
mutate(context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 286, in mutate
mutate_list_of_nodes(result, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 352, in mutate_list_of_nodes
mutate_node(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 311, in mutate_node
mutate_list_of_nodes(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 352, in mutate_list_of_nodes
mutate_node(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 311, in mutate_node
mutate_list_of_nodes(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 352, in mutate_list_of_nodes
mutate_node(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 311, in mutate_node
mutate_list_of_nodes(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 352, in mutate_list_of_nodes
mutate_node(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 311, in mutate_node
mutate_list_of_nodes(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 352, in mutate_list_of_nodes
mutate_node(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 311, in mutate_node
mutate_list_of_nodes(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 352, in mutate_list_of_nodes
mutate_node(i, context=context)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 330, in mutate_node
children=getattr(i, 'children', None),
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/tri/declarative/__init__.py", line 341, in evaluate
return func_or_value(**kwargs)
File "/Users/sobolev/Documents/github/wemake-python-styleguide/.venv/lib/python3.6/site-packages/mutmut/__init__.py", line 184, in expression_mutation
if children[2].value == '=':
IndexError: list index out of range
Versions
» pip freeze
Could not find setup.py for directory /Users/sobolev/Documents/github/wemake-python-styleguide (tried all parent directories)
added-value==0.8
alabaster==0.7.12
atomicwrites==1.2.1
attrs==18.2.0
Babel==2.6.0
bandit==1.5.1
certifi==2018.10.15
chardet==3.0.4
Click==7.0
coverage==4.5.1
dataclasses==0.6
doc8==0.8.0
docutils==0.14
eradicate==0.2.1
flake8==3.6.0
flake8-bandit==1.0.2
flake8-broken-line==0.1.0
flake8-bugbear==18.8.0
flake8-builtins==1.4.1
flake8-coding==1.3.1
flake8-commas==2.0.0
flake8-comprehensions==1.4.1
flake8-debugger==3.1.0
flake8-docstrings==1.3.0
flake8-eradicate==0.1.1
flake8-isort==2.5
flake8-logging-format==0.5.0
flake8-pep3101==1.2.1
flake8-polyfill==1.0.2
flake8-print==3.1.0
flake8-pytest==1.3
flake8-quotes==1.0.0
flake8-string-format==0.2.3
flake8-type-annotations==0.1.0
gitdb2==2.0.5
GitPython==2.1.11
glob2==0.6
idna==2.7
imagesize==1.1.0
isort==4.3.4
Jinja2==2.10
m2r==0.2.1
MarkupSafe==1.0
mccabe==0.6.1
mistune==0.8.4
more-itertools==4.3.0
mutmut==0.0.22
mypy==0.641
mypy-extensions==0.4.1
packaging==18.0
parso==0.3.1
pbr==5.1.0
pep8-naming==0.7.0
pluggy==0.8.0
pockets==0.7.2
py==1.7.0
pycodestyle==2.4.0
pydocstyle==3.0.0
pyflakes==2.0.0
Pygments==2.2.0
Pympler==0.6
pyparsing==2.2.2
pytest==3.9.3
pytest-cov==2.6.0
pytest-flake8==1.0.2
pytest-isort==0.2.1
pytest-randomly==1.2.3
pytz==2018.7
PyYAML==3.13
requests==2.20.0
restructuredtext-lint==1.1.3
six==1.11.0
smmap2==2.0.5
snowballstemmer==1.2.1
Sphinx==1.8.1
sphinx-autodoc-typehints==1.3.0
sphinx-readable-theme==1.3.0
sphinxcontrib-napoleon==0.7
sphinxcontrib-websupport==1.1.0
stevedore==1.30.0
testfixtures==6.3.0
tomlkit==0.4.6
tri.declarative==1.0.6
tri.struct==2.5.6
typed-ast==1.1.0
typing==3.6.6
typing-extensions==3.6.6
urllib3==1.24
-e git+https://github.com/wemake-services/wemake-python-styleguide.git@8b9cfd896ae500e99a5a47a5f10386150a26082c#egg=wemake_python_styleguide
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
I am using:
python3.6poetry@12
Configuration
I have added these values to my setup.cfg:
[mutmut]
paths_to_mutate = wemake_python_styleguide/
backup = False
runner = pytest
tests_dir = tests/
Any ideas?
Error on run with empty source
Hi,
I've got a problem when running mutmut v1.3.0. IndexError: string index out of range.
Full stack trace:
1. Using cached time for baseline tests, to run baseline again delete the cache file
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mutmut/__main__.py", line 675, in <module>
climain()
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 722, in __call__
return self.main(*args, **kwargs)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mutmut/__main__.py", line 66, in wrapper
f(*args, **kwargs)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mutmut/__main__.py", line 204, in climain
version, suspicious_policy, untested_policy))
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mutmut/__main__.py", line 331, in main
add_mutations_by_file(mutations_by_file, filename, _exclude, dict_synonyms)
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mutmut/__main__.py", line 595, in add_mutations_by_file
dict_synonyms=dict_synonyms,
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mutmut/__init__.py", line 417, in __init__
if source is not None and source[-1] != '\n':
IndexError: string index out of range
I've tried it on this codebase by running python3 -m mutmut run --runner="python3 -m pytest" --paths-to-mutate="vakt/" --dict-synonyms="Struct, NamedStruct".
Seems this is caused by empty __init__.py files.
Possible fix:
in mutmut/__ init __.py do
class Context(object):
def __init__(self, source=None, mutation_id=ALL, dict_synonyms=None, filename=None, exclude=lambda context: False, config=None):
self.index = 0
self.remove_newline_at_end = False
if source is not None and source != '' and source[-1] != '\n':
source += '\n'IndexError in argument_mutation function
Stacktrace:
File "env/lib/python3.6/site-packages/mutmut/__main__.py", line 317, in main
add_mutations_by_file(mutations_by_file, filename, _exclude, dict_synonyms)
File "env/lib/python3.6/site-packages/mutmut/__main__.py", line 520, in add_mutations_by_file
dict_synonyms=dict_synonyms,
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 363, in list_mutations
mutate(context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 271, in mutate
mutate_list_of_nodes(result, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 301, in mutate_node
mutate_list_of_nodes(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 342, in mutate_list_of_nodes
mutate_node(i, context=context)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 320, in mutate_node
children=getattr(i, 'children', None),
File "env/lib/python3.6/site-packages/tri/declarative/__init__.py", line 340, in evaluate
return func_or_value(**kwargs)
File "env/lib/python3.6/site-packages/mutmut/__init__.py", line 80, in argument_mutation
elif context.stack[-4].type == 'power':
IndexError: list index out of range
Value of context.stack:
[PythonNode(decorated, [<Decorator: @register.simple_tag(name='XXiconXX')@14,0>, <Function: icon@15-20>]), <Decorator: @register.simple_tag(name='XXiconXX')@14
,0>, PythonNode(argument, [<Name: name@14,21>, <Operator: =>, <String: 'XXiconXX'>])]
Relevant source code part:
@register.simple_tag(name='icon')
def icon(name):
if name is None:
return ''
tpl = '<span class="glyphicon glyphicon-{}"></span>'
return format_html(tpl, name)Ctrl-C needs to be used twice to interrupt
2. Checking mutants
⠇ 7/282 🎉 3 ⏰ 0 🤔 0 🙁 4^C
Aborted!
^CException ignored in: <module 'threading' from '/usr/lib/python3.7/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 1273, in _shutdown
t.join()
File "/usr/lib/python3.7/threading.py", line 1032, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
mutmut 1.0.0.
No mutations when running with --use-coverage
Steps to reproduce:
- In an empty directory, create the file
example_test.pywith the following content:
def test_stuff():
assert True- Run
pytest --covto generate.coveragefile. - Run
mutmut run --paths-to-mutate=example_test.py --use-coverage.
Expected result: 1 killed mutant (I tried running mutmut without --use-coverage and there was indeed 1 mutant; and coverage html shows that that line is covered).
Actual result: no mutants whatsoever.
1. Running tests without mutations
⠇ Running... Done
Using coverage data from .coverage file
2. Checking mutants
⠏ 0/0 🎉 0 ⏰ 0 🤔 0 🙁 0
Version info:
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
pytest==4.1.1
pytest-cov==2.6.1
mutmut==1.2.0
Python 3 supported?
Is mutmut supposed to work with Python 3? On Mac OS X, using Python 3.6.1 I get this:
$ pip3 install mutmut
...
$ mutmut --help
File "/Library/Frameworks/Python.framework/Versions/3.6/bin/mutmut", line 47
print 'ERROR: no mutations performed. Are you sure the index is not too big?'
^
SyntaxError: Missing parentheses in call to 'print'
Github link in documentation is broken
Currently, the link at the bottom of index.html is attempting to take users to https://github.com/boxed/github, rather than https://github.com/boxed/mutmut.
Improving output: verbose mode
How currently the output looks like?
900 out of 1067 (wemake_python_styleguide/violations/consistency.py should_use_text = FAILED: mutmut wemake_python_styleguide/violations/consistency.py --apply --mutation " should_use_text = False⤑1"
901 out of 1067 (wemake_python_styleguide/violations/consistency.py error_template = FAILED: mutmut wemake_python_styleguide/violations/consistency.py --apply --mutation " error_template = 'Found parens right after a keyword'⤑0"
902 out of 1067 (wemake_python_styleguide/violations/consistency.py error_template =
903 out of 1067 (wemake_python_styleguide/violations/consistency.py code = 313⤑0)
It shows two cases: failing and successful state.
Let's talk about both of them. I will also try to cover some general thoughts that I have.
First of all, I would like to analyze what happened. Since this process it rather long, I am not able to watch it in real time. The final output is also quite verbose, so it is hard to read it though. What do I suggest? It would be nice to have some short report that will cover:
- What mutants managed to survive: what was changed, what was the change, in what file
- What tests killed the most mutants (so these tests can be documented as important ones, etc), what tests did not caught any (so these tests might be removed in the future or refactored)
Secondly, it is possible to have better "inline experience". What do I mean by that?
- Currently when reading through the test cases I am missing some required information: what source line number was changed, to what, what mutation rule caused this
- In case that the mutant was killed I guess we can show how many tests did fail. It would be very helpful and will give you a better understanding of what is going on
I would like to say, that this tool is absolutely awesome! Thank you for building it!
Mutate copy.copy to copy.deepcopy and vice versa
`cache.py` is broken while running python2
I couldn't run tests on python2 because zip_longest does not exist needs to be replaced with izip_longest.
____________________________________________________________________________ ERROR collecting tests/test_misc.py _____________________________________________________________________________
ImportError while importing test module '/home/savo/Documents/Projects/mutmut/tests/test_misc.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.tox/py27/local/lib/python2.7/site-packages/six.py:709: in exec_
exec("""exec _code_ in _globs_, _locs_""")
tests/test_misc.py:1: in <module>
from mutmut.cache import sequence_ops
mutmut/cache.py:9: in <module>
from itertools import groupby, zip_longest
E ImportError: cannot import name zip_longest
Can't manage to pass the mutmut test
Hi, I'm trying your lib but I can't manage to pass the mutmut test, here is my code :
def addition(a, b):
return a < b
and the associated test :
def test_addition():
a = 1
b = 2
result = addition(a, b)
assert result == True
def test_addition_2():
a = 1
b = 1
result = addition(a, b)
assert result == False
and my command :
mutmut app/add.py
mutmut tests/
Is there anything wrong ?
Thanks,
Samuel
Number mutation doesn't handle decimal numbers between 0 and 1
Hi,
Attempting to mutate the following line fails with an assertion error:
self.something = 0.1
File "[...]/venv/lib/python3.5/site-packages/mutmut/init.py", line 44, in number_mutation
assert base == 10
AssertionError
This seems to be related to the following lines of mutmut/init.py (lines 37-45):
...
elif value.startswith('0') and len(value) > 1: # pragma: no cover (python 2 specific)
base = 8
value = value[1:]
else:
base = 10
if '.' in value:
assert base == 10
parsed = float(value)
...
self.something = 10.1 works
self.something = .1 works
but
self.something = 0.1 does not work
I've worked around this currently by replacing all numbers between 0 and 1 without a '0' before the decimal point, however this is not a long term solution or suitable for a large project.
I am using Version: 0.0.22
Thanks
Add --cache-only flag
Instead of having CI run the full mutation stage we could check in the entire mutmut cache dir and in CI run with a a --cache-only flag where it fails if the cache is out of date. This would make it nice to have a mutation testing badge on your project on github, but without the hassle of storing and loading the mutmut cache database or the cost of running the full mutation testing on travis on each commit.
Using click's subcommands
As discussed on #55, we should add support for click's subcommand feature: http://click.palletsprojects.com/en/7.x/commands/. This provides greater control on arguments for each individual command, and should be a backwards compatible change. This would also improve extensibility of the CLI.
Incorrectly mutates "in" keyword inside `for x in y` to invalid syntax `for x not in y`
--show-times: improve output
% mutmut --show-times --runner 'python -m pytest testing/test_collection.py' src/_pytest/nodes.py
Running tests without mutations... Done
--- starting mutation ---
Unchanged tests, using cache
1 out of 193 (src/_pytest/nodes.py SEP = \"/\"⤑0)time: 0:00:04.674225
There should be a space, or ", " / something better in front of the timing information.
mutmut command fails if no arguments are given
running the mutmut cli without arguments prints a trace back. I realize that this is not how it is supposed to be used but it is not very friendly. Perhaps the help text could be printed.
Windows 10 Enterprise
Python 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:14:34) [MSC v.1900 32 bit (Intel)] on win32
mutmut (0.0.3)
Mutate dict literal keys when they are written as dict(..)
When you have
foo = {'a': a}we mutate the key 'a', but if you write the exact same code as
foo = dict(a=a)that won't be mutated. This should be fixed, and we should also allow to specify synonyms for dict, like Struct from tri.struct. It's probably time to introduce a config file at this point! Probably just hook into setup.cfg
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.