tophat / syrupy Goto Github PK
View Code? Open in Web Editor NEW:pancakes: The sweeter pytest snapshot plugin
Home Page: https://tophat.github.io/syrupy
License: Apache License 2.0
:pancakes: The sweeter pytest snapshot plugin
Home Page: https://tophat.github.io/syrupy
License: Apache License 2.0
Is your feature request related to a problem? Please describe.
Snapshots are currently deserialized into python objects before comparison is done, this is unnecessary
Describe the solution you'd like
No deserialization is done and comparison is done on the serialized representation
Describe alternatives you've considered
N/A
Additional context
Unlocks using yaml.dumps
in place of yaml.safe_dumps
because loading yaml data is needless
Is your feature request related to a problem? Please describe.
The current migration instructions are misleading since syrupy uses a different default snapshot directory than snapshottest.
Describe the solution you'd like
Include full instructions on how to migrate from snapshottest, including a shell command to delete snapshottest snapshots.
Describe alternatives you've considered
We could add a migrate command, e.g. python -m syrupy migrate
which could look for snapshottest files and directories and delete them.
Is your feature request related to a problem? Please describe.
It'd be nice to see the impact a diff has on the plugin performance
Describe the solution you'd like
master
)Describe alternatives you've considered
N/A
Additional context
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
Diffing complex objects on assertion failure does not show only the change but the entire snapshot
Describe the solution you'd like
Should show the exact line and characters (tokens) that differ between snapshots
Describe alternatives you've considered
N/A
Additional context
N/A
Is your feature request related to a problem? Please describe.
When there are unused snapshots there should be a behaviour to fail the tests especially for running in ci.
Describe the solution you'd like
Fail when unused snapshots are detected.
Describe alternatives you've considered
Flag to fail in ci --ci
.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
It useful to know how syrupy can be used especially in contrast with other snapshot test libraries
Describe the solution you'd like
Add documentation on the various ways syrupy can be used and extended, with example test code for each.
Describe alternatives you've considered
Just documentation no examples
Additional context
A couple for starters
Describe the bug
Syrupy relies on pytest's APIs, specifically its hooks, and yet it doesn't specify what version of pytest is supported.
Expected behavior
Syrupy should list pytest in the setup.py with the maximum version range supported.
Additional context
syrupy v0.2.0
Describe the bug
Specifying test cases in a test module does not filter out unused snapshots from reporting as unused
To Reproduce
pytest --pyargs tests.test_extension_amber::TestClass
7 snapshots passed. 38 snapshots unused.
pytest tests/test_extension_amber.py::TestClass
7 snapshots passed. 38 snapshots unused.
Expected behavior
1a. 7 snapshots passed.
2a. 7 snapshots passed.
Screenshots
Desktop (please complete the following information):
Additional context
N/A
Describe the bug
The snapshot file holding the serialized assertions gets deleted when running in partial mode and update snapshots enabled
To Reproduce
Steps to reproduce the behavior:
. script/bootstrap
inv test -du -t=test_multiple_snapshots
Expected behavior
Additional context
Add any other context about the problem here.
Describe the bug
Syrupy removes other snapshot files in the same snapshot directory
To Reproduce
Steps to reproduce the behavior:
test_file1.py
, test_file2.py
) in the same directory where each one has a snapshot assertionpytest --snapshot-update
see two snapshot filespytest test_file1.py --snapshot-update
Expected behavior
Syrupy should update the file for the file under test and not remove the other snapshot file
Desktop (please complete the following information):
Describe the bug
Snapshotting a class instance which does not explicitly define a serializable repr or str, will use the default object repr which includes the memory address of the class instance. This changes every test run.
To Reproduce
Add to a test file:
class MyClass:
pass
def test_class(snapshot):
assert snapshot = MyClass()
then
inv test -d -u
Expected behavior
Snapshots should not change between test runs if the code is deterministic.
Additional context
It's possible we don't want to try support this, as it clearly implies the data type is not meant to be serialized.
Describe the bug
If you serialize an object containing a cycle, it will recurse until call stack overflows / python can no longer handle it, rather than a graceful application error.
To Reproduce
cycle = { "a": {} }
cycle["a"]["b"] = cycle
def test_cycle(snapshot):
assert snapshot == cycle
Expected behaviour
Error should be thrown if a cycle is detected.
Additional context
We already track "indent". We could add some logic to throw an error if indent exceeds some level as it's essentially equivalent to depth.
Easy way to accidentally commit credentials.
I think adding --no-emit-find-links
to pip tools compile in tasks might work.
Is your feature request related to a problem? Please describe.
No integration tests mean wide ranging can introduce bugs and regressions not caught.
Describe the solution you'd like
๐ฏ coverage!
Describe alternatives you've considered
No bragging rights for ๐ฏ coverage
Additional context
N/A
Is your feature request related to a problem? Please describe.
When asserting on snapshots support filtering out properties that should not be matched on. This would allow deterministic snapshots when matching on objects that generate realtime values in tests without needing to mock.
Describe the solution you'd like
def test_my_dict(snapshot):
my_dict = {
"match_me": True,
"do_not_match_me": time.time(),
"nested_do_not_match": {
"do_not_match_me": time.time(),
},
}
assert my_dict == snapshot(exclude=("do_not_match_me",))
# name: test_my_dict
<class 'dict'> {
'match_me': True,
'nested_do_not_match': <class 'dict'> {
},
}
---
Describe alternatives you've considered
This is already possible but is cumbersome to implement, and cannot easily be extended to other serialization types.
class OmitDataSerializer(DataSerializer):
@classmethod
def __filter_data(cls, data) -> Dict[str, Any]:
return {
key: data[key]
for key in data
if key not in ("do_not_match_me",)
}
@classmethod
def serialize_dict(cls, data, **kwargs) -> str:
return super().serialize_dict(cls.__filter_data(data), **kwargs)
class OmitDataExtension(AmberSnapshotExtension):
def serialize(self, data):
return OmitDataSerializer.serialize(data)
def test_my_dict(snapshot):
my_dict = {
"match_me": True,
"do_not_match_me": time.time(),
"nested_do_not_match": {
"do_not_match_me": time.time(),
},
}
assert my_dict == snapshot(extension=OmitDataExtension)
Additional context
Describe the bug
Notice how in the screenshot that 5/6 tests are deselected, but the test run reports 2 unused snapshots.
To Reproduce
Steps to reproduce the behavior:
pip install pytest pytest-watch syrupy
import pytest
def test_thing_1(snapshot):
assert snapshot == "123"
def test_thing_2(snapshot):
assert snapshot == "456"
def test_thing_11(snapshot):
assert snapshot == "123"
def test_thing_22(snapshot):
assert snapshot == "456"
@pytest.mark.wip
def test_thing_111(snapshot):
assert snapshot == "123"
def test_thing_222(snapshot):
assert snapshot == "456"
pytest --snapshot-update
ptw -- -m wip
to watch test files and only run tests with the wip
markpytest -m wip
reports two unused snapshots.Expected behavior
Here I'd expect no unused snapshots to be reported -- seems like deselected tests are sometimes being considered not having used their snapshot, which is technically true, but given that the test didn't run I'd say there's no way to assess whether or not the snapshot is out of date.
Screenshots
Included above
Desktop (please complete the following information):
Additional context
The behaviour doesn't occur with two tests, i.e. the following test file does report an unused snapshot when run with pytest -m wip
:
import pytest
def test_thing_1(snapshot):
assert snapshot == "123"
@pytest.mark.wip
def test_thing_2(snapshot):
assert snapshot == "456"
Is your feature request related to a problem? Please describe.
Black is just a formatter and does not catch things like unused imports.
Describe the solution you'd like
pylint/flake8 to precommit hook and CI.
Describe the bug
When running a subset of all tests with the pytest keyword flag -k
, snapshots in tests matching the keyword should be updated and deleted appropriately. This does not work for pytest.mark.parametrize
tests
To Reproduce
Steps to reproduce the behavior:
syrupy/tests/test_snapshots.py
Line 37 in 7cde8a7
inv test -d -u -t=<test_name>
Expected behavior
Test case removed from the parametrized definition gets deleted
Screenshots
N/A
Desktop (please complete the following information):
Additional context
N/A
Describe the bug
Running syrupy with pytest 5.0.1 produces:
AttributeError: 'Config' object has no attribute 'invocation_params'
To Reproduce
pip install pytest==5.0.1 syrupy==0.3.5
echo "def test_example(snapshot):\nassert snapshot == 'hi'" > test_file.py
pytest
If we don't support 5.0.1, we should be explicit about min versions.
Would it be more idiomatic to have the current SerializerClass
be replaced in name by IOClass
but retain the same functionality.
For replacing with AbstractSnapshotIO |
For keeping Serializer nomenclature |
---|---|
IO handles the file system, the serialization is just a step in the process | Serializer handles writing and reading from file system |
IO (input, output) is more encompassing allowing for extensions to functionality like preparing assertion diffs | Serializer is a more familiar concept to most people making the naming appropriate for its function |
Line 71 in 44b0f6c
Lines 66 to 71 in 44b0f6c
syrupy/src/syrupy/assertion.py
Lines 71 to 75 in 44b0f6c
syrupy/src/syrupy/assertion.py
Lines 135 to 138 in 44b0f6c
See README. The license badge says "missing" even though the license is correctly detected in pypi and the LICENSE file is included in the package manifest.
Describe the bug
Syrupy assertion diff does not show missing carriage return.
To Reproduce
Add this test.
def test_example(snapshot):
assert snapshot == "line 1\r\nline 2"
Run pytest --snapshot-update
.
Remove the \r
from the string in the test case so you get:
def test_example(snapshot):
assert snapshot == "line 1\nline 2"
Run pytest
. The test will fail with a useless diff of "...".
Expected behavior
Syrupy should output each line with the missing carriage return, with some indication of what's missing.
Additional context
Syrupy v0.3.1
#113 fixed the bug where carriage returns were not being serialized. This issue addresses the missing functionality in the snapshot assertion diff reporter.
Is your feature request related to a problem? Please describe.
Syrupy supports multiple serializes including custom serializes. If a codebase using syrupy selects a non-yaml serializes, there is no reason to install the unused dependency.
Describe the solution you'd like
List pyyaml as an optional dependency. Throw error if the seralizer is used if pyyaml is not present in system.
Describe alternatives you've considered
Additional context
Python dependency management is awful ๐
Describe the bug
Snapshots are being deleted immediately after being written when syrupy is run with --snapshot-update
.
To Reproduce
class TestMyClass:
@pytest.mark.parametrize(
"some_arg",
[
"abc",
"def",
],
)
def test_param(self, some_arg, snapshot):
assert some_arg == snapshot
Expected behavior
Snapshots should not be deleted if they're being used.
Additional Context
Syrupy v0.0.14 using the Image serializer.
Describe the bug
Frozen sets are the only custom types allowed in the test, others fail with the error message
TypeError: '<' not supported between instances of 'str' and '...'
To Reproduce
Steps to reproduce the behavior:
git checkout unsupported_sort
inv test -d
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
syrupy/tests/test_extension_amber.py
Lines 76 to 77 in 9e8dea8
Is your feature request related to a problem? Please describe.
To help build a positive community around Python snapshot testing we should agree on an official code of conduct and hold ourselves accountable.
This is part of GitHub's community checklist.
Is your feature request related to a problem? Please describe.
Currently you have to override the snapshot fixture in your project's root conftest.py in order to specify the default serializer class. It's potentially not clear what the final serializer is (order of shadowing).
Describe the solution you'd like
A configuration option to specify the python module path to load for the default serializer.
--snapshot-default-plugin syrupy.extensions.image.SVGImageExtension
Describe alternatives you've considered
Adding documentation for overriding the default fixture.
Describe the bug
If a carriage return is present in a string, Syrupy will strip out the carriage return when serializing the data thus making the assertion persistently fail even after a snapshot-update.
To Reproduce
def test_carriage(snapshot):
assert snapshot == "line 1\r\nline 2"
pytest --snapshot-update; pytest
Expected behavior
Syrupy should be able to serialize any control character.
Additional context
Syrupy v0.3.0
Describe the bug
Snapshot report should be shown after test failures are reported not before
To Reproduce
Steps to reproduce the behavior:
echo 'def test_example(snapshot): assert "abc" == snapshot' > tests /test_example.py
pytest tests/test_example.py
(syrupy.venv) (iamogbz-patch-1) ฮป pytest tests/test_example.py
========================= test session starts =========================
platform darwin -- Python 3.8.1, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /Users/Emmanuel/Sources/syrupy, inifile: pytest.ini
plugins: syrupy-2020.3.1.111512023105
collected 1 item
tests/test_example.py F [100%]
1 snapshot failed.
============================== FAILURES ===============================
____________________________ test_example _____________________________
snapshot = SnapshotAssertion(name='snapshot', num_executions=1)
def test_example(snapshot):
> assert 'abc' == snapshot
E assert received == snapshot
E Snapshot does not exist!
tests/test_example.py:2: AssertionError
========================== 1 failed in 0.05s =========================
Expected behavior
Snapshot report should show after the failing tests
Screenshots
N/A
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Is your feature request related to a problem? Please describe.
Snapshots should be helpful in indicating at a glance what has changed.
Describe the solution you'd like
def test_my_dict(snapshot):
my_dict = {
"field_0": True,
"field_1": "no_value",
"nested": {
"field_0": 1,
},
}
assert my_dict == snapshot(id="case1")
my_dict["field_1"] = "yes_value"
assert my_dict == snapshot(diff="case1")
my_dict["nested"]["field_0"] = 2
assert my_dict == snapshot(id="case3", diff=1)
# name: test_my_dict.case_1
<class 'dict'> {
'field_0': True,
'field_1': 'no_value',
'nested': <class 'dict'> {
'field_0': 1
},
}
---
# name: test_my_dict.1
'field_1': 'yes_value',
---
# name: test_my_dict.case_3
'field_0': 2
---
Describe alternatives you've considered
N/A
Additional context
Is your feature request related to a problem? Please describe.
The README contains an outdated usage gif which is misleading.
Describe the solution you'd like
Record a new version of it.
Describe the bug
Pytest raises an error from argparse because of a conflicting arguments: both snapshottest and syrupy declare a --snapshot-update argument, so you have to trash your virtualenv or explicitly uninstall snapshottest to get syrupy working.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
It would be nice if you didn't get this confusing error when migrating, it would make syrupy an easier drop-in replacement for snapshottest, but it seems like this may be out of control of plugins. Could be worth calling it out explicitly in the docs for other people who encounter it, if a programmatic solution is not available.
Desktop (please complete the following information):
Is your feature request related to a problem? Please describe.
Show how many snapshots were used in the test session
Describe the solution you'd like
Showing the number summary e.g.
1 snapshot failed | 5 snapshots passed | 1 snapshot updated | 3 snapshots generated | 2 snapshots unused
Describe alternatives you've considered
N/A
Additional context
This requires an overhaul of how snapshot assertions are tracked
pyyaml, for example, should not be pinned in requirements.txt
Describe the bug
Named tuples when serialised loose argument information
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Parameter names should be sorted and preserved like a dictionary or class
Screenshots
N/A
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
Opening an issue for discussion purposes, no defined requirements or suggestions for how (or whether) this should be implemented):
See https://jestjs.io/docs/en/snapshot-testing#inline-snapshots for how jest handles inline snapshots.
It would be cool if syrupy could support something this since it's really nice for certain cases.
This is a general issue to track notes about readability and accessibility in diffs. See this jest blog post for more information: https://jestjs.io/blog/2020/01/21/jest-25, and this jest PR jestjs/jest#9132
They seem to have put a lot of effort into improving diff outputs recently so it's definitely something to consider and take inspiration from.
Sorry for the lack of specificity on this issue, mostly created it for tracking purposes with the idea of triggering an investigation.
Investigate and add security vulnerability code analysis tools. Not just for dependencies but for patterns as well (e.g. use of exec).
Describe the bug
snapshot.assert_match(ANY)
always passed
To Reproduce
Steps to reproduce the behavior:
snapshot.assert_match(None)
syrupy/tests/test_snapshots.py
Line 14 in a5c46b1
inv test
Expected behavior
Should fail with the same error as assert None == snapshot
Current tests are starting to be hard to reason about. Some clarification about the behaviour each test is attempting to verify is getting more essential.
Describe the solution you'd like
Make each test case group a single file testing all the behaviours with module and test doc string explaining the purpose
Describe alternatives you've considered
N/A
Additional context
The tests/test_integration_default.py
file is an example of this complexity.
Is your feature request related to a problem? Please describe.
One of the project's motivations is to provide a uniform, idiomatic way to assert snapshots. We're keeping snapshot.assert_match
for backwards compatibility with snapshottest for the moment, however no one uses the snapshot(actual)
syntax. It's a remnant of testing out different approaches.
Describe the solution you'd like
Remove the snapshot call syntax:
syrupy/src/syrupy/assertion.py
Line 123 in 7cc4047
Is your feature request related to a problem? Please describe.
An empty list takes up 3 lines in an ambr file.
<class 'list'> [
]
Describe the solution you'd like
Represent an empty iterable in a more concise format:
<class 'list'> []
or
<class 'list'> [ ]
Describe alternatives you've considered
There's an argument that the more verbose format produces smaller git diffs if adding an element, since we don't change the line indicating a list is being used.
Describe the bug
Syrupy does not support nested test blocks
To Reproduce
Steps to reproduce the behavior:
Given
# test_str.py
class TestClass:
class TestSubClass:
def test_sub_class_has_parent(self, snapshot):
assert "this should be in a nested class" == snapshot
Result
# name: TestSubClass.test_sub_class_has_parent
'this should be in a nested class'
---
Expected behavior
# name: TestClass.TestSubClass.test_sub_class_has_parent
'this should be in a nested class'
---
Screenshots
N/A
Desktop (please complete the following information):
Additional context
N/A
Is your feature request related to a problem? Please describe.
This is mainly a refactor concern: in Python 3.4+, pathlib
can serve as a more powerful (and built-in) replacement for os.path
, dealing with flexible pathlike objects compatible anywhere where a string path may be used. It offers extra features over just dealing with paths, such as navigating filetrees with ease using object methods. Succinctly put, it provides a cleaner experience when dealing with file structures.
Describe the solution you'd like
Refactor syrupy
so that it uses pathlib
wherever it deals with paths instead of os.path
.
Describe alternatives you've considered
Continuing with os.path
and accepting increased complexity.
Additional context
Since Syrupy already requires python >= 3.6
, the version requirement for pathlib
isn't a concern.
Is your feature request related to a problem? Please describe.
Package releases are currently manual. We should automate these releases via semantic commits.
This means we need to enforce semantic commits as part of precommit hook and CI, and parse the commits to determine version bump.
Alternatives
Continuing manual releases w/ thoroughly documented process.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.