GithubHelp home page GithubHelp logo

kiwicom / pytest-recording Goto Github PK

View Code? Open in Web Editor NEW
382.0 4.0 30.0 209 KB

A pytest plugin that allows recording network interactions via VCR.py

License: MIT License

Python 100.00%
pytest vcr testing python cassettes hacktoberfest

pytest-recording's Introduction

pytest-recording

codecov Build Version Python versions License

A pytest plugin that records network interactions in your tests via VCR.py.

Features

  • Straightforward pytest.mark.vcr, that reflects VCR.use_cassettes API;
  • Combining multiple VCR cassettes;
  • Network access blocking;
  • The rewrite recording mode that rewrites cassettes from scratch.

Usage

import pytest
import requests

# cassettes/{module_name}/test_single.yaml will be used
@pytest.mark.vcr
def test_single():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'

# cassettes/{module_name}/example.yaml will be used
@pytest.mark.default_cassette("example.yaml")
@pytest.mark.vcr
def test_default():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'

# these cassettes will be used in addition to the default one
@pytest.mark.vcr("/path/to/ip.yaml", "/path/to/get.yaml")
def test_multiple():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'
    assert requests.get("http://httpbin.org/ip").text == '{"ip": true}'

# Make assertions based on the cassette calls/responses:
@pytest.mark.vcr
def test_call_count(vcr):
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'
    assert requests.get("http://httpbin.org/ip").text == '{"ip": true}'
    # See https://vcrpy.readthedocs.io/en/latest/advanced.html for more info
    # about the Cassette object:
    assert vcr.play_count == 2

Run your tests:

pytest --record-mode=once test_network.py

Default recording mode

pytest-recording uses the none VCR recording mode by default to prevent unintentional network requests. To allow them you need to pass a different recording mode (e.g. once) via the --record-mode CLI option to your test command. See more information about available recording modes in the official VCR documentation

Configuration

You can provide the recording configuration with the vcr_config fixture, which could be any scope - session, package, module, or function. It should return a dictionary that will be passed directly to VCR.use_cassettes under the hood.

import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {"filter_headers": ["authorization"]}

For more granular control you need to pass these keyword arguments to individual pytest.mark.vcr marks, and in this case all arguments will be merged into a single dictionary with the following priority (low -> high):

  • vcr_config fixture
  • all marks from the most broad scope ("session") to the most narrow one ("function")

Example:

import pytest

pytestmark = [pytest.mark.vcr(ignore_localhost=True)]

@pytest.fixture(scope="module")
def vcr_config():
    return {"filter_headers": ["authorization"]}

@pytest.mark.vcr(filter_headers=[])
def test_one():
    ...

@pytest.mark.vcr(filter_query_parameters=["api_key"])
def test_two():
    ...

Resulting VCR configs for each test:

  • test_one - {"ignore_localhost": True, "filter_headers": []}
  • test_two - {"ignore_localhost": True, "filter_headers": ["authorization"], "filter_query_parameters": ["api_key"]}

You can get access to the used VCR instance via pytest_recording_configure hook. It might be useful for registering custom matchers, persisters, etc.:

# conftest.py

def jurassic_matcher(r1, r2):
    assert r1.uri == r2.uri and "JURASSIC PARK" in r1.body, \
        "required string (JURASSIC PARK) not found in request body"

def pytest_recording_configure(config, vcr):
    vcr.register_matcher("jurassic", jurassic_matcher)

You can disable the VCR.py integration entirely by passing the --disable-recording CLI option.

Rewrite record mode

It is possible to rewrite a cassette from scratch and not extend it with new entries as it works now with the all record mode from VCR.py.

However, it will rewrite only the default cassette and won't touch extra cassettes.

import pytest

@pytest.fixture(scope="module")
def vcr_config():
    return {"record_mode": "rewrite"}

Or via command-line option:

$ pytest --record-mode=rewrite tests/

Blocking network access

To have more confidence that your tests will not go over the wire, you can block it with pytest.mark.block_network mark:

import pytest
import requests

@pytest.mark.block_network
def test_multiple():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'

...
# in case of access
RuntimeError: Network is disabled

Besides marks, the network access could be blocked globally with --block-network command-line option.

However, if VCR.py recording is enabled, the network is not blocked for tests with pytest.mark.vcr.

Example:

import pytest
import requests

@pytest.mark.vcr
def test_multiple():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'

Run pytest:

$ pytest --record-mode=once --block-network tests/

The network blocking feature supports socket-based transports and pycurl.

It is possible to allow access to specified hosts during network blocking:

import pytest
import requests

@pytest.mark.block_network(allowed_hosts=["httpbin.*"])
def test_access():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'
    with pytest.raises(RuntimeError, match=r"^Network is disabled$"):
        requests.get("http://example.com")

Or via command-line option:

$ pytest --record-mode=once --block-network --allowed-hosts=httpbin.*,localhost tests/

Or via vcr_config fixture:

import pytest

@pytest.fixture(autouse=True)
def vcr_config():
    return {"allowed_hosts": ["httpbin.*"]}

Additional resources

Looking for more examples? Check out this article about pytest-recording.

Contributing

To run the tests:

$ tox -p all

For more information, take a look at our contributing guide

Python support

Pytest-recording supports:

  • CPython 3.7, 3.8, 3.9, 3.10, 3.11, and 3.12
  • PyPy 7 (3.6)

License

The code in this project is licensed under MIT license. By contributing to pytest-recording, you agree that your contributions will be licensed under its MIT license.

pytest-recording's People

Contributors

andrecimander avatar dcermak avatar dsummersl avatar hartwork avatar jakkdl avatar kianmeng avatar martina-oefelein avatar penguincabinet avatar renovate[bot] avatar s-t-e-v-e-n-k avatar stranger6667 avatar svtkachenko avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

pytest-recording's Issues

Fuzzy cassettes

For unhappy path testing, there could be an extra parameter that will mutate recorded cassettes.
Use cases:

  • Check how the app will work if the format of responses will change
  • Validate error handling

Possible fuzzes:

  • Mutate response body. JSON - add/remove fields in objects / lists. Change values. Make JSON invalid.
  • Completely change response content type
  • Change response status code
  • Mutate requests during recording??
  • Raise exceptions instead of real responses
  • Add delays to responses

All these could be combined and generate test cases

[FEATURE] Record pytest fixtures

Is your feature request related to a problem? Please describe.
In my project i am using fixtures with scope="session" and the requests that are made in the fixture setup are not being recorded, neither in the test casettes, nor in a separate casette, when i mark the fixture with @pytest.mark.vcr

Describe the solution you'd like
I would like it to be possible to decorate fixtures with @pytest.mark.vcr() with the result of a casette being recorded for the individual fixtures, such taht i can use pytest-recording to record fixtures with scope="session"

Describe alternatives you've considered
if i use scope="function" the fixture setup requests are recorded in each test casette and the offline testing works.

[FEATURE] Run Python 3.9 builds

  • Add 39 to envlist in tox.ini
  • Add 39 to depends of the coverage-report job in tox.ini
  • Add Programming Language :: Python :: 3.9 to the classifiers list in setup.py
  • Add Python 3.9 job in .github/workflows/build.yml similarly to other builds

Restore handling relative paths in `pytest.mark.vcr`

It was possible in 0.4.0, but now it is not. However, before it was applied only to the first argument in pytest.mark.vcr, other arguments were treated like absolute. It will be nice to support relative paths for all arguments to the vcr mark

Cassettes ordering can be changed in the multi-cassette mode

You use startmap to make the cassette list unique, but it leads to the situation when ordering of the cassettes can change, because map keys isn't ordered. I faced with this problem, when my tests failed in CI, but worked locally because of this.

requests, responses = starmap(unpack, zip(*all_content))

BTW, what for do you make the cassette list unique? I would like to put one cassette multiple times in one testcase, so ensure that the same request is performed many times. Maybe just get rid of this?

P.S. Thanks a lot for the great lib, I really like it!

Use default mode `once` in README

I'm new to VCR.py and pytest-recording. Just setup for my scraper project today.

Followed by the README doc, I used --record-mode=all to run pytest. However, the tests will failed in CI (I used CircleCI) and the YAMLs under cassettes will be changed.

After read VCR.py doc, I change to --record-mode=once. Now all the cassettes won't be updated and all tests passed in CI.

IMO, I suggest to use --record-mode=once in README, which may be easier for beginner.

[BUG] record_mode set via vcr_config fixture and marker not honored by block_network

Describe the bug
Setting the record_mode via config fixture nor marker is not honored by block_network. The record_mode variable in block_network is still in default state "none".

To Reproduce

import pytest
import requests


@pytest.fixture(autouse=True)
def vcr_config():
    return {
        "filter_headers": ["authorization"],
        "ignore_localhost": True,
        "record_mode": "once",
    }


@pytest.mark.vcr()
@pytest.mark.block_network(record_mode="once")
def test_request():
    requests.get("https://google.com")

Expected behavior
Shouldn't raise a RuntimeError since we are supposed to be recording.

Environment (please complete the following information):

  • OS: Linux(Manjaro)
  • Python version: 3.8.7
  • pytest-recording version: 0.11.0
  • pytest version: 6.2.4

Simplify working with network blocker

Probably two functions will be helpful - enable / disable, in some cases context manager is not needed and calling __enter__ / __exit__ manually is not convenient

More robust message for network access blocking

Hi!
It would be great to see a more robust error message than "Network is disabled".
This would be especially useful for newcomers or people who aren't aware that this package is installed on the project they are working on.

Some ideas for what it could contain:

  1. A title saying that the error is coming from this package
  2. A short summary of what happened. Why the error was raised.
  3. The http request that was made
  4. The list of configured allowed hosts (for quick comparison with the actual request made)
  5. Possible ways to solve the issue (change the request, record a cassette, add allowed hosts...)
  6. A link to the relevant section in the docs.

A great example of a similar error message is the message in Ruby's VCR implementation which you can see here: https://stackoverflow.com/questions/25799165/vcr-throwing-cant-handle-request-on-the-second-attempt-to-run-testfirst-attempt

Thanks!

[Bug] Using `once` record_mode gives error

Problem

Using once gives an error:

vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ([PATH_OF_CASETTE_FILE]) in your current record mode ('once').
E               No match for the request (<Request (GET) https://postman-echo.com/get?foo1=bar1&foo2=bar2>) was found.
E               No similar requests, that have not been played, found.

I ran this the first time so PATH_OF_CASETTE_FILE shouldn't exist although the error seems to say it exists. I checked the PATH_OF_CASETTE_FILE that the error specifies before I run my test the 2nd time and indeed no such file exists.

To Reproduce

import pytest
import requests

@pytest.mark.vcr(record_mode="once")
def test_vcr():
    response = requests.get("https://postman-echo.com/get?foo1=bar1&foo2=bar2")
    assert response.status_code == 200

Stacktrace

Click to expand!
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/requests/api.py:76: in get
    return request('get', url, params=params, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/requests/api.py:61: in request
    return session.request(method=method, url=url, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/requests/sessions.py:530: in request
    resp = self.send(prep, **send_kwargs)
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/requests/sessions.py:643: in send
    r = adapter.send(request, **kwargs)
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/requests/adapters.py:439: in send
    resp = conn.urlopen(
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/urllib3/connectionpool.py:670: in urlopen
    httplib_response = self._make_request(
../../Library/Caches/pypoetry/virtualenvs/featuredproductsswitch-q2O7GQgj-py3.8/lib/python3.8/site-packages/urllib3/connectionpool.py:417: in _make_request
    httplib_response = conn.getresponse(buffering=True)

[BUG] having this plugin enabled breaks Intellij Idea failed test reports

Ok, so this was a weird one to debug...

Simply having the pytest-recording plugin enabled breaks Intellij Idea pytest failed test reports in some specific test cases (here is an example):

  • a test cassette is being recorded via plain vcrpy syntax or via pytest-recording decorator;
  • two (or more) network calls are being executed and recorded: the first one succeeds, the second fails and then an
    error is raised by requests' raise_for_status() method.

Instead of reporting the correct stack trace and main error, Idea reports there has been a failed string comparison
involving url paths.

My guess is that pytest-recording breaks something Idea's test runner relies on to generate errors messages, because:

  • pytest output in the terminal is consistent and correct with or without the plugin installed;
  • disabling the pytest-recording plugin in Idea ui by adding -p no:recording as additional argument restore the correct
    error message;
  • removing the plugin also restore the correct error message.

How to reproduce the issue

Checkout the minimal test repo with git clone https://github.com/CarloDePieri/pytest-recording-idea-issue.

Create a virtualenv and install all dependencies there:

cd pytest-recording-idea-issue
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

IMPORTANT:

  • this DOES NOT install pytest-recording;
  • the test we are going to launch DOES NOT use this plugin decorator, but plain vcrpy.

Then:

  • import the folder into Idea as a new project;
  • add the created virtualenv as a python sdk for the project;
  • run the test from the Idea ui: observe that the test fails with the correct error message reporting a 404;
  • manually install pytest-recording with pip install pytest-recording in the venv;
  • relaunch the test from the Idea ui: the error message is now completely off track: it reports a difference between
    expected and actual values '/api/users/23' != '/api/users/2'.

Under the hood

Idea uses this test runner
to launch pytest tests and generate the report message. Launching the script directly in the terminal shows indeed the
wrong error message when pytest-recording is installed.

Installed software versions

python: 3.10.4
pytest: 7.1.2
pytest-recording: 0.12.0
vcrpy: 4.1.1
requests: 2.27.1
Idea Ultimate: Build #IU-221.5591.52, built on May 10, 2022
os: arch linux

[BUG]

Describe the bug
Can't write cassette for file uploading. Is it possible to achieve that?

To Reproduce
Steps to reproduce the behavior:

  1. Sample piece of code, which I want to cover with integration test
async def upload_file(filename):
    # some code omitted for simplicity
    data = {'file': open(filename, 'rb')}
    async with aiohttp.ClientSession() as session:
        async with session.post(url, headers=headers, data=data) as response:
            result = await response.read()
            return response.status, result
  1. Test:
@pytest.mark.vcr()
async def test_upload_file():
        img = open(f'{test_data_path}/logo.png', 'rb')
        status, result = await client.upload_file(img)
  1. While test is passed, vcr failed to write cassette:
../venv/lib/python3.8/site-packages/vcr/cassette.py:91: in __exit__
    next(self.__finish, None)
../venv/lib/python3.8/site-packages/vcr/cassette.py:69: in _patch_generator
    cassette._save()
../venv/lib/python3.8/site-packages/vcr/cassette.py:331: in _save
    self._persister.save_cassette(self._path, self._as_dict(), serializer=self._serializer)
../venv/lib/python3.8/site-packages/vcr/persisters/filesystem.py:20: in save_cassette
    data = serialize(cassette_dict, serializer)
../venv/lib/python3.8/site-packages/vcr/serialize.py:58: in serialize
    return serializer.serialize(data)
../venv/lib/python3.8/site-packages/vcr/serializers/yamlserializer.py:15: in serialize
    return yaml.dump(cassette_dict, Dumper=Dumper)
../venv/lib/python3.8/site-packages/yaml/__init__.py:290: in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
../venv/lib/python3.8/site-packages/yaml/__init__.py:278: in dump_all
    dumper.represent(data)
../venv/lib/python3.8/site-packages/yaml/representer.py:27: in represent
    node = self.represent_data(data)
../venv/lib/python3.8/site-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
../venv/lib/python3.8/site-packages/yaml/representer.py:207: in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
../venv/lib/python3.8/site-packages/yaml/representer.py:118: in represent_mapping
    node_value = self.represent_data(item_value)
../venv/lib/python3.8/site-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
../venv/lib/python3.8/site-packages/yaml/representer.py:199: in represent_list
    return self.represent_sequence('tag:yaml.org,2002:seq', data)
../venv/lib/python3.8/site-packages/yaml/representer.py:92: in represent_sequence
    node_item = self.represent_data(item)
../venv/lib/python3.8/site-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
../venv/lib/python3.8/site-packages/yaml/representer.py:207: in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
../venv/lib/python3.8/site-packages/yaml/representer.py:118: in represent_mapping
    node_value = self.represent_data(item_value)
../venv/lib/python3.8/site-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
../venv/lib/python3.8/site-packages/yaml/representer.py:207: in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
../venv/lib/python3.8/site-packages/yaml/representer.py:118: in represent_mapping
    node_value = self.represent_data(item_value)
../venv/lib/python3.8/site-packages/yaml/representer.py:48: in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
../venv/lib/python3.8/site-packages/yaml/representer.py:207: in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
../venv/lib/python3.8/site-packages/yaml/representer.py:118: in represent_mapping
    node_value = self.represent_data(item_value)
../venv/lib/python3.8/site-packages/yaml/representer.py:52: in represent_data
    node = self.yaml_multi_representers[data_type](self, data)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <yaml.dumper.Dumper object at 0x10cea75b0>, data = <_io.BufferedReader name='tests/integration/data/logo.png'>

    def represent_object(self, data):
        # We use __reduce__ API to save the data. data.__reduce__ returns
        # a tuple of length 2-5:
        #   (function, args, state, listitems, dictitems)
    
        # For reconstructing, we calls function(*args), then set its state,
        # listitems, and dictitems if they are not None.
    
        # A special case is when function.__name__ == '__newobj__'. In this
        # case we create the object with args[0].__new__(*args).
    
        # Another special case is when __reduce__ returns a string - we don't
        # support it.
    
        # We produce a !!python/object, !!python/object/new or
        # !!python/object/apply node.
    
        cls = type(data)
        if cls in copyreg.dispatch_table:
            reduce = copyreg.dispatch_table[cls](data)
        elif hasattr(data, '__reduce_ex__'):
>           reduce = data.__reduce_ex__(2)
E           TypeError: cannot pickle '_io.BufferedReader' object

../venv/lib/python3.8/site-packages/yaml/representer.py:317: TypeError

Expected behavior
pytest-recording should not fail and provide some mock for IO objects or smth else

Environment (please complete the following information):

  • OS: MacOS
  • Python version: 3.8.2
  • pytest-recording version: 0.10.0
  • pytest version: 6.1.2

Additional context
Add any other context about the problem here.

Disallow multiple `network.block()` calls

If the --block-network CMD option is passed, and if network.block is called in code, then one will overwrite another, and it depends on the execution order, which may lead to hard-to-track bugs in tests (it happened in one of our projects)

Implementation notes:

  • In the network.block call check if socket.socket.connect is already replaced
  • If so, raise a RuntimeError with some explanation

Don't add YAML extension to JSON cassettes

Thank you for the plugin. Unfortunately, it is really hard to find it as pytest-vcr always comes first in the search.

I have the following issue:
If I use serializer="json" and I use something like @pytest.mark.vcr('list_rooms.json'), then the plugin adds yaml extension to the cassette (list_rooms.json.yaml), though it is saved as json correctly.

Decode compressed response

I have the following in my conftest.py:

@pytest.fixture(scope="module")
def vcr_config():
    return {
        "cassette_library_dir": f"{TESTS_DIR}/fixtures/cassettes",
        "decode_compressed_response": True,
        "filter_headers": ["authorization"]
    }

But decoding is happening intermittently.

[FEATURE] A way to allow all local IPs in network blocking context

Is your feature request related to a problem? Please describe.
When running in docker-compose you need to resolve local services first (e.g. DB) and then pass them to --allowed-hosts.

Describe the solution you'd like
Add --allow-local-ips flag to CLI + related arg to the block mark to allow requests to local IPs.
Use something like ipaddress.is_local to check all IPs in such a case.

Additional context
The feature & approach was proposed by @selevit

How to register matchers?

I would like to register my own matcher.

I looked at pytest-recording's source and noticed vcr is in _vcr.py ; Since it starts with an underscore, I'm guessing that it's not meant to be imported outside of the package?

How do I register my own matcher with pytest-recording?

TypeError when using network blocking with address as bytes

pattern = '(127.0.0.2)', string = b'127.0.0.1', flags = 0

    def match(pattern, string, flags=0):
        """Try to apply the pattern at the start of the string, returning
        a Match object, or None if no match was found."""
>       return _compile(pattern, flags).match(string)
E       TypeError: cannot use a string pattern on a bytes-like object

bytearray is also a possible type

And the code to trigger:

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect((b"127.0.0.1", 80))

Custom VCR config via separate fixture

Currently, there is no way to specify a module/session-level config without wrapping everything with pytest.mark.vcr.
E.g. test module has 10 tests, 2 of them use pytest.mark.vcr - if they share the same config it will require manual explicit specification on each mark, which means some duplication. An alternative solution is to have a separate fixture of any scope - vcr_config that will be added to the VCR config implicitly (like it is in pytest-vcr).


@pytest.fixture(scope="module")
def vcr_config():
    return {}


@pytest.mark.vcr
def test_a():
    pass


@pytest.mark.vcr
def test_b():
    pass


def test_c():
    pass

In this case, we can avoid VCR setup where it is not needed and the vcr_config fixture could be the first in the merging process - all marks will override colliding keys

Control what components to patch in `block_network`

Sometimes it is needed to use only one or another library to access the network. E.g., sometimes socket is ok, but pycurl is not, or in the case if there will be more libraries to patch (maybe psycopg2 or some others), then it would be nice to have some granular control over what to disable and what is not.

Implementation notes:

  • Add a global container of available transports, e.g. TRANSPORTS = ("socket", "pycurl")
  • Then add a new transports argument to network.block with the default value of TRANSPORTS
  • Add conditions inside network.block to patch only transports defined in the new arguments

Provide extra config option for shared cassettes dir(s)

It could be a separate fixture that will return a list of paths where to look for cassettes. Currently, we need to use the full path to the cassette in pytest.mark.vcr which is tedious. However, shared cassettes could be used via a separate mark to avoid breaking the existing interface - pytest.mark.vcr_shared("first.yaml", "second.yaml")

[BUG] Default record_mode is 'none' instead of 'once'

Describe the bug
When I use the decorator without passing any record_mode (@pytest.mark.vcr), I get an error saying,

vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/path/test_single.yaml') in your current record mode ('none').

The VCR docs says the default mode is "once" and pytest-recording doesn't mention the default mode, implying it follows VCR's.

To Reproduce
Use the sample test in the docs,

@pytest.mark.vcr
def test_single():
    actualResponse = requests.get("https://httpbin.org/get")
    assert actualResponse.status_code == 200

Expected behaviour
No errors.

Environment (please complete the following information):

  • OS: MacOS
  • Python version: 3.8.7
  • pytest-recording version: 0.11.0
  • pytest version: 6.2.2

Improve Readme

Readme doesn't document requirements, setup process, etc.

I've installed pytest-recording and hacking around now to get it working but would be great if this was all part of the Readme.

[FEATURE] Delete casette if test fails when running with --record-mode=once

Is your feature request related to a problem? Please describe.
When I'm building a new test, I'll run the test with --record-mode=once, but the test might fail (since it's new) and I then have to go delete the file so it'll re-record. It's kinda an annoyance to go find and make sure I delete the right file.

Describe the solution you'd like
I'd love it if there was some logic where when running with --record-mode=once it only saved the files from the tests that pass, then I could write a bunch of new tests, run them all, the good ones would get saved, and then I'd go fix the ones that didn't pass.

Just kinda a nice to have I'd think. Definitely not a big issue. Though it seems odd to save a cassette for a failed test in general.

Describe alternatives you've considered
Considered --record-mode=any, but that would re-run unrelated tests that I'm not working on.

Host names are not working in `allowed_hosts`

Example from the README file:

import pytest
import requests

@pytest.mark.block_network(allowed_hosts=["httpbin.*"])
def test_access():
    assert requests.get("http://httpbin.org/get").text == '{"get": true}'
    with pytest.raises(RuntimeError, match=r"^Network is disabled$"):
        requests.get("http://example.com")

It will fail on the first line:

self = <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 0)>, address = ('100.25.11.135', 80), args = (), kwargs = {}, host = '100.25.11.135'

    def network_guard(self, address, *args, **kwargs):
        host = ""
        if self.family in (socket.AF_INET, socket.AF_INET6):
            host = address[0]
        elif self.family == socket.AF_UNIX:
            host = address
        if is_host_in_allowed_hosts(host, allowed_hosts):
            return original_func(self, address, *args, **kwargs)
>       raise RuntimeError("Network is disabled")
E       RuntimeError: Network is disabled

Write multiple tests to the same cassette

Is it possible to write multiple tests to the same cassette? If multiple tests are making the same API calls, it's inefficient to record them again and again, better to use a shared cassette.

This is possible with vcr.use_cassette, but doesn't seem to be possible with pytest.mark.vcr, since the default cassette name is always the test name.

For example:

@pytest.mark.vcr("tests/cassettes/shared_cassette.yaml")
def test_a(): ...

@pytest.mark.vcr("tests/cassettes/shared_cassette.yaml")
def test_b(): ...

In this example, you will still write new cassettes to tests/cassettes/test_a.yaml and tests/cassettes/test_b.yaml, rather than using the shared one.

[FEATURE] Provide a way to use pytest_recording without `pytest.mark.vcr` decorator

Is your feature request related to a problem? Please describe.
It might be convenient to use _vcr.use_cassette directly for granular control of VCR setup & teardown mechanisms.

Describe the solution you'd like
Create a separate function that will not depend on pytest-related types & expose it as a part of the public API. It might be used by _vcr.use_cassette under the hood (or, likely, _vcr.use_cassette will be renamed to something else, and the new function will take this name).

Additional context
Requested by @selevit

What is the relationship to pytest-vcr?

I have seen pytest-recording (on pypi) which looks pretty similar:

  • Uses @pytest.mark.vcr
  • Both seem to be wrappers around vcr.py
  • 49 stars, 3 forks, 2 contributors vs 67 stars, 17 forks, 7 contributors
  • First tag in 2019 vs first tag in 2017

My guess is that pytest-recording is a fork of pytest-vcr. But why was the fork done and what is the relationship to pytest-vcr? When should one use one or the other?

Use the plugin on the autouse fixture

Is there a way to use this plugin on the autouse fixture? Scenario: the test introduces a side effect to HTTP service, so it would be good to have autouse setup/teardown which removes this side-effect allowing the whole test suite to run properly when I delete all cassettes and record all of them.

[FEATURE] Support pytest config file options

Is your feature request related to a problem? Please describe.
Support options in pytest config file

Describe the solution you'd like

[pytest]
block_network = True
ignore_localhost = True
allowed_hosts = [ "::1" ] # Ipv6 localhost isn't considered localhost.

Describe alternatives you've considered
I know I can use addopts, but my addopts line is getting very, very long... ;)

Additional context
Great plugin, love it!

[FEATURE] Parameterize vcr_config

Is your feature request related to a problem? Please describe.

I'm working on a library that supports multiple web services behind a single interface.

Now I've written some tests for that interface and would like to run them for each of the web services I support. This is where pytest's fixture parameterization shines and it works well without VCR in the picture, the implementations simply make real requests.

Obviously that's not ideal, so I wanna use VCR. I need different configurations for each web service though. Unfortunately I can't find a way to parameterize vcr_config.

Describe the solution you'd like

No idea, sorry.

Describe alternatives you've considered

I guess I could use the vcr fixture somehow?

[FEATURE] usage for doctests

Foreword: thank you for this plugin, I like the API, and it is really useful.

Is your feature request related to a problem? Please describe.
The plugin works very well for normal tests, but I'd really like to use it with doctests.
I run those with pytest --doctest-modules and/or pytest --doctest-glob="*.rst". I can define a conftest.py file, but in it I can define fixtures, and not tests, so I cannot use the pytest.mark.vcr

Describe the solution you'd like
I'd like to be able to configure the tests to use a cassette/several cassettes for the tests in (say) the README and the modules, and then the pytest --doctest-modules would record and use the cassette instead of calling the network.

Describe alternatives you've considered
I can add this in conftest.py:

from vcr import use_cassette

@pytest.fixture(autouse=True)
def with_cassette():
    with use_cassette("tests/doctests/cassette.yaml", record_mode="once"):
        yield

But this does not respect whatever will be passed to the command line arguments as it hardcodes the cassette and record mode. If it creates a cassette for each file, that would be better, instead of using one big cassette.

[FEATURE] Tap into additional vcr_config parameters in block_network for allowed_hosts

This is kinda a chimeric issue. Related to my #68, block_network should also have the option to set allowed_hosts via vcr_config, imho.

But then I thought "aren't the ignore_localhost and ignore_hosts parameters from the vcr_config basically the same as allowed_hosts?". VCRpy is adding localhost, 0.0.0.0 and 127.0.0.1 if ignore_localhostis truthy, and also adds ignore_hosts to the set.

I think it would make for a more seamless integration with vcrpy if allowed_hosts is also using ignore_localhost and ignore_hosts from the vcr_config to construct the list while retaining backwards compatibility :-)

new_episodes doc ?

I wonder what is the intended logic behind the 'new_episodes' option ? I couldn't find it in the README, and I had some "strange" behavior with it...

I ran multiple requests to the same endpoint, in one vcr with --record-mode=new_episodes.
But only the first attempt of each request was registered.

So something like :

import time

import pytest
import aiohttp


@pytest.mark.asyncio
@pytest.mark.vcr()
async def test_multi_call():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://httpbin.org/get') as resp:
            print(resp.status)
            print(await resp.text())

        async with session.get('http://httpbin.org/ip') as resp:
            print(resp.status)
            print(await resp.text())

        time.sleep(1)
        async with session.get('http://httpbin.org/get') as resp:
            print(resp.status)
            print(await resp.text())


        async with session.get('http://httpbin.org/ip') as resp:
            print(resp.status)
            print(await resp.text())

if __name__ == '__main__':
    #pytest.main(['-s', __file__, '--block-network'])
    # record run
    pytest.main(['-s', __file__,'--record-mode=new_episodes'])

Will store only 2 requests/responses, one for http://httpbin.org/get and one for http://httpbin.org/ip.

I was naively expecting to record all requests/responses in one run, and then record new cassettes if a difference between the registered cassettes and the current "run" was found (calling to one more url, or not calling an expected url, etc.)...

So does "episode" means "url" and not "run" ?
Thanks for any info.

Consider always keeping the cassette with default name

For example if there are a test and a cassette for it and then we need to add some common data:

@pytest.mark.vcr("extra.yaml")
def test_something():
    pass

In this case, we will look only into extra.yaml because of this - https://github.com/kiwicom/pytest-recording/blob/master/src/pytest_recording/plugin.py#L84

However, it is useful to be able to add some extra common data and still using some specific cassette for the test (named by the default rules). Now it is possible only with specifying it manually in the vcr mark

Stripe library (urlfetch) just waits with `block_network`

Hey,

Thanks a lot for this library!

I managed to get everything working, except that when I run our test suite with block_network it just waits and nothing happens.

VCR functionality does work even if I don't pass the flag so I can run the tests with recorded network communication data, but it seems for some reason the Stripe library (using urlfetch apparently) just waits when the network is blocked.

I guess maybe #16 would solve this issue?

Flag to turn on/off vcr

Hi,
I'd like to have some flag (the best option env variable), to turn on/off vcr for tests marked with @pytest.mark.vcr.
By default it should work as is. But if I set env variable "USE_VCR=false" it should run real http requests and no cassetes should be rewritten. Or it is possible to achieve somehow with CLI options, custom decorators, etc?

How to setup common VCR config for all tests

The following config seems not working under tests/__init__.py, but works within tests/test_foo.py, tests/test_bar.py...

import pytest


@pytest.fixture(scope='module', autouse=True)
def vcr_config():
  return {'ignore_hosts': ['oauth2.googleapis.com']}

Add isort

We need to sort imports in the repo + validate it on CI

Doesn't seem to work with requests inside a Twisted Reactor process

Hi, I'm new to Python so apologies in advance if this is obvious or I don't explain things very well.

For a learning project I'm writing a web scraper with tests. In the following example it works as expected with the commented out line but requests inside the CrawlerProcess/Twisted Reactor aren't being picked up.

    @pytest.mark.vcr
    def test_parse(self):
        # assert requests.get("http://httpbin.org/ip").text == '{"ip": true}'
        BaseSpider.start_urls = ['thoughtbot']

        BaseSpider.custom_settings = {
            'ROBOTSTEXT_OBEY': False
        }

        process = CrawlerProcess()
        process.crawl(BaseSpider)
        process.start()

I'm not sure if this is a bug or I'm just doing something wrong?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.