smarie / pytest-patterns

A couple of examples showing how pytest and its plugins can be combined to solve real-world needs.

pytest-patterns

A couple of examples showing how to use core pytest mechanisms and existing plugins (no additional hooks or hacks) to solve real-world problems. In other words:

"pytest for humans ;)"

This is the readme for developers. The documentation for users is available here: https://smarie.github.io/pytest-patterns/

Want to contribute ?

Contributions are welcome ! Simply fork this project on github, commit your contributions, and create pull requests.

Here is a non-exhaustive list of interesting open topics: https://github.com/smarie/pytest-patterns/issues

pytest-patterns's Issues

pytest_patterns/test is missing

README mentions how to run like pytest -v pytest_patterns/test but the target folder is missing. When I run pytest -v pytest_patterns it says fixture 'module_results_df' not found

Build the doc website

Provide an "advanced" benchmark example with

• plots at each node
• steps
• multi-module challengers (one module per challenger class)

data_science_benchmark example fails

Hi,

It looks like the data_science_benchmark fails.

I have followed the steps in https://smarie.github.io/pytest-patterns/examples/data_science_benchmark/

And more specifically, here is what I did ( on an Ubuntu 18.04 machine / docker ) -

# - create venv with requirements
conda create -n pytest-dsb-env python=3.7 pip
conda activate pytest-dsb-env
pip install pytest pytest-cases pytest-harvest tabulate numpy pandas

# - clone
git clone https://github.com/smarie/pytest-patterns.git
git log --oneline | head -n 1
b3db7b3 Removed wrong information from Readme

# - run example
cd pytest-patterns/pytest_patterns
pytest data_science_benchmark -v

And here are the errors:

===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.12, pytest-7.1.0, pluggy-1.0.0 -- /home/mobileye/harelw/.conda/envs/pytest-dsb-env/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/mobileye/harelw/repos-github/pytest-patterns, configfile: setup.cfg
plugins: cases-3.6.10, harvest-1.10.3
collected 13 items                                                                                                                                                                            

data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=1] ERROR                                                                                    [  7%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=2] ERROR                                                                                    [ 15%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=3] ERROR                                                                                    [ 23%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=4] ERROR                                                                                    [ 30%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-v-shape] ERROR                                                                                           [ 38%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-constant-1] ERROR                                                                                        [ 46%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=1] ERROR                                                                                    [ 53%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=2] ERROR                                                                                    [ 61%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=3] ERROR                                                                                    [ 69%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=4] ERROR                                                                                    [ 76%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-v-shape] ERROR                                                                                           [ 84%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-constant-1] ERROR                                                                                        [ 92%]
data_science_benchmark/test_polyfit.py::test_synthesis FAILED                                                                                                                           [100%]

=========================================================================================== ERRORS ============================================================================================
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=1] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=2] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=3] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=4] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
______________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-csvfile-v-shape] ______________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
____________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-csvfile-constant-1] _____________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=1] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=2] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=3] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=4] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
______________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-csvfile-v-shape] ______________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
____________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-csvfile-constant-1] _____________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
========================================================================================== FAILURES ===========================================================================================
_______________________________________________________________________________________ test_synthesis ________________________________________________________________________________________

module_results_df = Empty DataFrame
Columns: []
Index: []

    def test_synthesis(module_results_df):
        """
        Creates the benchmark synthesis table
        Note: we could do this at many other places (hook, teardown of a session-scope fixture...)
        as explained in `pytest-harvest` plugin
        """
        # ----------- (1) `module_results_df` contains the raw (12 rows) table -----------
        # rename columns and only keep useful information
        module_results_df = rename_with_checks(module_results_df, columns={'challenger_param': 'degree',
>                                                                          'dataset_param': 'dataset'})

data_science_benchmark/test_polyfit.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

df = Empty DataFrame
Columns: []
Index: [], columns = {'challenger_param': 'degree', 'dataset_param': 'dataset'}, kwargs = {}, missing = {'challenger_param', 'dataset_param'}

    def rename_with_checks(df, columns, **kwargs):
        """
        Same than df.rename(columns=<columns>, **kwargs) but checks that columns exist before executing.
        """
        missing = set(columns.keys()).difference(set(df.columns))
        if len(missing) > 0:
>           raise ValueError("Missing columns: %s" % missing)
E           ValueError: Missing columns: {'challenger_param', 'dataset_param'}

data_science_benchmark/test_polyfit.py:127: ValueError
=================================================================================== short test summary info ===================================================================================
FAILED data_science_benchmark/test_polyfit.py::test_synthesis - ValueError: Missing columns: {'challenger_param', 'dataset_param'}
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=1]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=2]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=3]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=4]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-v-shape]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-constant-1]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=1]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=2]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=3]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=4]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-v-shape]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-constant-1]
================================================================================ 1 failed, 12 errors in 0.23s =================================================================================

As more more accurate env details -

❯ conda env export
name: pytest-dsb-env
channels:
  - defaults
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - ca-certificates=2022.2.1=h06a4308_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=11.2.0=h1d223b6_14
  - libgomp=11.2.0=h1d223b6_14
  - libnsl=2.0.0=h7f98852_0
  - libstdcxx-ng=11.2.0=he4da1e4_14
  - libzlib=1.2.11=h36c2ea0_1013
  - ncurses=6.3=h9c3ff4c_0
  - openssl=3.0.0=h7f98852_2
  - pip=22.0.4=pyhd8ed1ab_0
  - python=3.7.12=hf930737_100_cpython
  - python_abi=3.7=2_cp37m
  - readline=8.1.2=h7f8727e_1
  - setuptools=60.10.0=py37h89c1867_0
  - sqlite=3.38.0=hc218d9a_0
  - tk=8.6.12=h27826a3_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - xz=5.2.5=h516909a_1
  - zlib=1.2.11=h36c2ea0_1013
  - pip:
    - attrs==21.4.0
    - decopatch==1.4.10
    - importlib-metadata==4.11.3
    - iniconfig==1.1.1
    - makefun==1.13.1
    - numpy==1.21.5
    - packaging==21.3
    - pandas==1.3.5
    - pluggy==1.0.0
    - py==1.11.0
    - pyparsing==3.0.7
    - pytest==7.1.0
    - pytest-cases==3.6.10
    - pytest-harvest==1.10.3
    - python-dateutil==2.8.2
    - pytz==2021.3
    - six==1.16.0
    - tabulate==0.8.9
    - tomli==2.0.1
    - typing-extensions==4.1.1
    - zipp==3.7.0

Thanks in advance,

Harel

Show how easy (hopefully :) ) it is to run "Fashion-MNIST" benchmark

It would be a nice crash-test to reproduce this benchmark: https://github.com/zalandoresearch/fashion-mnist

Doing it in less than 1h coding time would hopefully prove that this is usable to win time on real-world projects

Improve benchmark example

• add LaTeX table export

• explain in doc/motivation that one motivation is to avoid research papers errors due to

  • errors in manual copy/paste
  • errors in results coming from distinct evaluation protocol codes (typically copy/paste + modifications just on one side)

Migrate to github actions