GithubHelp home page GithubHelp logo

smarie / pytest-patterns Goto Github PK

View Code? Open in Web Editor NEW
35.0 3.0 4.0 9.78 MB

A couple of examples showing how pytest and its plugins can be combined to solve real-world needs.

Home Page: https://smarie.github.io/pytest-patterns

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%
pytest test case decorator parametrize data file separate concerns design

pytest-patterns's Issues

Improve benchmark example

  • add LaTeX table export

  • explain in doc/motivation that one motivation is to avoid research papers errors due to

    • errors in manual copy/paste
    • errors in results coming from distinct evaluation protocol codes (typically copy/paste + modifications just on one side)

data_science_benchmark example fails

Hi,

It looks like the data_science_benchmark fails.

I have followed the steps in https://smarie.github.io/pytest-patterns/examples/data_science_benchmark/

And more specifically, here is what I did ( on an Ubuntu 18.04 machine / docker ) -

# - create venv with requirements
conda create -n pytest-dsb-env python=3.7 pip
conda activate pytest-dsb-env
pip install pytest pytest-cases pytest-harvest tabulate numpy pandas

# - clone
git clone https://github.com/smarie/pytest-patterns.git
git log --oneline | head -n 1
b3db7b3 Removed wrong information from Readme

# - run example
cd pytest-patterns/pytest_patterns
pytest data_science_benchmark -v

And here are the errors:

===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.12, pytest-7.1.0, pluggy-1.0.0 -- /home/mobileye/harelw/.conda/envs/pytest-dsb-env/bin/python3.7
cachedir: .pytest_cache
rootdir: /home/mobileye/harelw/repos-github/pytest-patterns, configfile: setup.cfg
plugins: cases-3.6.10, harvest-1.10.3
collected 13 items                                                                                                                                                                            

data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=1] ERROR                                                                                    [  7%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=2] ERROR                                                                                    [ 15%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=3] ERROR                                                                                    [ 23%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=4] ERROR                                                                                    [ 30%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-v-shape] ERROR                                                                                           [ 38%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-constant-1] ERROR                                                                                        [ 46%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=1] ERROR                                                                                    [ 53%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=2] ERROR                                                                                    [ 61%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=3] ERROR                                                                                    [ 69%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=4] ERROR                                                                                    [ 76%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-v-shape] ERROR                                                                                           [ 84%]
data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-constant-1] ERROR                                                                                        [ 92%]
data_science_benchmark/test_polyfit.py::test_synthesis FAILED                                                                                                                           [100%]

=========================================================================================== ERRORS ============================================================================================
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=1] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=2] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=3] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-anscombes_quartet-id=4] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
______________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-csvfile-v-shape] ______________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
____________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=1-csvfile-constant-1] _____________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=1] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=2] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=3] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
__________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-anscombes_quartet-id=4] ___________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
______________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-csvfile-v-shape] ______________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
____________________________________________________________ ERROR at setup of test_poly_fit[polyfit-degree=2-csvfile-constant-1] _____________________________________________________________
ScopeMismatch: You tried to access the function scoped fixture dataset_data with a session scoped request object, involved factories:
pytest_patterns/data_science_benchmark/test_polyfit.py:32:  def dataset(dataset_data, request)
<makefun-gen-16>:1:  def dataset_data(anscombes_quartet, csvfile, request)
========================================================================================== FAILURES ===========================================================================================
_______________________________________________________________________________________ test_synthesis ________________________________________________________________________________________

module_results_df = Empty DataFrame
Columns: []
Index: []

    def test_synthesis(module_results_df):
        """
        Creates the benchmark synthesis table
        Note: we could do this at many other places (hook, teardown of a session-scope fixture...)
        as explained in `pytest-harvest` plugin
        """
        # ----------- (1) `module_results_df` contains the raw (12 rows) table -----------
        # rename columns and only keep useful information
        module_results_df = rename_with_checks(module_results_df, columns={'challenger_param': 'degree',
>                                                                          'dataset_param': 'dataset'})

data_science_benchmark/test_polyfit.py:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

df = Empty DataFrame
Columns: []
Index: [], columns = {'challenger_param': 'degree', 'dataset_param': 'dataset'}, kwargs = {}, missing = {'challenger_param', 'dataset_param'}

    def rename_with_checks(df, columns, **kwargs):
        """
        Same than df.rename(columns=<columns>, **kwargs) but checks that columns exist before executing.
        """
        missing = set(columns.keys()).difference(set(df.columns))
        if len(missing) > 0:
>           raise ValueError("Missing columns: %s" % missing)
E           ValueError: Missing columns: {'challenger_param', 'dataset_param'}

data_science_benchmark/test_polyfit.py:127: ValueError
=================================================================================== short test summary info ===================================================================================
FAILED data_science_benchmark/test_polyfit.py::test_synthesis - ValueError: Missing columns: {'challenger_param', 'dataset_param'}
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=1]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=2]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=3]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-anscombes_quartet-id=4]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-v-shape]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=1-csvfile-constant-1]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=1]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=2]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=3]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-anscombes_quartet-id=4]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-v-shape]
ERROR data_science_benchmark/test_polyfit.py::test_poly_fit[polyfit-degree=2-csvfile-constant-1]
================================================================================ 1 failed, 12 errors in 0.23s =================================================================================

As more more accurate env details -

❯ conda env export
name: pytest-dsb-env
channels:
  - defaults
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - ca-certificates=2022.2.1=h06a4308_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=11.2.0=h1d223b6_14
  - libgomp=11.2.0=h1d223b6_14
  - libnsl=2.0.0=h7f98852_0
  - libstdcxx-ng=11.2.0=he4da1e4_14
  - libzlib=1.2.11=h36c2ea0_1013
  - ncurses=6.3=h9c3ff4c_0
  - openssl=3.0.0=h7f98852_2
  - pip=22.0.4=pyhd8ed1ab_0
  - python=3.7.12=hf930737_100_cpython
  - python_abi=3.7=2_cp37m
  - readline=8.1.2=h7f8727e_1
  - setuptools=60.10.0=py37h89c1867_0
  - sqlite=3.38.0=hc218d9a_0
  - tk=8.6.12=h27826a3_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - xz=5.2.5=h516909a_1
  - zlib=1.2.11=h36c2ea0_1013
  - pip:
    - attrs==21.4.0
    - decopatch==1.4.10
    - importlib-metadata==4.11.3
    - iniconfig==1.1.1
    - makefun==1.13.1
    - numpy==1.21.5
    - packaging==21.3
    - pandas==1.3.5
    - pluggy==1.0.0
    - py==1.11.0
    - pyparsing==3.0.7
    - pytest==7.1.0
    - pytest-cases==3.6.10
    - pytest-harvest==1.10.3
    - python-dateutil==2.8.2
    - pytz==2021.3
    - six==1.16.0
    - tabulate==0.8.9
    - tomli==2.0.1
    - typing-extensions==4.1.1
    - zipp==3.7.0

Thanks in advance,

Harel

pytest_patterns/test is missing

README mentions how to run like pytest -v pytest_patterns/test but the target folder is missing. When I run pytest -v pytest_patterns it says fixture 'module_results_df' not found

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.