doctests about pytest-stress

[nicoddemus]

So the run without pytest-stress passes, indeed seems like a problem with pytest-stress, thanks for confirming it.

I will defer now to the maintainers to investigate further. 😁

[nicoddemus]

Hi @peisenha,

I suggest trying to run it using python -m doctest -v temfpy/package/temfpy/uncertainty_quantification.py and see if you have the same results. This will help narrow down if it is a doctest related problem or if it is related to pytest-stress.

I have looked at the code in question and indeed simple_linear_function is defined in the same module, so things should just work... 🤔

[peisenha]

Thanks for the swift response. Here is the output.

(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ python -m doctest -v temfpy/uncertainty_quantification.py
Trying:
    x = [1, 2, 3, 4, 5, 6, 7, 8]
Expecting nothing
ok
Trying:
    y = borehole(x)
Expecting nothing
ok
Trying:
    np.testing.assert_almost_equal(y, 34.43500403827335)
Expecting nothing
ok
Trying:
    x = [1, 2, 3]
Expecting nothing
ok
Trying:
    y = eoq_model(x, r=0.1)
Expecting nothing
ok
Trying:
    np.testing.assert_almost_equal(y, 18.973665961010276)
Expecting nothing
ok
Trying:
    x = [1, 2, 3]
Expecting nothing
ok
Trying:
    y = ishigami(x)
Expecting nothing
ok
Trying:
    np.testing.assert_almost_equal(y, 10.037181146302519)
Expecting nothing
ok
Trying:
    x = [1, 2, 3]
Expecting nothing
ok
Trying:
    y = simple_linear_function(x)
Expecting nothing
ok
Trying:
    np.testing.assert_almost_equal(y, 6)
Expecting nothing
ok
1 items had no tests:
    uncertainty_quantification
4 items passed all tests:
   3 tests in uncertainty_quantification.borehole
   3 tests in uncertainty_quantification.eoq_model
   3 tests in uncertainty_quantification.ishigami
   3 tests in uncertainty_quantification.simple_linear_function
12 tests in 5 items.
12 passed and 0 failed.
Test passed.

[nicoddemus]

Thanks! So indeed it seems a problem with pytest-stress or pytest.

Can you please uninstall pytest-stress and execute pytest again?

Also please post the full output you see in the terminal on failure.

[peisenha]

Sorry for the delay. I am back at the keyboard and will respond quicker going forward now. Here you go ..

(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ py.test
Test session starts (platform: linux, Python 3.7.7, pytest 5.4.2, pytest-sugar 0.9.3)
Using --randomly-seed=1591724936
rootdir: /home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package, inifile: tox.ini
plugins: flakes-4.0.0, sugar-0.9.3, randomly-3.4.0, black-0.3.8, hypothesis-5.11.0, cov-2.8.1
collecting ... 
 temfpy/tests/test_uncertainty_quantification.py ✓✓✓✓✓✓                                                                                                                                                                                                             100% ██████████
 temfpy/tests/__init__.py ✓✓                                                                                                                                                                                                                                         29% ██▉       
 temfpy/uncertainty_quantification.py ✓✓✓✓✓✓                                                                                                                                                                                                                         97% █████████▋
 temfpy/optimization.py ✓✓✓✓                                                                                                                                                                                                                                         90% █████████ 
 docs/source/conf.py ✓✓                                                                                                                                                                                                                                              71% ███████▏  
 temfpy/__init__.py ✓✓                                                                                                                                                                                                                                               65% ██████▌   
 setup.py ✓✓                                                                                                                                                                                                                                                         94% █████████▍
 bin/run-figures ✓                                                                                                                                                                                                                                                   39% ███▉      
 scripts/run_figures.py ✓✓                                                                                                                                                                                                                                           74% ███████▌  
 docs/_static/codes/fig-eoq-tradeoff.py ✓✓                                                                                                                                                                                                                           84% ████████▍ 
 temfpy/config.py ✓✓                                                                                                                                                                                                                                                 87% ████████▊ 
================================================================================================================================ warnings summary =================================================================================================================================
/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_flakes.py:51: 11 tests with warnings
  /home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_flakes.py:51: PytestDeprecationWarning: direct construction of FlakesItem has been deprecated, please use FlakesItem.from_parent
    return FlakesItem(path, parent, flakes_ignore)

/home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_black.py:26: 10 tests with warnings
  /home/peisenha/local/anaconda3/envs/temfpy/lib/python3.7/site-packages/pytest_black.py:26: PytestDeprecationWarning: direct construction of BlackItem has been deprecated, please use BlackItem.from_parent
    return BlackItem(path, parent)

temfpy/tests/test_uncertainty_quantification.py::test_simple_linear_function
  /home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:200: RuntimeWarning: invalid value encountered in double_scalars
    return sum(x)

temfpy/tests/test_uncertainty_quantification.py::test_borehole
  /home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:60: RuntimeWarning: divide by zero encountered in double_scalars
    rslt = a / (b * (1 + c / d + e))

temfpy/tests/test_uncertainty_quantification.py::test_borehole
  /home/peisenha/external-storage/ownCloud/office/OpenSourceEconomics/temfpy/package/temfpy/uncertainty_quantification.py:60: RuntimeWarning: invalid value encountered in double_scalars
    rslt = a / (b * (1 + c / d + e))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================================================================================== Hypothesis Statistics ==============================================================================================================================

Results (2.48s):
      31 passed
(base) peisenha@heracles:~/OpenSourceEconomics/temfpy/package$ 

[ImXron]

@nicoddemus Thanks for that initial troubleshooting!

@peisenha I'll check this out tonight after work. Thanks!

[peisenha]

Thanks!

[ImXron]

So I did some initial digging and found that:

• The tests pass the first loop but fail for the loops after that.
• The same 6 tests consistently fail:

  FAILED temfpy/optimization.py::temfpy.optimization.ackley
  FAILED temfpy/optimization.py::temfpy.optimization.rastrigin
  FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.borehole
  FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.eoq_model
  FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.ishigami
  FAILED temfpy/uncertainty_quantification.py::temfpy.uncertainty_quantification.simple_linear_function
  

  So I fixed it by disabling those 👆 tests. Just kidding!
• The unit tests consistently pass:

  temfpy/tests/test_uncertainty_quantification.py::test_eoq_model PASSED
  temfpy/tests/test_uncertainty_quantification.py::test_ishigami PASSED
  temfpy/tests/test_uncertainty_quantification.py::test_borehole PASSED
  temfpy/tests/test_uncertainty_quantification.py::test_simple_linear_function PASSED
  

Seems like some state may not exist due to the looping nature of pytest-stress, compared to a fresh pytest session. I'm also not familiar with doctest to be honest.

I basically copied the logic from pytest's main.py and wrapped it in a while loop.

I'll do more of deep dive this weekend (my brain is too fried on the weekdays 🤯).

Thanks for your patience!

[ImXron]

@peisenha I did some more digging.

It seems like doctest namespace gets cleared out after pytest-stress loop.

In the mean time, one quick hack you can do is explicitly add an import statement for the associated function in each doctest:

>>> from temfpy.optimization import ackley
>>> x = [0, 0]
>>> y = ackley(x)
>>> np.testing.assert_almost_equal(y, 0)

Is your use-case to run your tests for a certain amount of time or perhaps number of times (pytest-repeat)?

@nicoddemus Sorry to bug you, but do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).

Thanks!

[peisenha]

Dear Imran, Thank you very much for your effort. Keep us posted! I am just integrating pytest-stress in this toy package first to learn about it more to then use it for our fleshed out research codes. There, with their long running test batteries, the features would be very useful. We have hacked sth. similar together which we are desperate to get rid of :) Thanks again!

…

--- Philipp Eisenhauer Economist Mail [[email protected]](mailto:[email protected]) Web www.eisenhauer.io Repository https://github.com/peisenha Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, June 15, 2020 1:44 AM, Imran Mumtaz ***@***.***> wrote: ***@***.***(https://github.com/peisenha) I did some more digging. It seems like doctest namespace gets cleared out after pytest-stress loop. In the mean time, one quick hack you can do is explicitly add an import statement for the associated function in each doctest: >>> from temfpy . optimization import ackley >>> x = [ 0 , 0 ] >>> y = ackley ( x ) >>> np . testing . assert_almost_equal ( y , 0 ) Is your use-case to run your tests for a certain amount of time or perhaps number of times ([pytest-repeat](https://github.com/pytest-dev/pytest-repeat))? ***@***.***(https://github.com/nicoddemus) Sorry to bug you, but do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code). Thanks! — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#4 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AA3QYJYOIBOZWZAQXXBXLJLRWVOGNANCNFSM4NZRFKTA).

[nicoddemus]

Hi @ImXron,

do you have any thoughts on this? I'm happy to investigate further if you could point me in some direction (Pytest source code).

pytest itself doesn't meddle with the doc test runner globals:

https://github.com/pytest-dev/pytest/blob/4f4c2638d0e15076bc959fa7ec87d0dd8ac2b46b/src/_pytest/doctest.py#L269-L289

But doctest itself will clear the namespace at the end of the test:

https://github.com/python/cpython/blob/25f38d7044a3a47465edd851c4e04f337b2c4b9b/Lib/doctest.py#L1490-L1493

So your questions about the same test item being run more than once are pertinent, can @peisenha confirm it?

[peisenha]

Yes, it passes the first time but then fails for the subsequent iterations.