nschloe / pytest-codeblocks

:page_facing_up: Test code blocks in your READMEs

Test code blocks in your READMEs.

This is pytest-codeblocks, a pytest plugin for testing code blocks from README files. It supports Python and shell code.

Install with

pip install pytest-codeblocks

and run pytest with

pytest --codeblocks

================================= test session starts =================================
platform linux -- Python 3.9.4, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /path/to/directory
plugins: codeblocks-0.11.0
collected 56 items

example.md .......................                                              [ 50%]
README.md .......................                                               [100%]

================================= 56 passed in 0.08s ==================================

pytest-codeblocks will only pick up code blocks with python and sh/bash/zsh syntax highlighting.

Marking code blocks

It is possible to use pytest.mark for marking code blocks. For example, to skip a code block use pytest.mark.skip or pytest.mark.skipif:

Lorem ipsum

<!--pytest.mark.skip-->

```python
foo + bar  # not working
```

dolor sit amet.

<!--pytest.mark.skipif(sys.version_info <= (3, 7), reason="Need at least Python 3.8")-->

You can skip code blocks on import errors with

<!--pytest-codeblocks:importorskip(sympy)-->

Skip the entire file by putting

<!--pytest-codeblocks:skipfile-->

in the first line.

For expected errors, use pytest.mark.xfail:

The following gives an error:

<!--pytest.mark.xfail-->

```python
1 / 0
```

Merging code blocks

Broken-up code blocks can be merged into one with the pytest-codeblocks:cont prefix

Lorem ipsum

```python
a = 1
```

dolor sit amet

<!--pytest-codeblocks:cont-->

```python
# this would otherwise fail since `a` is not defined
a + 1
```

If you'd like to prepend code that you don't want to show, you can just comment it out; pytest-codeblocks will pick it up anyway:

Lorem ipsum

<!--
```python
a = 1
```
-->

dolor sit amet

<!--pytest-codeblocks:cont-->

```python
# this would otherwise fail since `a` is not defined
a + 1
```

Expected output

You can also define the expected output of a code block:

This

```sh
print(1 + 3)
```

gives

<!--pytest-codeblocks:expected-output-->

```
4
```

Use expected-output-ignore-whitespace if you'd like whitespace differences to be ignored.

(Conditionally) Skipping the output verfication works by prepending the first block with skip/skipif (see above).

pytest-codeblocks's Issues

"could not get source code"

Hello I am getting this error message with a project that uses a native extension I cannot share.

Any idea as to what is going on (exec not being able to load it ?) or what kind of workaround could be used ?

pytests method fails

testing pytests method on a readme file fails with "function pytests..exec_raise at 0x000001DB8B537160>" message.
extract method runs sucessfully on the same file

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/setup-python action to v5

Detected dependencies

github-actions

.github/workflows/tests.yml

• actions/checkout v4
• pre-commit/action v3.0.0
• actions/setup-python v4
• actions/checkout v4
• codecov/codecov-action v4-beta

pep621

pyproject.toml

• pytest >= 7.0.0

---

• Check this box to trigger a request for Renovate to run again on this repository

custom code discovery

Hi,

I am using pytest-codeblocks to test documentation code snippets that get built with sphinx and the myst_parser. The problem I am running into is that I have to write code blocks like this to get the outuput to show up in documentation:

```{code-cell}
print("mycode")
```

Is there anyway to modify pytest-codeblocks discovery logic to also test these blocks (eg assume "{code-cell}" means "python"

Thanks!

embed codeblock results on README?

• Could there be a mechanism added to this package for when a REAMDE code block is executed to embed the produced images /results directly onto the README automatically?
• I realize this somewhat falls beyond the scope of testing the code block strictly. However, since you open the possibility to test code written on a README with this package, I think naturally it would be useful to have some way to embed the images (or in general result(s)) onto the README without having to manually upload the images to Github and relink them in the README.

Suport for interactive Python shell code?

It is quite often markdown contains interactive Python shell code like:

>>> print("Hello World!")
Hello World!
>>> def add(a, b):
...     return a + b
...
>>> add(2,3)
5

Currently, I got following exception for these blocks in my README.md.

invalid syntax (<string>, line 1)

Can you add support to test it? I expect the behaviour to be same as pytest --doctest-modules, as if the code block is a function's doc-string.

Unicode decoding error under Windows

First off, many thanks for the really nice package!

I started working with it for a collection of Python teaching material in markdown format where I would like to test (some of) the code blocks.
When adding tests for Windows I ran into the following error:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 76: character maps to <undefined>

I later realized it comes from a markdown file that had some non-intended characters (from copy-pasting code):

s = “my text”

In a fork of pytest-codeblocks I now added macos and Windows to the CI and a test running on an example markdown file with the problematic characters.
The error can be "fixed" by adding a 'replace' to the open() function. Could be there's better ways to handle this, but it at least worked for me. --> florian-huber#5

Let me know if that would be interesting for a pull request. Thanks!

I

encoding parameter for open()

thanks for the package
would be great to have an option to pass the encoding parameter to open()

Assign blocks names that are printed in pytest output?

First off, this package is awesome, thank you for putting it together! As a longtime R knitr/sweave user, this fills a massive hole in python for me (different use case, but conceptually similar).

I was wondering if there was any way to add block names to each code block, so that when I run pytest, I can easily tell which tests are which? Right now, it's a little tricky to visually parse this:

============================= test session starts ==============================
platform linux -- Python 3.9.6, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python3
cachedir: .pytest_cache
rootdir: /builds/neuralink/sw/util/python_packages/nlk_utils_n1plot
plugins: codeblocks-0.12.2
collecting ... collected 20 items
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]
README.md::README.md PASSED                                              [  5%]

Also apologies if this isn't an issue, happy to move this somewhere else if need be.

Thanks again!

Supporting Bash code blocks and more advanced combining

Hi, thanks for your work with this repo. I forked this repo a long time ago, added some things to it and then haven't touched it since. I wanted to know your opinion on some of the changes I made and whether they're something you'd be interested in implementing here yourself (or accepting via a pull request).

1. Supporting bash codeblocks. I added this by detecting code blocks for bash and then adding some wrapping around it to run the commands in a subprocess in Python for pytest compatibility. I saved the stdout and validated it using custom Python commands in markdown annotations (see the code and the test case). In theory, you could support any language provided the appropriate backend existed in your code to deal with the language.
2. Combining multiple code blocks. I see you've added this with your 'cont' annotation. I had something similar where it would by default combine with the previous code block but you could also specify an index to combine it with a particular one (see this test case).

It's been a while since I looked at my own code but if there's anything I can do to help, let me know.

max num lines exceeded

I'm using exdown in a unit testing framework.

It's telling me my README is too long to test. How do I set max_num_lines?

____________________ ERROR collecting tests/test_README.py _____________________
tests/test_README.py:15: in <module>
    exdown.extract(this_dir.parent / "README.md", syntax_filter="python"),
.tox/py3/lib/python3.7/site-packages/exdown/main.py:3: in extract
    return from_buffer(handle, *args, **kwargs)
.tox/py3/lib/python3.7/site-packages/exdown/main.py:31: in from_buffer
    f"File too large (> {max_num_lines} lines). Set max_num_lines."
E   RuntimeError: File too large (> 10000 lines). Set max_num_lines.