pyvista / pytest-pyvista Goto Github PK

IMO, this default value is not the best choice:

I'd like to request some clarification on this choice and propose altering it to use the current working directory.

I would expect pytest . to properly handle the image cache or default to save the cache in a location that is visible/relative to the active project. This current default dumps the images in a non-visible location.

An alternative option is to have cache_dir be easily stable in the conftest.py for a project -- is this currently possible?

The flag is doing nothing for me locally

This is useful when used together with pytest-rerunfailures, and it can help to also with try context managers.

Let's add some solid documentation for this tool. I think it can be used just about anywhere that uses pyvista, and in many places that don't that simply want to compare images.

Documentation should use sphinx and follow the pydata-theme. Ping @banesullivan to create a new subdomain for this. Something like "pytest.pyvista.org".

🐞 Problem
It looks like the development version, the one in main, is 0.1.dev0 even if there has been a release/0.2. Therefore, a pull-request should be opened to update the version number in main.

It would be useful to run the full pyvista tests on one python/vtk version in this package to avoid unintended breakages downstream.

The old behavior was that if the image was missing, it would be created. Now you have to use --reset_image_cache to generate figures for new tests.

Can we make it so that if the cached image is missing, it gets generated?

I'm dealing with a scenario right now where a lot of tests are failing the image regression test on Windows but not other platforms on CI. It would be nice if there were an option to dump the failing images to a specific path so that I can upload those as a CI artifact to inspect. I'm currently manually doing this

See #49 for some effects where it seems that verify_image_cache is leaking outside of test scope.

We should teardown, e.g. removing the object from the theme, to reduce cross talk, after each test.

🐞 Problem
the appveyor.yml file got generated as a consequence of the cookiecutter template. This file is used to configure the CI/CD in the Appveyor platform. However, it looks like this repository takes advantage of GitHub actions for CI/CD purposes. Therefore, it looks like this file is no longer needed.

Currently, VerifyImageCache class attributes are accessed directly. Instead we should be using getters and setters to read and modify the attributes.

In Pytest you can do:

import pyvista as pv

@pytest.mark.parametrize(
    "title",
    [
        "Mechanical",
        "mechanical",
        "meCHANICAL",
        "ux",
        "UX",
        "CSGZ",
    ],
)
def test_bc_plot_bc_labels(title):

    # Create a simple mesh (a sphere)
    mesh = pv.Sphere()

    # Create a plotter object with a custom title
    plotter = pv.Plotter(title=title)

    # Add the mesh to the plotter
    plotter.add_mesh(mesh)

    # Set up the plotter window and display the plot
    plotter.show()

Pytest won't be able to generate "mechanical" and "meCHANICAL". Nor "UX".

Reference:

• https://github.com/ansys/pymapdl/actions/runs/6379994959/job/17313552654?pr=2349#step:10:2383
• https://github.com/ansys/pymapdl/blob/727dde9d65c6d30dace1efa8a175f1e659074606/tests/test_plotting.py#L163

It would be very useful to have an option to check that all cached images were compared during a test. This is a complement to checking that all comparisons have an image in the cache.

This is tricky if tests are skipped for any reason. So, it would be only useful if there is a canonical version for which no test is skipped. I'm not sure if this is the case today for pyvista. It is very possible that at least one plotting test is skipped on every vtk version. But, this could be useful for downstream libs that have less wide-open vtk version support.

I've seen a few times recently where installing pytest-pyvista after having a dev install of pyvista causes a reinstall of pyvista from PyPI. I believe this is due to the pyvista>=0.37 constraint on the package and the fact that pyvista is currently at 0.38.dev0 on main which pip doesn't handle gracefully. I suspect this issue will resolve on the next MINOR version release of pyvista, but until then....

Is pyvista>=0.37 really necessary for this extension? I'm wondering if we could free that constraint or use an earlier compatible version?

Having a dedicated release branch is massive overkill for this package and adds overhead to cutting releases. We should switch the versioning mechanism to use setuptools_scm and tag directly on the main branch.

@pyvista/developers, thoughts?

Hi @pyvista/maintainers, pinging @germa89 for visibility.

I was wondering if I could make a release for the plugin, since we need to use the new flag introduced in #82

If that's okay, I will proceed.

Pytest arguments replace the spaces by minus (-) when defining arguments. For example:

pytest --durations=10 --durations-min=1.0

https://docs.pytest.org/en/6.2.x/usage.html#profiling-test-execution-duration

whereas in pytest-pyvista all the arguments are separated by "_".

This is a minor thing. But I just wanted to highlight it for consistency.

This would enable reducing duplicate images in tests. For example, tests in pyvista/pyvista#5428 generate multiple images that are duplicates of one another. It could be something like:

def test_identical_images(verify_image_cache):
    verify_image_cache.file_compare = "identical_images"
    # generate image with first plot and compare to "identical_images.png"
    verify_image_cache.file_compare = "identical_images"
    # generate image with second plot and compare to same "identical_images.png"

I think this is a very specific need, but it might be useful to somebody else.

I would like to generate the cache only for the images that fails.

The flags --fail_extra_image_cache and --add_missing_images are incompatible with each other. This is not documented anywhere. We should:

• Update the documentation to cover this.
• Choose what flag will take priority in case of conflict.
• Raise a warning in the code to notify that both flags are incompatible when both flags are active, and tell the user which one is prioritized.