Comments (6)
Yeah, ok. That makes sense. Since Tyche is about live feedback, I'm tempted to just ignore any data more than a minute or two older than the newest data, but I'll play around with those heuristics.
from hypothesis.
Our schema doesn't have any concept or way to represent an invocation of a test suite, only invocations of individual test functions. This is harder to pin down than it might sound - a short list of options includes:
- Manually calling an
@given()
-decorated function (hereafter "test"; the "property" terminology made sense for Haskell but not Hypothesis)- in a short-lived process / via a script
- in a long-lived process such as a notebook
- Running via
doctest
- Running with
unittest
- default serial configuration
- custom parallel configuration (OK if we don't support this, but avoiding nonsense would be nice)
- Running with
pytest
- single process/thread
- multiprocess with
pytest-xdist
- have to pass around the start timestamp somehow - selecting a subset of available tests, by filename or e.g.
-k
selector - (combinations of these)
- Test runs triggered via editor integrations (possibly via pytest or unittest?)
- Overlapping runs, e.g. triggering a specific test in-editor while waiting for a longer run which uses pytest-xdist
So logging the start and end timestamps of a suite (e.g. as information messages) is unsatisfactory; we do actually need to include some kind of suite identifier in the test-case message. Presumably we'll want to use something like an optional test-suite
key under metadata
, containing timestamp, string identifier, and maybe some other metadata about how it was invoked such as sys.argv
or 'via vs code'. With some work we can get all of that for pytest runs via our own pytest plugin, though for others including unittest I don't think we can do much better than sys.argv
and maybe the parent process ID (and timestamps? needs a psutil
dependency...).
Thoughts?
from hypothesis.
Yeah, I suspected this would be the sticking point. But your proposed solution of a best effort test_suite
field sounds good to me, and the heuristics you're proposing for collecting that data also make sense.
I'm not sure if it should be under metadata or lifted to the top level. My perspective has been that metadata contains information that generic tools can't depend on but that users could choose to analyze manually if they're inclined. If we want downstream analyses to be able to use this to group properties, I'd suggest we make it a top-level data field.
from hypothesis.
I lean towards metadata because I don't think tools can depend on it, even from Hypothesis - we can do best-effort, but there are certainly going to be cases and users for whom it doesn't work at all 😢
from hypothesis.
Hmm, ok. I buy that.
Then that leaves open the question of what tools should do when they have a pile of data and no way to know which of those lines are fresh enough to show. Should we consider a time horizon (e.g., an hour or a day)? Should we just show the latest data from every property that's available?
from hypothesis.
🤷 depends on the frontend tool I think!
I think it'd be reasonable to show a little ⓘ this test was run {duration} ago
if it's older than the user might expect. Heuristic grouping by per-test timestamp probably also works pretty well - look at the distribution of intervals between sorted start times, and your most recent large outlier is probably the gap between suite runs.
from hypothesis.
Related Issues (20)
- Failing test for Django 5.0 HOT 1
- Using `builds` arguments for reprs may produce worse results than pretty printing HOT 2
- Improve error message when a package only has submodules for ghostwriter HOT 3
- Improve support for new and user-defined Numpy dtypes (e.g. `np.dtypes.StringDType`)
- Change Flaky to be an ExceptionGroup
- Error while patching time.perf_counter HOT 4
- Thread safety when usage with Pytest, and Coverage cause unexpected hang at exit HOT 3
- Falsifying examples not being printed HOT 8
- System.AccessViolationException with Hypothesis 6.108.x HOT 3
- Improve pretty-printer for overly long integers and signalling NaN
- Support `fuzz_one_input` on instance methods
- test.ghostwriter.py:test_obj_name fails because of tmp dir HOT 3
- Investigate `Examples` performance
- Python 3.14: StructuralCoverageTag.__init__() takes 1 positional argument but 2 were given HOT 9
- Improve support of `TypedDict`
- Support `LiteralString`
- Question: Write strategies to fuzz docstrings matching different styles (google, numpydoc, sphinx) HOT 3
- Support `TypeIs`
- Support `TypeVar` default
- Limit the maximum number of values reuse in `consume` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hypothesis.