Comments (14)
Thanks, I will start with #3764 and then take on the different issue on #3914.
from hypothesis.
yeah, this is a hard one to parallelize 😄. Some of the steps may subtly depend on others in ways that aren't obvious until one is knee deep in implementing it.
Nice! I agree with the reasoning here. Added a task for this. This probably needs to be the absolute last thing to switch to the ir.
from hypothesis.
Definitely the last thing to switch, I just got nerdsniped 😅
from hypothesis.
467ab23
(#3924) uses a nocover pragma to get the PR merged after I reduced our use of ParetoFront
- I think it was tested mostly by accident before, but you'll have a better sense than I for where deliberate tests should go.
from hypothesis.
This should be covered by test_data_with_misaligned_ir_tree_is_invalid
. I think the coverage there is just flaky because the condition is too permissive. Will address in #3923 by splitting the test
from hypothesis.
That would work! I'm also fine with the IR draw_integer
remaining non-uniform above 24 bits, if that's easier.
from hypothesis.
This is super interesting!
Thank you for writing this detailed issue.
I would like to get involved with hypothesis.
What would constitute a good first contribution here?
from hypothesis.
Welcome, Jonathan! We'd love to have you continue contributing - I already really appreciate the type-annotation-improvements for our numpy and pandas extras, so this would be a third contribution 😻
@tybug might have some ideas here, but my impression is that the "refactor for an IR" project in this issue is more-or-less a serialized set of tasks and so adding a second person is unlikely to help much - even with just one we've had a few times where there were two or three PRs stacked up and accumulating merge conflicts between them.
As an alternative, #3764 should be a fairly self-contained bugfix. On the more ambitious side, #3914 would also benefit from ongoing work on that - testing, observability, reporting whatever bugs you surface, etc. Or of course you're welcome to work on any other open issue which appeals to you!
from hypothesis.
We may still use the bitstream representation for some things (database?).
I was thinking that we'd still serialize to a bytestring - that's the ultimate interop format, and when we need to handle weird unicode and floats like subnormals or non-standard bitpatterns for nan
I don't want to trust whatever database backend our users cook up to round-trip correctly. Existing formats like protobuf or msgpack all have constraints like "unicode strings must be valid utf-8" or "numbers limited to bits", so I wrote a custom serializer instead 🙂
from hypothesis.
I'm working on migrating shrinker block programs. Our upweighting for large integer ranges is giving the shrinker trouble, because it means that a simpler tree can result in a longer buffer: the buffer runs through the weighted distribution and draws n
bits from some small bucket, while the tree runs through the uniform distribution (as a result of forced=True
) and draws m > n
bits, where the difference in m
and n
is large enough that it offsets whatever simplification is made by the tree.
Real example of this:
b1 = b'\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x01\x00\x01\x00\x00\x00\x00'
b2 = b'\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00'
s = st.lists(st.integers(0, 2**40))
print("complex result, smaller buffer", ConjectureData.for_buffer(b1).draw(s))
# complex result, smaller buffer [0, 0, 0, 0, 0]
print("simpler result, larger buffer", ConjectureData.for_buffer(b2).draw(s))
# simpler result, larger buffer [0, 0, 0, 0]
As a result I'd like to look at moving that weighting logic into IntegerStrategy
, which imo is where it logically belongs anyway, not at the ir layer. To accommodate this with weights
, we'll need a structure that can express weights for entire ranges, not just "weight points and everything else is uniform". What do you think of weights=[(a, b, p), ...]
where union((a, b), ...) == [min_value, max_value]
, sum(p) == 1
, and len((a, b), ...) <= 255
?
from hypothesis.
What if we forced even more instead?
If we choose a smaller bits
size, instead of drawing the main value from a narrower range we draw a value-to-force from the narrower range, and then force-draw it from the full range. The choice of fewer bits is then cleanly deletable without changing the interpretation of subsequent bits.
from hypothesis.
We could do that! I'm fairly confident exactly what you stated, or some small variation, would work.
I was thinking of killing two birds with one stone here, though. Do you think the upweighting belongs in the ir or in st.integers()
? If we're going to move it out of the ir eventually anyway, I think now is the right time to do it, both while it's causing problems and we're changing the weights
interface.
from hypothesis.
I think doing it 'below' the IR, so we just represent a single integer value with a minimum of redundancy, is the principled approach here. "Literally just give me an integer" feels like it should be bijective 😅
from hypothesis.
The concern is that moving the weighting to st.integers()
will result in drawing an integer correspond to more than one ir draw? I think we can avoid this via weights
(and wouldn't want to move the weighting if we couldn't). I was thinking of something like this, where we combine the probability distributions upfront and pass it to weights
. We wouldn't need to draw a boolean with p=7/8. Probability computations are pseudocode for whatever representation we use.
class IntegersStrategy(SearchStrategy):
...
def do_draw(self, data):
weights = None
if self.end is not None and self.start is not None:
bits = (self.end - self.start).bit_length()
# For large ranges, we combine the uniform random distribution from draw_bits
# with a weighting scheme with moderate chance. Cutoff at 2 ** 24 so that our
# choice of unicode characters is uniform but the 32bit distribution is not.
if bits > 24:
def weighted():
# INT_SIZES = (8, 16, 32, 64, 128)
# INT_SIZES_SAMPLER = Sampler((4.0, 8.0, 1.0, 1.0, 0.5), observe=False)
total = 4.0 + 8.0 + 1.0 + 1.0 + 0.5
return (
(4.0 / total) * (-2**8, 2**8),
# ...except split these into two ranges to avoid double counting bits=8
(8.0 / total) * (-2**16, 2**16),
(1.0 / total) * (-2**32, 2**32),
(1.0 / total) * (-2**64, 2**64),
(0.5 / total) * (-2**128, 2**128),
)
weights = (
(7 / 8) * weighted()
+ (1 / 8) * uniform()
)
# for bounded integers, make the near-bounds more likely
weights = (
weights
+ (2 / 128) * self.start
+ (1 / 64) * self.end
+ (1 / 128) * (self.start + 1)
+ (1 / 128) * (self.end - 1)
)
# ... also renormalize weights to p=1, or have the ir do that
return data.draw_integer(
min_value=self.start, max_value=self.end, weights=weights
)
Now the ir draw_integer
is truly uniform, but st.integers()
keeps the same distribution as before.
from hypothesis.
Related Issues (20)
- Die Hard example uses outdated style of defining settings HOT 2
- Logging each test before running it HOT 1
- Can we consider optionally disabling automatic repr-printing and jsonification when `TESTCASE_CALLBACKS` are present? HOT 2
- How to exclude property? for `hypothesis.errors.ResolutionFailed: Could not infer a strategy for <django.db.models.fields.AutoField: id>` HOT 1
- Fix missing imports for `hypothesis write numpy.save`
- from_type fails on nested annotated type with mysterious error HOT 7
- `hypothesis.strategies.text` fails if min_size is (unreasonably) large HOT 2
- hypothesis.errors.InvalidArgument: min_size=8_640 is larger than Hypothesis is designed to handle HOT 1
- numpy arrays() strategy does not respect (unsigned) dtype (anymore) HOT 1
- Some type annotations for `hypothesis.extra.pandas` are missing HOT 1
- Generate `st.fixed_dictionaries()` with varied iteration orders
- Empty dictionaries generated with st.dictionaries and st.just despite expected behavior HOT 2
- Implement an Atheris fuzzing backend
- Stable support for symbolic execution HOT 15
- TypeError on CPython 3.13 for 6.99.2 HOT 1
- Investigate and fix lambda bug on Python 3.13
- Structural difference between `sampled_from` and `one_of`: motivation and possible unification? HOT 2
- Internal error: tests using unique collections containing `nan` are flaky HOT 5
- Ideas for some strategy optimizations HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hypothesis.