hypothesmith-inspired compiler fuzzer used to validate PEP 709 implementation

carljm

Hi! Thought you might be interested to know that I wrote (and @JelleZijlstra greatly improved) a compiler fuzzer to validate the implementation of PEP 709 (inlined comprehensions) in Python 3.12: https://github.com/carljm/compgenerator I started out with Hypothesmith, but I wanted to constrain the generated examples a lot more, and as I started replacing Hypothesmith strategies one by one, I realized that it would be a lot simpler to work with AST rather than with LibCST. We're fuzzing the compiler, not the parser, so we don't care about syntactic trivia. The resulting fuzzer works pretty well (although it's slow, because we filter too much; I haven't worked out yet how to better constrain generation so we don't have to filter so much), and it's caught, so far, I think five or six different bugs in the PEP 709 implementation (all now fixed.) No particular action item here, feel free to close :) I guess one possible thing to consider would be whether AST-based strategies would make any sense for Hypothesmith, for non-parser-focused uses.

Zac-HD

Awesome! It seems like #28 would be a nice way to ensure that there are always some listcomps without rejection sampling - although the implementation is a bit of a lift to cover all node types. Relatedly I love that you used the AST for easier implementation, but that doesn't seem to help much here if we want to have a cst-based strategy too. The runtime gains we can probably get with "swarm testing" to probabilistically disable all the weird flourishes - almost as fast as doing it always, but a bit less cognitive load (from fewer options) and a chance of finding unexpected bugs too...