wasm-smith: introduce `modify_exports` option about wasm-tools

[alexcrichton]

I think this is an excellent idea, and I think it'd be reasonable to add!

Now excuse me while I also take a moment here to digress a bit and pontificate slightly about other possible designs here. First though I think it's reasonable to have a PR for this and we can always iterate on it in the future, no need for you to take on my ideas below here.

One nice part about wasm-smith right now is that it's possible to use the CLI to configure all the various options, notably there's a "serializable form" of Config which can make it kind of nice to share with others. This option naturally would not be configurable on the CLI since it's a custom callback. A possible alternative, though, would be to take a leaf from the notebook of available_imports. My thinking would be that we could generalize the available_imports option to "please generate a module of this shape". For example we could rename available_imports to desired_shape and then wasm-smith would guarantee that it would generate a module with at most the imports provided and exactly the exports provided.

That way you could generate a desired shape and then pass it to wasm-smith to manufacture a module of that shape. One consequence of this is that you'd have to have your own version of, for example, deciding whether the module should have an export called canister_heartbeat. The main reason I say this is that I'm not sure how to encode "maybe this can be exported but it also doesn't have to be" as a request to wasm-smith, but having "the module must export this function of this signature" makes sense to me from a wasm-smith point of view. Do you think it'd be easy enough for y'all to generate a random "shape" which adheres to the IC spec?

(and again if it's easiest for you to implement modify_exports that's totally ok to send a PR for, this is just me thinking and something I might make an update towards later)

[fitzgen]

My thinking would be that we could generalize the available_imports option to "please generate a module of this shape". For example we could rename available_imports to desired_shape and then wasm-smith would guarantee that it would generate a module with at most the imports provided and exactly the exports provided.

I like this a lot 👍

[adambratschikaye]

Thanks @alexcrichton, yeah that would be sad to ruin the serializability of Config. The desired_shape idea would allow us to generate only IC compatible modules, but it would also be overly restrictive. For example, we allow a module to export many functions with names matching the patterns canister_update * or canister_query * as long as they have type () -> (). So I guess the serialization format would need a mini DSL or regex support to fully describe this.

Of course we could just pick a bunch of valid names and put them in the desired_shape, but then there'll be some code paths that fuzzing doesn't check.

[adambratschikaye]

We could allow both by having a variant that's either a static list of allowed exports or a callback, and we panic when trying to serialize the callback variant. Then users that care more about the serialization can use the static list, but you still have the full power if it's required. Maybe that's a bit over-engineered though.

[alexcrichton]

True yeah, and to be clear don't feel the need to take this on if you don't want. The serializability property of Config isn't a hard guarantee and is fine to have updated over time.

My thinking though is that for canister_{update,query} * your fuzzer would generate the "shape" of the module randomly. You'd always have the same set of imports and would then generate exports given the fuzz input. Then given that shape the rest of the fuzz input would be handed off to wasm-smith to generate the full module. So not a drop-in replacement for mutating exports in place and would require more integration work, but would that shape of solution perhaps work for your use case?

[adambratschikaye]

Ok, that makes sense, but you're saying the "shape" would still be part of the config, right? So we would generate a new random config for each fuzz input?

[fitzgen]

The serializability property of Config isn't a hard guarantee and is fine to have updated over time.

Small clarification: we do require that the config implement Serialize and Deserialize as that is part of the CLI, but it is okay if, for now, this new feature doesn't round trip through serialization and deserialization.

[alexcrichton]

So we would generate a new random config for each fuzz input?

Indeed! I'd recommend doing this for otherwise orthogonal reasons as that ends up implementing swarm testing which can help improve coverage (if I'm understanding that right). Otherwise though, yes, with each fuzz input you'd generate a new Config with a new "shape".

Small clarification: we do require that the config implement Serialize and Deserialize as that is part of the CLI

Oh oops, I forgot about that!

[adambratschikaye]

I ended up making a separate field exports instead of combining it with available_imports into desired_shape. Just because if we combine them then we loose the ability to make each one separately optional (if you want to specify the imports, but allow arbitrary exports or vice versa).