atul9 / mutagen

Breaking your Rust code for fun and profit

Breaking your Rust code for fun & profit

This is a work in progress mutation testing framework. Not all components are there, those that are there aren't finished, but it's already somewhat usable as of now.

Mutation Testing

The idea behind mutation testing is to insert changes into your code to see if they make your tests fail. If not, your tests obviously fail to test the changed code. The difference to line or branch coverage is that those measure if the code under test was executed, but that says nothing about whether the tests would have caught any error.

This repo has three components at the moment: The mutagen test runner, a helper library and a procedural macro that mutates your code.

How mutagen Works

Mutagen works as a procedural macro. This means two things:

1. You'll need a nightly rust toolchain to compile the plugin.
2. it only gets to see the code you mark up with the #[mutate] annotation, nothing more.

It also will only see the bare AST, no inferred types, no control flow or data flow, unless we analyse them ourselves. But not only that, we want to be fast. This means we want to avoid doing one compile run per mutation, so we try to bake in all mutations into the code once and select them at runtime via a mutation count. This means we must avoid mutations that break the code so it no longer compiles.

This project is basically an experiment to see what mutations we can still apply under those constraints.

A Word of Warning

mutagen will change the code you annotate with the #[mutate] attribute. As long as you use it with safe code, all is well. However, running mutagen against unsafe code will very probably break its invariants, with possible dire consequences. So don't run mutagen against modules containing unsafe code under any circumstances.

Using mutagen

Again, remember you need a nightly rustc to compile the plugin. Add the plugin and helper library as a dev-dependency to your Cargo.toml:

[dev-dependencies]
mutagen = "0.1.2"
mutagen-plugin = "0.1.2"

Now, you can add the plugin to your crate by prepending the following:

#![cfg_attr(test, feature(plugin))]
#![cfg_attr(test, plugin(mutagen_plugin))]
#![feature(custom_attribute)]

#[cfg(test)]
extern crate mutagen;

Now you can advise mutagen to mutate any function, method, impl, trait impl or whole module (but not the whole crate, this is a restriction of procedural macros for now) by prepending:

#[cfg_attr(test, mutate)]

This ensures the mutation will only be active in test mode.

Running mutagen

Install cargo-mutagen. Run cargo mutagen on the project under test.

If you want the development version, run cargo install in the runner dir.

If you want to do this manually you can run cargo test as always, which will mutate your code and write a list of mutations in target/mutagen/mutations.txt. For every mutation, counting from one, you can run the test binary with the environment variable MUTATION_COUNT=1 target/debug/myproj-123456, MUTATION_COUNT=2 .., etc.

You can run cargo mutagen -- --coverage in order to reduce the time it takes to run the mutated code. When running on this mode, it runs the testsuite at the beginning of the process and checks which tests are hitting mutated code. Then, for each mutation, instead of running the whole testsuite again, it executes only the tests that are affected by the current mutation. This mode is specially useful when the testsuite is slow or when the mutated code affects a little part of it.

Contributing

Issues and PRs welcome! See CONTRIBUTING.md on how to help.