brson / cargo-crusader

Test the downstream impact of Rust crate changes before publishing

Cargo Crusader

Hark, Rust crate author! The battle for Rust's reputation as The Most Reliable Software Platform Ever is here, and nobody is free of responsibility. The future of Rust, dear Rustilian, is in your hands.

Join the Cargo Crusade and bring the Theory of Responsible API Evolution to the non-believers.

Cargo Crusader is a tool to help crate authors evaluate the impact of future API changes on downstream users of that crate before they are published to crates.io.

How?

When you run cargo-crusader from the source directory of your published crate, Crusader asks crates.io for all of its reverse dependencies - published crates that DEPEND ON YOU. It then downloads each of them, and builds them: first against your crate as currently published, then against your local work-in-progress (i.e. the next version you are going to publish). It then reports differences in behavior.

Getting Started

IMPORTANT SECURITY WARNING: This program executes arbitrary untrusted code downloaded from the Internet. You are strongly recommended to take your own sandboxing precautions before running it.

First, download and build Cargo Crusader, and put the cargo-crusader command in your PATH environment variable:

$ git clone https://github.com/brson/cargo-crusader
$ cd cargo-crusader
$ cargo build --release
$ export PATH=$PATH:`pwd`/target/release/

Now change directories to your source and run cargo-crusader:

$ cargo-crusader
crusader: downloading reverse deps for hyper
crusader: 10 reverse deps
crusader: testing crate aloft
crusader: testing crate austenite
crusader: result 1 of 10, aloft 0.3.1: broken
crusader: testing crate bare
crusader: result 2 of 10, austenite 0.0.1: broken
crusader: testing crate catapult
crusader: result 3 of 10, bare 0.0.1: broken
crusader: testing crate chan
crusader: result 4 of 10, catapult 0.1.2: broken
crusader: testing crate chatbot
crusader: result 5 of 10, chan 0.1.14: passed
crusader: testing crate click_and_load
crusader: result 6 of 10, chatbot 0.2.2: regressed
crusader: testing crate coinbaser
crusader: result 7 of 10, click_and_load 0.0.1: broken
crusader: testing crate doapi
crusader: result 8 of 10, coinbaser 0.1.0: regressed
crusader: testing crate ease
crusader: result 9 of 10, doapi 0.1.0: broken
crusader: result 10 of 10, ease 0.2.1: regressed

passed: 1
regressed: 3
broken: 6
error: 0

full report: ./crusader-report.html

A full run will take quite a while. After its done it will print a summary, as well as produce an HTML file containing the full results, including all the compiler output for each test.

Tests result in four possible statuses: 'passed', if the reverse dependency built both before and after the upgrade; 'regressed', if it built before but not after; 'broken', if it didn't even build before upgrading; and 'error', for internal Crusader errors.

Future improvements

Presently Crusader will override reverse dependencies with your local revision even if the version they requested is not semver compatible with your work-in-progress. Crusader might first verify whether or not the WIP qualifies as a semver-valid upgrade.

Testing upstream as well - Crusader could ask for all the WIP branches of your dependencies and then override your build to see if upcoming changes are going to break your crate.

Sandboxing.

License

MIT/Apache-2.0 is the official license of both The Rust Project and The Cargo Crusade.

cargo-crusader's Issues

Build failure on nightly.

See here. I think it should be fixed by bumping the rustc-serialize requirement up to 0.3.16.

Support --help

Cargo crusader should not start working on unknown command line arguments.

It would be cool if both these were supported: cargo-crusader --help and cargo help crusader (Yep! Cargo picks up custom commands, just like git and others).

Crusader fails

Not only it fails, but also it uses outdated crates that do not compile with recent openssl versions on my system.

Example of failure https://travis-ci.org/contain-rs/bit-vec/jobs/533429609

crusader: downloading reverse deps for bit-vec
https://crates.io/api/v1/crates/bit-vec/reverse_dependencies?per_page=100&page=1
error: decoder error
ParseError(SyntaxError("invalid syntax", 1, 1))
The command "bash crusader.sh" exited with 255.

Recommend or suggest some sandboxing precautions

I'm grateful of the security warning in the README:

IMPORTANT SECURITY WARNING: This program executes arbitrary
untrusted code downloaded from the Internet. You are strongly
recommended to take your own sandboxing precautions before running
it.

Are there some entry-level sandbox precautions that could be suggested here?

I'm wondering if perhaps it could be something along the lines of "As a minimum download Docker for Desktop, use [this] Dockerfile and launch it [this way] to get a reasonably locked and isolated environment".

(As a side note, thanks for writing this tool - it looks really useful!)

Test more than 10 reverse deps

@DanielKeep discovered that crates.io's revdep API is paginated and only returns 10 results at a time.