1000x performance regression in development builds about color-eyre

[yaahc]

Edit: I think i figured out the issue, the explanation is in the last comment, so you can skip straight to that and try out the suggested fix before reading this.

wow... 3-5 seconds, that's pretty unexpected...

The main perf implication should happen around formatting the errors. Theres some dynamic allocation involved when wrapping errors but that shouldn't have perf differences between release and debug afaik. Also this is unchanged relative to eyre. The only non-formatting differences between this and eyre by default is that this will capture a backtrace and a spantrace, and it does backtrace capture on stable.

There is the new global backtrace printer which is stored in a OnceCell. But I wouldn't expect that to cause major perf regressions, it should be a set once thing with no locking needed to access it, so even if you're formatting a ton of reports simultaneously they shouldn't step on each other's toes.

Other than that the main performance considerations just printing errors to the screen. color-backtrace requires an extra allocation due to the limited interface provided by termcolor. But color-eyre and color-spantrace both use ansi_term which lets them format text in a std::fmt::Write compatible way, so they should just be writing straight into the same output buffer, and I wouldn't expect there to be huge perf differences in them depending on if its debug or release.

My best bet is it's something about how tracing behaves, assuming you're using tracing. Storing the SpanTrace in the error might be changing the drop behavior of the subscriber? I'm not sure how it all works tbh. It should be easy to test, the tracing integration already has a feature flag to disable it, you can try this immediately to see if it has any difference while I go add instrumentation and some tests to profile it in debug / release.

[dependencies]
eyre = "0.4"
color-eyre = { version = "0.3", default-features = false }

cc @hawkw

[yaahc]

@thurn is there a github repo somewhere that is showing this perf regression that I can use to tease out the cause, or alternatively, can you setup a minimal test case that reproduces the issue? I'm trying to figure out whats wrong in #20 but the best I've been able to come up with is that it takes .3 seconds to capture a backtrace, which isn't really avoidable and shouldn't be different from eyre when used on nightly with backtrace capture enabled.

[yaahc]

Actually, I checked it out and it looks like the performance of std::backtrace::Backtrace is independent of what build type you use, so capturing a Backtrace on nightly debug will always be fast, compared to backtrace::Backtrace capture which is compiled as a crate with the debug level you select. So capturing a backtrace is much slower in color-eyre than in eyre.

I think there are ways to configure cargo to build deps with optimization enabled, I'll figure out the invocation for you. I'm not sure they can be applied in color-eyre or if they have to be added in application crates to work. If its the former I'll just make the change in configuration and leave it be, if its the latter I'll probably end up adding a section to the readme about setting up the configuration needed to get fast debug Backtraces.

---

The specific difference is that it seems to take 300 ms to capture a backtrace::Backtrace in debug mode and 15 ms to capture a std::backtrace::Backtrace in debug/release mode. This doesn't quite account for the 1000x difference but maybe if the stack is much larger the diff reaches that 1k, my example programs definitely aren't stress testing the backtrace machinery.

[yaahc]

Okay so I set this in my Cargo.toml

[profile.dev.package.backtrace]
opt-level = 3

and it manged to bring down the time for creating and printing a report with backtraces enabled in a debug release from 300ms to 24ms. The documentation for profiles in the cargo book implies that this setting wont be included in reverse dependencies. You can double check this by patching your dep to use the debug-perf branch on github and see if that improves your perf, in all likelyhood it wont, but if you add the above to your crate it may solve your issue. Let me know how it goes!

color-eyre on  debug-perf [$!] is 📦 v0.3.2 via 🦀 v1.43.1
❯ RUST_BACKTRACE=1 cargo run --example debug_perf
   Compiling backtrace v0.3.48
   Compiling color-backtrace v0.4.0
   Compiling color-eyre v0.3.2 (/home/jlusby/git/rust/color-eyre)
    Finished dev [unoptimized + debuginfo] target(s) in 4.72s
     Running `target/debug/examples/debug_perf`
Error:
   0: wrapped error
   1: fake error

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: debug_perf::time_report_inner
      at examples/debug_perf.rs:21
   1: debug_perf::time_report
      at examples/debug_perf.rs:16

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 3 frames hidden ⋮
   4: debug_perf::time_report_inner::haa14648d105ef613
      at /home/jlusby/git/rust/color-eyre/examples/debug_perf.rs:24
   5: debug_perf::time_report::hfbcea083c195f359
      at /home/jlusby/git/rust/color-eyre/examples/debug_perf.rs:18
   6: debug_perf::main::h739a052aa690a567
      at /home/jlusby/git/rust/color-eyre/examples/debug_perf.rs:11
                                ⋮ 11 frames hidden ⋮

Suggestion: try using a file that exists next time
[examples/debug_perf.rs:33] end - start = 24.748325ms

color-eyre on  debug-perf [$!] is 📦 v0.3.2 via 🦀 v1.43.1 took 4s
❯ cargo run --example debug_perf
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/examples/debug_perf`
Error:
   0: wrapped error
   1: fake error

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

   0: debug_perf::time_report_inner
      at examples/debug_perf.rs:21
   1: debug_perf::time_report
      at examples/debug_perf.rs:16

Suggestion: try using a file that exists next time
[examples/debug_perf.rs:33] end - start = 181.804µs

[thurn]

I can try and get an example together later this week, it does seem like creating a Backtrace is the expensive part.

I'm somewhat surprised that this cost exists in the non-Error case, I'd sort of naively expected there would be almost no overhead if you're just returning Oks everywhere?

[yaahc]

It definitely shouldn't exist in the error case and if you're seeing perf regressions without encountering errors something extra weird is going on. You might be using constructors that are being eagerly evaluated, making them immediately drop reports, like if you use .ok_or(eyre!(..)) instead of .ok_or_else(|| eyre!(..)). Clippy has some lint's for detecting code like this, or you can use a drop bomb to detect it.

[thurn]

Oh right, I was like "or_or_else is too much typing" :p. Sorry, just started learning Rust for this project.

[yaahc]

Aah, absolutely no problem. Highly recommend integrating clippy in your workflow, it's great for learning about these little sharp edges in the language. I'm gonna go ahead and close this issue since it seems like we figured it out! Ty for the bug report, learning about the perf cost differences between backtrace::Backtrace and std::backtrace::Backtrace was really fun.