Errors not printed on Raspberry Pi when backtrace enabled about color-eyre

[yaahc]

This looks like an explicit panic in backtrace-rs

What happens if you depend on backtrace directly and manually capture and print the backtrace on the pi?

[duncanrhamill]

Ok so with using backtrace explicitly and no dependence on eyre or color-eyre doesn't seem like there's anything wrong.

[yaahc]

Okay, it might take me a little while to fix this one because I have to find and setup a raspberry pi to do it.

Any additional testing you could do to help isolate the issue would be helpful, such as the specific code you used to show the problem doesn't happen without using color-eyre, or any gdb output trying to trace down exactly what it was doing when it panicked.

And fwiw, I'm 99% sure this isn't an issue from color-eyre, its probably a bug in backtrace-rs where it makes assumptions based on x86 arch that didn't hold true when you ran it on arm, but I'm happy to help you figure out the issue and fix it upstream.

[yaahc]

Oh, @duncanrhamill, did you run this on stable? Can you try reproing the backtrace on nightly? It might give more relevant panic location info

[duncanrhamill]

Ok, thank you very much! I'm thinking the same thing with arch changes, maybe something along the lines of this?

I've rerun everything on nightly and I've put together the outputs from running on the pi here, including the source used to generate them.

If you want me to setup remote ssh on the pi I can.

[yaahc]

This is awesome, ty

As for the ssh, we found a pi but I still have to set it up on the network and do the ip addr discovery, but that shouldn't be hard so I don't think that will be necessary, but if I run into trouble getting a repro I'll let you know so you can setup ssh.

[duncanrhamill]

Ahh great, let me know if you need anything else!

[yaahc]

I'm sorry I haven't gotten around to this yet, but I haven't forgotten.

One thing I realized is that it might be the specific backtrace features that color-backtrace sets. Specifically this:

[features]
default = ["gimli-symbolize"]
failure-bt = ["failure"]
gimli-symbolize = ["backtrace/gimli-symbolize"]

Can you try the example again with gimli-symbolize enabled in backtrace and see what happens?

[duncanrhamill]

No worries, just reran it with

backtrace = { version="0.3", features=["gimli-symbolize"] }

and it seems like that's the cause, panicked before printing the backtrace. Full backtrace from this one:

thread 'main' panicked at 'assertion failed: (self.addr as usize) < (isize::MAX as usize)', /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/macros.rs:13:23
stack backtrace:
   0:   0x595170 - backtrace::backtrace::libunwind::trace::h19f0e9723381df32
                       at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1:   0x595170 - backtrace::backtrace::trace_unsynchronized::h4689672582c6f73d
                       at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2:   0x595170 - std::sys_common::backtrace::_print_fmt::h555e818d074d456c
                       at src/libstd/sys_common/backtrace.rs:78
   3:   0x595170 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hda0f74c2888e4e45
                       at src/libstd/sys_common/backtrace.rs:59
   4:   0x5ad140 - core::fmt::write::hd5deabd920c4e4ab
                       at src/libcore/fmt/mod.rs:1069
   5:   0x593074 - std::io::Write::write_fmt::h6d8723e6380c1d4d
                       at src/libstd/io/mod.rs:1537
   6:   0x5976d8 - std::sys_common::backtrace::_print::h1a240c08a14b3ea3
                       at src/libstd/sys_common/backtrace.rs:62
   7:   0x5976d8 - std::sys_common::backtrace::print::he8a632cb741f8cfd
                       at src/libstd/sys_common/backtrace.rs:49
   8:   0x5976d8 - std::panicking::default_hook::{{closure}}::ha92d8871ef11f458
                       at src/libstd/panicking.rs:198
   9:   0x597398 - std::panicking::default_hook::hbe3dd080d55d3d81
                       at src/libstd/panicking.rs:218
  10:   0x597db4 - std::panicking::rust_panic_with_hook::h8566666fb90d07b7
                       at src/libstd/panicking.rs:477
  11:   0x565c0c - std::panicking::begin_panic::ha4fc229687bffc16
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/panicking.rs:404
  12:   0x4dd1cc - <findshlibs::linux::SharedLibrary as findshlibs::SharedLibrary>::virtual_memory_bias::h8aeff40c4e8ca5f2
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/macros.rs:13
  13:   0x4c7b44 - backtrace::symbolize::gimli::Cache::new::{{closure}}::hca979ae6354ba91a
                       at /home/pi/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/symbolize/gimli.rs:427
  14:   0x4d9334 - findshlibs::linux::SharedLibrary::callback::{{closure}}::h9850d3a8fe0dbadd
                       at /home/pi/.cargo/registry/src/github.com-1ecc6299db9ec823/findshlibs-0.5.0/src/linux/mod.rs:185
  15:   0x4cbc6c - core::ops::function::FnOnce::call_once::h5c77f59632c53541
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libcore/ops/function.rs:232
  16:   0x4c5078 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h085e56f9f200b22c
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/panic.rs:318
  17:   0x502384 - std::panicking::try::do_call::hbd7d397f01034b36
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/panicking.rs:297
  18:   0x506f18 - __rust_try
  19:   0x5022ac - std::panicking::try::h9ce3104d3376fb8d
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/panicking.rs:274
  20:   0x4c5314 - std::panic::catch_unwind::h21f9bfa8d822b5b1
                       at /rustc/9912925c254589f58338cb2993163e618475ff75/src/libstd/panic.rs:394
  21:   0x4d91ac - findshlibs::linux::SharedLibrary::callback::he57ce783a7ea4f2e
                       at /home/pi/.cargo/registry/src/github.com-1ecc6299db9ec823/findshlibs-0.5.0/src/linux/mod.rs:181

[yaahc]

Perfect, I've opened an issue on backtrace-rs because I'm gonna low key attempt to pass the buck but if they end up too busy to fix it I will try to figure it out and fix it upstream.

[yaahc]

Looks like its already fixed upstream, so we will bump our dependency versions in color-eyre and color-backtrace to make it use the newest version.

Can you verify that this issue doesn't reproduce when you set the backtrace version to 0.3.48?

[yaahc]

@duncanrhamill Once you've verified that 0.3.48 fixes the issue I will merge #10

[duncanrhamill]

Yep, 0.3.48 works with the backtrace only version and gimli enabled.

Thanks for your effort on this, really appreciated!

[yaahc]

my pleasure!