Error backtraces are discarded when converting to color_eyre::Context about color-eyre

[yaahc]

It is impossible to preserve the inner backtrace from the error being wrapped. The backtrace type used in this library is from backtrace-rs and is incompatible with the one available on nightly via the backtrace fn on the error trait.

I have an open RFC to fix the error Trait so we can do that but until then or until std stabilizes Backtrace I can't do this.

[yaahc]

Actually I might be able to do this with some cfg shenanigans and an internal enum type to specify which backtrace kind we're using. As long as it is entirely hidden from the API I can do all the cfg in match statements for nightly vs stable and it won't case breaking changes when you use different compiler channels. However we would need to patch color-backtrace to add support for std Backtrace before we can even start using it.

[nightkr]

I thought it would be possible to make EyreContext generic over the dynamic Error type, but that seems like a horrible rabbit hole. I don't think it's worth messing even more with eyre::error's vtable shenanigans.. :/

[yaahc]

Yea, and I don't even think we could do a good job with the vtable. We might be able to do some shenanigans using autoref specialization inside of the eyre macro where we add an extension trait on Error that people can implement for their errors that lets them return a reference to their inner Backtrace, but they'd have to implement that on each error type that they want to be able to have provide the backtrace.

The problem with this though is that this won't be work with any error converted via From because we cannot specialize that impl, so it will only be able to act on the Error trait itself which is still too restrictive. And beyond that, I'm not even sure how such an extension trait would interact with wrapping errors internally, overall I think its the wrong route.

I did have another plan, I was going to add a generic version of TracedError that let you specify what context to bundle with the inner Error, we could add support for that, which would allow extracting Backtrace or even HelpInfo out of leaf errors.

I'm planning on writing this extract thing anyways, so I'll probably end up doing that today, once its done I'll show you what I mean and how it would interact with color_eyre and you can lmk if you'd end up using it or not.

[nightkr]

Oh, that sounds like an interesting idea. To explain where I'm coming from a bit better.. I generally tend to prefer SNAFU for my error types (I guess I'm just a masochist.. :P), but wanted to use color_eyre for the final error rendering.. something like https://gitlab.com/teozkr/kube-rt/-/blob/45d66b631add448cc2e039f4b7e68152bcdef87e/examples/configmapgen_controller.rs

For that use-case I'm 100% fine with needing an extra impl TracedError (or whatever the API ends up being), but it's also pretty much the worst possible case for the current approach, which seems to assume that the boundary between eyre::Report and std::error::Error is also the boundary between application and libraries.

[nightkr]

kube_rt also has an issue where it currently doesn't support anyhow/eyre errors from the reconciler (because it requires the error type to impl std::error::Error), but I think that should be fixable separately.

[yaahc]

but it's also pretty much the worst possible case for the current approach, which seems to assume that the boundary between eyre::Report and std::error::Error is also the boundary between application and libraries.

To be fair this is exactly what the std::error::Error trait is meant for, to represent an open set of errors and provide interfaces to report them and downcast them, so I think your approach is entirely valid.

kube_rt also has an issue where it currently doesn't support anyhow/eyre errors from the reconciler (because it requires the error type to impl std::error::Error), but I think that should be fixable separately.

I think there are ways to deal with this by leveraging Deref

[nightkr]

I think there are ways to deal with this by leveraging Deref

Yeah, or SNAFU's AsErrorSource. Then again, that still means that we lose the backtrace on the other end.. :/

[yaahc]

Then again, that still means that we lose the backtrace on the other end.. :/

Does kube_rt handle printing the error for you? Because if you have a chance to turn it back into a Report before printing it should still be able to print the innermost Backtrace via the extract type, so long as the source chain is intact.

[nightkr]

Sorry, I was unclear. kube_rt::controller itself doesn't print any errors, but it wraps user errors to collect them with a bunch of system-generated errors: https://gitlab.com/teozkr/kube-rt/-/blob/45d66b631add448cc2e039f4b7e68152bcdef87e/src/controller.rs#L15-33, which it's then up to the user application to print: https://gitlab.com/teozkr/kube-rt/-/blob/45d66b631add448cc2e039f4b7e68152bcdef87e/examples/configmapgen_controller.rs#L139.

Normally this isn't a problem, because the errors always flow from library -> application (or another calling library), but kube_rt::controller needs to see them in order to help you set up correct retry logic (https://gitlab.com/teozkr/kube-rt/-/blob/45d66b631add448cc2e039f4b7e68152bcdef87e/examples/configmapgen_controller.rs#L121-123). This means that, if the application uses eyre (or anyhow, or some other library that relies on dyn Error), the final error structure becomes eyre::Report(kube_rt::controller::Error::ReconcilerFailed(eyre::Report(...))). SNAFU doesn't understand Eyre's backtraces, nor vice versa, so the correct backtrace falls away at both conversion points.

[nightkr]

On nightly this mostly falls away, since SNAFU and eyre can both use std::backtrace::Backtrace and std::error::Error::backtrace() if asked to, but "just use nightly!" doesn't feel like a great message to send to users either. And as mentioned in the OP, that works for eyre::DefaultContext but not for eyre_color::Context.

[yaahc]

eyre::Report(kube_rt::controller::Error::ReconcilerFailed(eyre::Report(...)))

Aah, yes, with this setup if the user used the extracter type I mentioned both Reports here would be able to skip their Backtrace capture and correclty use the user's backtrace, but if they dont do that both would capture the Backtrace, and I don't think I can apply the same extracter magic to the reporters themselves because they aren't actually errors in the chain.

I'm interested in learning more about how kube_rt is doing its retry logic, I feel like there might be ways to handle this without introducing the kube_rt::controller::Error type but I definitely need to understand whats going on better before I start giving unsolicited advice. Lmk if you're interested in bikeshedding alternative approaches there.

[nightkr]

Lmk if you're interested in bikeshedding alternative approaches there.

Absolutely. kube_rt is already an exercise in bikeshedding kube::runtime.. :P

I'm interested in learning more about how kube_rt is doing its retry logic, I feel like there might be ways to handle this without introducing the kube_rt::controller::Error type but I definitely need to understand whats going on better before I start giving unsolicited advice.

Not sure how familiar you are with the inner workings of Kubernetes. I didn't see anything related in your profile, but that is of course not everything. I'll try to explain the thought process that lead to the current design of kube_rt, but it might be helpful to have a basic understanding of Kubernetes' design principles. In particular, kube_rt is pretty inspired by their (Go) controller-runtime.

So basically, the role of the controller(-runtime) is to take a stream of external triggers (usually events from the Kubernetes cluster) and run a user-supplied "reconciler" function that attempts to make the external world conform to the configuration stored in the Kube object.

The problem is that reconciliation this can fail for umpteen reasons, most of which are temporary (cluster is down, external service is down, our cache is stale, waiting for some other controller, yadda yadda). So we need to be able to tell the controller that it should try to reconcile our object again later, which we can do using the ReconcilerAction type.

But it's not really ergonomic to have to add the same abort-and-request-retry logic to every failure location, especially when we already have the wonderful ? for this use case! So let's let the user return the Errors instead, and define a retry policy based on the returned errors.

That way, we also get a nice structured log of errors (from the controller, triggers, and reconciler), that the user can define how to handle (quit on first error, keep going, log to some external server, etc), without imposing any policy or logging library on them by decree.

[nightkr]

I guess one possible other approach would be to let the user control the ultimate error type, and just demand a From<kube::controller::Error> instance. I'm not a huge fan of using From/Into for error handling, but I suppose it could be fairly unambiguous in this case.

[yaahc]

Not sure how familiar you are with the inner workings of Kubernetes.

I have almost 0 familiarity with it, so this explanation is appreciated. I think I'm gonna check out the examples in kube_rt to get a better idea of what you mean and how it all fits together.

[nightkr]

Sure. The documentation is still a bit lacking since it's still in semi-prototype mode, but feel free to ask if (when?) anything is unclear.

[yaahc]

Okay, so heres my understanding of how this works.

Users provide a bunch of closures for the controller fn to provide retry rules and other things, and the various user errors that can be emitted from each part of the provided functions in the controller are then funnelled back to the user via controller::Error which primarily exists to indicate what part of the system the error came from, for example a failed retry (reconcile?).

First thing I notice is I don't think eyre::Report(kube_rt::controller::Error::ReconcilerFailed(eyre::Report(...))) is possible in practice, because eyre::Report doesn't satisfy the std:error::Error bound on controller::Error. There would have to be a newtype around the Report to forward to the inner error thru deref or as_ref.

I also get now that the retry logic isn't being specified by the error type, its being specified in the closure.

Generally as a rule of thumb I don't think people should be converting errors to Reports when they still might need to handle those errors rather than reporting them. Thats a big part of why I renamed the type rather than calling it Error like anyhow. I say this because I think its okay and maybe even good if you don't design your API with the intention of nicely supporting users who are returning error reporting types like anyhow::Error or eyre::Report. As far as I can tell so long as the users stick inside of snafu / std::error::Error land their error types should interoperate nicely with your controller::Error type which can then play nicely with color_eyre::Report.

I've mostly finished the extract stuff, so I'll push that shortly and show you how that will interoperate with color-eyre. I can show you how to update the controller::Error type to use this to capture the backtrace instead of snafu, though I'm not really sure how useful it is to capture a Backtrace there when its so high up in the stack, its basically just gonna tell you the exact same info that you already get from checking the current controller::Error variant. I think it might be better to just focus on adding nightly support for std::backtrace::Backtrace which will hopefully make all the error types work together seemlessly.

[nightkr]

Users provide a bunch of closures for the controller fn to provide retry rules and other things, and the various user errors that can be emitted from each part of the provided functions in the controller are then funnelled back to the user via controller::Error which primarily exists to indicate what part of the system the error came from, for example a failed retry (reconcile?).

Pretty much.

First thing I notice is I don't think eyre::Report(kube_rt::controller::Error::ReconcilerFailed(eyre::Report(...))) is possible in practice, because eyre::Report doesn't satisfy the std:error::Error bound on controller::Error. There would have to be a newtype around the Report to forward to the inner error thru deref or as_ref.

Correct. It was more of a hypothetical for if I got rid of the : std::error::Error bound (which doesn't seem to be as simple as I thought).

As far as I can tell so long as the users stick inside of snafu / std::error::Error land their error types should interoperate nicely with your controller::Error type which can then play nicely with color_eyre::Report.

Yeah, this is the current state of things (apart from losing the backtrace when converting from controller::Error -> color_eyre::Report).

Generally as a rule of thumb I don't think people should be converting errors to Reports when they still might need to handle those errors rather than reporting them. Thats a big part of why I renamed the type rather than calling it Error like anyhow. I say this because I think its okay and maybe even good if you don't design your API with the intention of nicely supporting users who are returning error reporting types like anyhow::Error or eyre::Report.

Definitely agreed when it comes to "is this a good idea to encourage". I'm just not sure I want to enforce having that overhead if you just want a blanket policy.

I've mostly finished the extract stuff, so I'll push that shortly and show you how that will interoperate with color-eyre. I can show you how to update the controller::Error type to use this to capture the backtrace instead of snafu, though I'm not really sure how useful it is to capture a Backtrace there when its so high up in the stack, its basically just gonna tell you the exact same info that you already get from checking the current controller::Error variant. I think it might be better to just focus on adding nightly support for std::backtrace::Backtrace which will hopefully make all the error types work together seemlessly.

Not sure I've understood you correctly here, but I think it might make more sense once I've had a go with the new API.

[yaahc]

Not sure I've understood you correctly here, but I think it might make more sense once I've had a go with the new API.

Fair enough. I've not finished updating color-eyre and friends to use it yet but I finished and published the stuff I was talking about here https://github.com/yaahc/extracterr

Should be pretty well documented, though I doubt docs.rs has finished compiling the docs so you might have to pull it to look at the docs. I'll update color-eyre to add an example showing it using a Backtrace from a leaf error via extracterr instead of capturing its own, hopefully within the next couple hours.

Once that's all together I think we should look into adding a snafu feature that optionally adds snafu as a dependency to allow it to try to extract snafu::Backtrace types specifically. It still wont work well with snafu's built in sugar for capturing errors via the context fn, but I think that with an error kind setup its just as ergonomic and it should work with snafu all the same.

[nightkr]

Hm, so the idea is that kube_rt wraps all errors internally as Bundled<RealError, Backtrace>, which color_eyre would pick up on when generating the Report?

[nightkr]

Personally I'm leaning more towards just adding a bound that ReconcilerErr: snafu::ErrorCompat, which is pretty trivial to implement for custom error types regardless of whether you want to care about backtraces. But for the rendering, that would require being able to create a Report with a custom backtrace...

[yaahc]

if its possible to extract snafu backtraces and convert them to std::backtrace::Backtrace you could add a compat shim right before you convert to a Report that looks to see if the chain of snafu errors has a std::backtrace, and if it does it copy it out and then wrap the entire chain in a Bundled so the report can access the backtrace.

[nightkr]

Yeah I was looking into that.

if its possible to extract snafu backtraces and convert them to std::backtrace::Backtrace

snafu::Backtrace can be configured using Cargo features to be an alias for std::backtrace::Backtrace or backtrace::Backtrace. Since this conversion is an application-level responsibility, that should be fine.

you could add a compat shim right before you convert to a Report that looks to see if the chain of snafu errors has a std::backtrace, and if it does it copy it out and then wrap the entire chain in a Bundled so the report can access the backtrace.

Almost.. :/ That'd require a way to bundle an Option<Backtrace>, since the bundling changes the error type.

[yaahc]

as long as you convert the error to a Report in both branches that attempt to Bundle the backtrace it should work fine and not matter if you change the error type or not.

It sounds like the ideal situation would be to have it attempt to bundle a backtrace::Backtrace unless fn backtrace is available via the Error trait in which case you'd just let the error trait handle everything. I can copy the same build.rs magic that anyhow/eyre use to conditionally enable the std backtrace logic and use that to enable / disable the extracterr logic and how it tries to access backtraces. Then you'd just need to do the same thing on your side and users can just rely on snafu to do the swapping for them.

[nightkr]

Okay, I got it to bundle the backtrace::Backtrace (https://gitlab.com/teozkr/kube-rt/-/merge_requests/1), but color_eyre::Report::from doesn't seem to be picking it up (when I run it with RUST_BACKTRACE=full the backtrace is still from where I convert the SNAFU error into a Report). I guess this part is not implemented yet?

[yaahc]

yea, I haven't finished that part yet. I plan on setting aside tomorrow to catch up on this and some other PRs, I'll let you know as soon as it's ready