Consider changing middleware errors to Box<dyn Error> about tower

[hntd187]

I'm in favor of this, I experienced a lot of pain composing tower middlewares and I can only imagine this getting out of hand if I wanted to add more into the mix.
https://github.com/toshi-search/Toshi/blob/manual_clustering/src/cluster/mod.rs#L87 was what I ended up writing so that I could just simply map_err my errors all into some general type. From someone integrating tower, but having written none of the code it's a pretty sizable pain point to have to deal with all these disparate error types that aren't compatible.

Another and probably less ideal option (and I'm not even sure if this is possible in rust) but have all the errors in the middlewares have a specific general marker trait so you can use that more generic type in return values, but that would end up placing a lot of boilerplate probably in the various tower services.

[seanmonstar]

cc @carllerche @hawkw @olix0r @jonhoo

[davidbarsky]

I'd really like to use Tower in aws-lambda-rust-runtime, and this would make our (expected) nested middleware stack much nicer.

[carllerche]

@davidbarsky could you provide more specifics about how it would make it nicer?

[hawkw]

I'd prefer to avoid returning Box<dyn Error> if at all possible. The main reason is because I've seen cases (such as in Linkerd) where traits other than Error are implemented for error types, and returning boxed trait objects would erase those additional traits. There are also the additional downsides that @seanmonstar mentioned above to consider, but the erasure of non-Error traits is my primary concern.

This doesn't resolve all the same issues as returning boxed trait objects, but one potential solution to the problem of unreadable debug output for errors is to just use the fmt::Display implementations instead. I believe in most cases, Tower's error types implement fmt::Display by proxying to the inner error, when they wrap another error. Using fmt::Display to log these errors would result in only the message of the original error type being logged, rather than a complex nested type.

[seanmonstar]

one potential solution to the problem of unreadable debug output for errors is to just use the fmt::Display implementations instead.

I didn't mean to say this was to fix debug output, I simply was showing what the type structurally looked like inside that function, and what match I would need to write to check for only certain variants.

For example, I'd need to write:

fn can_retry(err: &router::Error<buffer::Error<balance::Error<Either<Etc, MoreEtc>>>>) {
    match err {
        route::Error::Inner(buffer::Error::Inner(balance::Error::Inner(Either::A(reconnect::Error::Inner(Etc))))) => {}
    }
}

[davidbarsky]

@carllerche

Sure. I expect that we'd have a nested Tower middleware stack (starting with tower-retry, tokio-trace-tower, tower-balance) before any request even reaches user-level code. As Sean noted in his proposal, this would reduce the fragility of the middleware stack. Additionally, I suspect that a large chunk of errors deep within a middleware stack can't reasonably be handled by other middleware.

@seanmonstar

Perhaps I'm missing something—would this prevent returning something like the Never type? Additionally—and this is probably out of scope for this change—but are there plans to change tower's associated types to a futures 0.3-style that return a single Output type?

[seanmonstar]

It wouldn't prevent returning Never/!, since they can implement std::error::Error, but it would reduce the optimizations that could be performed because of it being uninhabited would be lost to the type system at higher levels.

FWIW, I'm not convinced this proposal is the best, just thought I'd put together my thoughts on my current experience trying to determine what protocol and Middleware errors could be safely retried.

[carllerche]

@seanmonstar I assume that this would just be a convention used by middleware, not a change to the Service trait?

[seanmonstar]

Correct, I mean only as a pattern of middleware.

[carllerche]

@seanmonstar what’s the conclusion?

[davidbarsky]

Given the discussion in #141 and my limited experience with Tower composing Tower's middleware I think I'm in favor of this change for a few reasons:

• Tower already encourages a large degree of nested types and reducing the amount of nested types an end-customer needs to worry about is a win IMO.
• I've already mentioned this in a prior comment, but given my suspicion that a large chunk of errors deep within a middleware stack can't be handled by other middleware layers, doing some type erasure via the cast to a boxed object is appealing. Plucking out specific errors using error.downcast_ref appears to be a pattern people are gravitating with Failure and Error::source().
• A pain-point I encountered with Tower is composing different error types between middleware layers. It might be me misusing Tower, but I suspect that other users might run into the same issue.

As Sean said,

Set my expect error type exactly, and thus use match e { UnNestAllTheLayers => () }. This is very fragile, since adding new middleware, or changing the order, will cause compilation to fail. It's also hugely tedious being sure we've match every correct combination.

I strongly concur on this point.

[carllerche]

This has been done on master.