xuanwo / backon

Retry with backoff without effort.

Hi, I'm Xuanwo 👋

My VISION is Data Freedom.

I aim to create an ecosystem where users can access ANY data, across ANY service, using ANY method, in ANY language.

• Find out what I'm doing at Github
• Follow me on Twitter
• Find more links about me at bento

If our visions align in some part, feel free to get in touch so we can collaborate.

• I'm an ASF Member, Apache OpenDAL PMC Chair, and a mentor at the Apache Incubator. If you have projects that align with my vision and are interested in bringing them to the Apache Incubator, please contact me.
• I actively contribute to many Rust projects. If you're interested in learning Rust and joining the RBIR journey, please contact me.
• I'm a believer of open source. If you're developing one that aligns with my vision, please contact me; I'm willing to offer some help.
• Last but not least: I'm not a native English speaker and am currently practicing my English. If you're interested in having an English conversation with me, I would really appreciate it. Please contact me.

---

Projects that I'm working on:

• opendal: A unified data access layer, empowering users to seamlessly and efficiently retrieve data from diverse storage services
• iceberg-rust: Rust implementation of Apache Iceberg.
• paimon-rust: The rust implementation of Apache Paimon.
• arrow-rs: Official Rust implementation of Apache Arrow.
• fury-rs: A blazingly fast multi-language serialization framework powered by JIT and zero-copy.
• databend: A modern Elasticity and Performance cloud data warehouse, activate your object storage for real-time analytics.

Projects that I maintained:

• sccache: Shared Compilation Cache.
• reqsign: Signing HTTP requests without heavy SDKs.
• backon: Another implementation of backoff via Iterator.

Let's rock!

backon's Issues

`FnMut` is rather too strong to implement

Sometimes only the Fn restriction is enough. And it's hard to acquire in asynchronous programming.

recipe could be added to the rustdocs

          @Xuanwo maybe this recipe could be added to the rustdocs? I also was confused by a similar compiler error .

Originally posted by @qrilka in #50 (comment)

Add support for notify

Send a message out to outer everytime we retry a future.

new idea: introduce fuzz into back to make sure it implemented correctly

Unlimited retries possible in implementation, but not trough public API

If I'm seeing this correctly, it should be possible to have unlimited retries in the implementation, but there seems no way of doing that via the public API, because the Default implementation defaults to 3 retries?

Is that intended or did I miss anything?

Make async support optional

Docs can be improved a lot

From reddit:

Does ExponentialBackoff's max_times and max_delay meets your need?

No.

First, the documentation could be better. "If max times is reaching" is unclear and ungrammatical. I might say "Backoff will return Some at most max_times times."

Second, what I'd want is an absolute deadline, and stopping early if the next sleep won't finish until after then. Same idea with the budget: I want a per-attempt check I want to do before the sleep, or a fallible callback to do the sleep. I could do this myself by using the Backoff as an Iterator, but then I don't get the slickness of the Retryable/BlockingRetryable traits.

backon support this feature by accepting a function notify

Oh, I missed that too, sorry! The doc could be better here too though. I might say "registers a notification to be called before waiting for a next attempt"; name the callback's parameters as previous_error and sleep_time, respectively; and amend the log message in the example to will retry after {dur:?} due to error: {err:?}". Could even have the arguments in a struct to allow adding more stuff there without callbacks needing to see stuff they're not interested in (e.g., could add attempt count).

Does not work out of the box with wasm32-unknown-unknown (but there's a workaround)

When using the crate in a project targeting wasm32-unknown-unknown, the following needs to be added to the project's cargo.toml:

instant = { version = "0.1.12", features = ["wasm-bindgen"] }

Otherwise an error described here is logged (in my case to the browser console).

Pick a more appropriate name for `with_error_fn`?

@Xuanwo This crate is awesome! I never thought of a retry mechanism that could be implemented in such an elegant way. 👍

However the function name with_error_fn is somewhat ambiguous for me. I cannot guess what it is used for without reading the comments.

How about renaming it to continue_if or break_if (on the contrary), following the naming convention in std::ops::ControlFlow:

let content = fetch
    .retry(ExponentialBackoff::default())
    .continue_if(|e| e.to_string() != "EOF") // continue if error is not "EOF"
    .await?;

v.s.

let content = fetch
    .retry(ExponentialBackoff::default())
    .break_if(|e| e.to_string() == "EOF") // stop retrying when error is "EOF"
    .await?;

Personally I think break_if is a good choice. What do you think of it?

Add Fibonacci backoff support

Can backoff reset?

I use ExponentialBackoff to retry in 1s..2s..4s then connected.
Then disconnected again and start retry in 8s...16s...32s.
What i want is backoff.reset(),then retry in 1s..2s..4s in second retry.

Potential for dynamic backoffs

I'm interesting in using backon for retrying HTTP requests. I'd like to handle the retry-after HTTP header, where the server essentially can hint at which point it'd like you to retry. Which means I'd like to adjust my backoff time depending on that value.

Currently I can't figure out how I'd accomplish this as Backoff is just an iterator over Durations. The trait would need to change to have a method implementation that e.g. took in the previous backoffs and the current error, so the error could be inspected for a retry time.

Is this a feasible change? Or perhaps this is already possible and I'm not understanding the API correctly?

the longest back off will be wasted

Sorry for forgetting to break after a success request.

We are using ExponentialBackOff directly so things get different with just using a backon::retry() method or backon::Retry structure.

When the iteration just restarts from the longest back off, the next item in iterator will be None, then the for loop will be quited and the longest back off will be wasted.

To avoid this, to retry the same request after the longest back off, a zero duration sleep will be added. If the very last request failed, sleep(0).await cost little, the loop will reach the end and the error will be returned in time.

Originally posted by @ClSlaid in datafuselabs/databend#7809 (comment)

backon can't be used in along with `&mut self`

async fn next(&mut self) -> Result<Option<Vec<oio::Entry>>> {
    { || self.inner.next() }
        .retry(&self.builder)
        .when(|e| e.is_temporary())
        .notify(|err, dur| {
            warn!(
                target: "opendal::service",
                "operation={} -> retry after {}s: error={:?}",
                Operation::Batch, dur.as_secs_f64(), err)
        })
        .map(|v| v.map_err(|e| e.set_persistent()))
        .await
}

Errors log:

error: captured variable cannot escape `FnMut` closure body
   --> src/layers/retry.rs:634:14
    |
633 |     async fn next(&mut self) -> Result<Option<Vec<oio::Entry>>> {
    |                    -------- variable defined here
634 |         { || self.inner.next() }
    |            - ----^^^^^^^^^^^^^
    |            | |
    |            | returns a reference to a captured variable which escapes the closure body
    |            | variable captured here
    |            inferred to be a `FnMut` closure
    |
    = note: `FnMut` closures only have access to their captured variables while they are executing...
    = note: ...therefore, they cannot allow references to captured variables to escape

Add examples for user easier to discover how to use backon

Retry a blocking function

Unable to get Retry to work

I'm sure I'm doing something completely stupid, but just can't work it out. In my reproduction of what I'm experiencing in my real project, I just can't get backon working with my future.

use backon::ConstantBackoff;
use backon::Retryable;
use std::error::Error;
use std::thread;
use std::time::Duration;

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn Error>> {
    let _ = foo().retry(&ConstantBackoff::default()).await?;
    Ok(())
}

async fn foo() -> Result<(), Box<dyn Error + Send + Sync>> {
    async { thread::sleep(Duration::from_secs(1)) }.await;
    Ok(())
}

results in the following error

error[E0599]: the method `retry` exists for opaque type `impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>`, but its trait bounds were not satisfied
 --> src/main.rs:9:19
  |
9 |     let _ = foo().retry(&ConstantBackoff::default()).await?;
  |                   ^^^^^ method cannot be called on `impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>` due to unsatisfied trait bounds
  |
  = note: the following trait bounds were not satisfied:
          `<impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>> as FnOnce<()>>::Output = _`
          which is required by `impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: Retryable<_, _, _, _, impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>>`
          `impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: FnMut<()>`
          which is required by `impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: Retryable<_, _, _, _, impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>>`
          `<&impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>> as FnOnce<()>>::Output = _`
          which is required by `&impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: Retryable<_, _, _, _, &impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>>`
          `&impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: FnMut<()>`
          which is required by `&impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: Retryable<_, _, _, _, &impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>>`
          `<&mut impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>> as FnOnce<()>>::Output = _`
          which is required by `&mut impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: Retryable<_, _, _, _, &mut impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>>`
          `&mut impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: FnMut<()>`
          which is required by `&mut impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>: Retryable<_, _, _, _, &mut impl Future<Output = Result<(), Box<(dyn std::error::Error + Send + Sync + 'static)>>>>`

warning: unused import: `backon::Retryable`
 --> src/main.rs:2:5
  |
2 | use backon::Retryable;
  |     ^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

For more information about this error, try `rustc --explain E0599`.
warning: `mvr` (bin "mvr") generated 1 warning
error: could not compile `mvr` due to previous error; 1 warning emitted

I'm completely baffled. I've tried adding in anyhow, thinking it may be doing some magic, but get the same error. What am I doing wrong?

How to pass parameters to function call

let content = fetch.retry(&ExponentialBuilder::default()).await?;

How do we pass parameters to function call? For example, say fetch took a string parameters representing the URL.

Define MSRV for backon

Add features provided by k8s.io/apimachinery/pkg/util/wait

https://pkg.go.dev/k8s.io/apimachinery/pkg/util/wait

Add jitter support for constant back off

backon 1.0

backon has been used in production for a while. It's time for us to clean up some old code in the repository, consider our project scope, and prepare a 1.0 release.

Use with sqlx?

Hey!

The library seems pretty cool! I am new to rust, trying to implement a retry strategy on a small rest API project. Could you help me understand how to use it?

I am trying to implement a retry strategy on a SQLx query. If it fails I want to retry it before returning a 5xx error.

I would normally run the code such as:

query
    .execute(pool.inner())
    .await

Now, with backon;
This doesn't work -

query
    .execute(pool.inner())
    .retry(&ExponentialBuilder::default())
    .await

Error:

| error[E0599]: the method `retry` exists for opaque type `impl Future<Output = Result<<Postgres as Database>::QueryResult, Error>>`, but its trait bounds were not satisfied
|    --> src/model/user/create.rs:105:10
|     |
| 103 |       let rows_affected = query
|     |  _________________________-
| 104 | |         .execute(pool.inner())
| 105 | |         .retry(&ExponentialBuilder::default())
|     | |         -^^^^^ method cannot be called due to unsatisfied trait bounds
|     | |_________|
|     | 
|     |
|     = note: the following trait bounds were not satisfied:
|             `<impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>> as FnOnce<(_,)>>::Output = _`
|             which is required by `impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: RetryableWithContext<_, _, _, _, _, impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>>`
|             `impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: FnMut<(_,)>`
|             which is required by `impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: RetryableWithContext<_, _, _, _, _, impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>>`
|             `<&impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>> as FnOnce<(_,)>>::Output = _`
|             which is required by `&impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: RetryableWithContext<_, _, _, _, _, &impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>>`
|             `&impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: FnMut<(_,)>`
|             which is required by `&impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: RetryableWithContext<_, _, _, _, _, &impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>>`
|             `<&mut impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>> as FnOnce<(_,)>>::Output = _`
|             which is required by `&mut impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: RetryableWithContext<_, _, _, _, _, &mut impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>>`
|             `&mut impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: FnMut<(_,)>`
|             which is required by `&mut impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>: RetryableWithContext<_, _, _, _, _, &mut impl std::future::Future<Output = Result<<Postgres as sqlx::Database>::QueryResult, sqlx::Error>>>`

And using a lamba doesn't seem to work either:

{
        || async {
            let connection = pool.acquire().await.map_err(anyhow::Error::from)?;
            query.execute(connection).await.map_err(anyhow::Error::from)
        }
    }
        .retry(&ExponentialBuilder::default())
        .await?;

Error:

| error[E0277]: the trait bound `sqlx::pool::PoolConnection<Postgres>: Executor<'_>` is not satisfied
|    --> src/model/user/create.rs:105:27
|     |
| 105 |             query.execute(connection).await.map_err(anyhow::Error::from)
|     |                   ------- ^^^^^^^^^^ the trait `Executor<'_>` is not implemented for `sqlx::pool::PoolConnection<Postgres>`
|     |                   |
|     |                   required by a bound introduced by this call
|     |
|     = help: the following other types implement trait `Executor<'c>`:
|               <&'c mut PgConnection as Executor<'c>>
|               <&'c mut PgListener as Executor<'c>>
|               <&'c mut AnyConnection as Executor<'c>>
|               <&Pool<DB> as Executor<'p>>
| note: required by a bound in `sqlx::query::Query::<'q, DB, A>::execute`
|    --> /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlx-core-0.7.4/src/query.rs:155:12
|     |
| 151 |     pub async fn execute<'e, 'c: 'e, E>(self, executor: E) -> Result<DB::QueryResult, Error>
|     |                  ------- required by a bound in this associated function
| ...
| 155 |         E: Executor<'c, Database = DB>,
|     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `Query::<'q, DB, A>::execute`

Use Boxed Fns for when and notify to allow closures

The notify API only takes fn(...) references right now, but I'd like to be able to write to my UI. I need to use a closure for that.

Instead, the API should use

pub fn notify(self, notify: impl Fn(&E, Duration));

The only downside to this is making the Retry objects 2 words larger, for erased user state. Note that Box<dyn Fn(..)> doesn't allocate for basic fns

xue ni le

Make jitter has a clear value range

          > > Maybe a new function like `ConstantBuilder::unlimited` would be better for this?

Seems not a good idea to me. I plan to add a FnBuilder that accept a function which will return the next delay like the following:

let f = FnBuilder::new(|| Some(Durtion::from_secs(1)));

What do you think?

It should follow the principle of least surprise. User of library shouldn't look into the implementation to find out what function will do. I've spent some time finding out why it nearly instantly exits with such setup:

backon::FibonacciBuilder::default()
            .with_jitter()
            .with_min_delay(Duration::from_millis(5))
            .with_max_delay(Duration::from_millis(100))

Originally posted by @0xdeafbeef in #48 (comment)

can we use use std::result::Result instad of anyhow:Result?

as title.

the exponential strategy is likely to panic when it exceeds the upper bound of u64

from reddit:

the exponential strategy is likely to panic when it exceeds the upper bound of u64. see Duration::mul_f32()

a safer alternative to it is Dutation::saturating_mul()