Retry if post fails because of rate limiting about slack-notifier

[siegy22]

I don't think that's the way to deal with an API rate limit, especially if it contains something like sleep(rand(1..10)). If you hit the rate limit, check why you're hitting it. Are you running a loop which does a lot of API requests which can be improved? Can you "verify" your application to not hit a rate limit as small as you're hitting it right now?

[anchit-desai]

The functions that call the API are invoked by user actions, so it's hard for us to tell when these will occur.

[paprikati]

I personally think the implementation of a backoff strategy is going to vary for each caller. I'm not sure there's a nice 'standard' approach we could apply within the gem that would be useful for lots of people

[rjhancock]

https://api.slack.com/docs/rate-limits , specifically under Responding to rate limiting conditions, will give you a time interval to use instead of the sleep command.

Probably a better approach.

[paprikati]

that could work as something reusable - how do we feel about whether it should be enabled or disabled by default?

[rjhancock]

I think it's something that should be non-optional. Have the gem just handle it and make a note in the documentation.

[paprikati]

should we retry infinitely? that seems sad... or keep a count of the number of attempts and cap it at 2 or 3? wdyt

[rjhancock]

I think utilizing the Retry content would be fine as eventually it'll get through unless someone is doing a very high volume of messages.

[paprikati]

I don't think retrying infinitely is a good idea as you can potentially lock up the thread forever (e.g. if you have some other service that is hammering slack for some reason, and you are sending slack messages in your production process rather than in a worker). I agree it requires a weird set of bugs to happen, but I wouldn't want to bring someone down like that (it would be super sad)

I propose we add this as optional behaviour (so its not a breaking change) as someone might want to handle the retries themselves (e.g. by enqueuing a new worker) rather than letting a thread sleep - for example that's what we do.

I think we should provide the following options:

• max_attempts (default 3, set to 0 for infinite)
• max_sleep_time (default 30, measured in seconds) - so if there's a slack bug / weirdness where it returns a very large number in Retry-After then again your thread is not sitting dead for ages and ages. Slack's docs seem to suggest they usually send 1-3 seconds so you're unlikely to hit this anyway.

I also think we should also raise a specific error if a request is rate-limited (and we aren't retrying), which exposes the retry-after so a client can handle it.

I'll write a PR this week and see if I can get one of the maintainers to take a look

[saiqulhaq]

this gem accepts custom HTTP client
you can use an HTTP client using this gem https://github.com/lostisland/faraday-retry that support exponential backoff
then use it as the README https://github.com/slack-notifier/slack-notifier#custom-http-client
so can we close this issue?

[pedros007]

this gem accepts custom HTTP client

Can anyone provide an example using Faraday as an alternate HTTP client? The example in the faraday-retry repo shows how to create a Faraday connection, not a client as required by the slack-notifier Gem API.

[saiqulhaq]

this gem accepts custom HTTP client

Can anyone provide an example using Faraday as an alternate HTTP client? The example in the faraday-retry repo shows how to create a Faraday connection, not a client as required by the slack-notifier Gem API.

@pedros007 you just need to create a custom faraday instance with retry middleware, then pass http_client

slack-notifier/lib/slack-notifier.rb

Line 19 in e9aace7

config.http_client(options.delete(:http_client)) if options.key?(:http_client)