kadwanev / retry

The command line retry tool

retry - The command line retry tool

Retry any shell command with exponential backoff or constant delay.

Instructions

Install:

retry is a shell script, so drop it somewhere and make sure it's added to your $PATH. Or you can use the following one-liner:

sudo sh -c "curl https://raw.githubusercontent.com/kadwanev/retry/master/retry -o /usr/local/bin/retry && chmod +x /usr/local/bin/retry"

If you're on OS X, retry is also on Homebrew:

brew install kadwanev/brew/retry

Not popular enough for homebrew-core. Please star this project to help.

Usage

Help:

retry -?

Usage: retry [options] -- execute command
    -h, -?, --help
    -v, --verbose                    Verbose output
    -t, --tries=#                    Set max retries: Default 10
    -s, --sleep=secs                 Constant sleep amount (seconds)
    -m, --min=secs                   Exponential Backoff: minimum sleep amount (seconds): Default 0.3
    -x, --max=secs                   Exponential Backoff: maximum sleep amount (seconds): Default 60
    -f, --fail="script +cmds"        Fail Script: run in case of final failure

State

Environment variable RETRY_ATTEMPT is set with the current attempt number.

Examples

No problem:

retry echo u work good

u work good

Test functionality:

retry 'echo "y u no work"; false'

y u no work
Before retry #1: sleeping 0.3 seconds
y u no work
Before retry #2: sleeping 0.6 seconds
y u no work
Before retry #3: sleeping 1.2 seconds
y u no work
Before retry #4: sleeping 2.4 seconds
y u no work
Before retry #5: sleeping 4.8 seconds
y u no work
Before retry #6: sleeping 9.6 seconds
y u no work
Before retry #7: sleeping 19.2 seconds
y u no work
Before retry #8: sleeping 38.4 seconds
y u no work
Before retry #9: sleeping 60.0 seconds
y u no work
Before retry #10: sleeping 60.0 seconds
y u no work
etc..

Limit retries:

retry -t 4 'echo "y u no work"; false'

y u no work
Before retry #1: sleeping 0.3 seconds
y u no work
Before retry #2: sleeping 0.6 seconds
y u no work
Before retry #3: sleeping 1.2 seconds
y u no work
Before retry #4: sleeping 2.4 seconds
y u no work
Retries exhausted

Bad command:

retry poop

bash: poop: command not found

Fail command:

retry -t 3 -f 'echo "oh poopsicles"' 'echo "y u no work"; false'

y u no work
Before retry #1: sleeping 0.3 seconds
y u no work
Before retry #2: sleeping 0.6 seconds
y u no work
Before retry #3: sleeping 1.2 seconds
y u no work
Retries exhausted, running fail script
oh poopsicles

Last attempt passed:

retry -t 3 -- 'if [ $RETRY_ATTEMPT -eq 3 ]; then echo Passed at attempt $RETRY_ATTEMPT; true; else echo Failed at attempt $RETRY_ATTEMPT; false; fi;'

Failed at attempt 0
Before retry #1: sleeping 0.3 seconds
Failed at attempt 1
Before retry #2: sleeping 0.6 seconds
Failed at attempt 2
Before retry #3: sleeping 1.2 seconds
Passed at attempt 3

License

Apache 2.0 - go nuts

retry's Issues

Environment variables not passed

This fails 👎

ENV=hot; retry -t 2 -e python -c 'import os; print os.environ["ENV"]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/V672336/.pyenv/versions/2.7.11/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'ENV'
Before retry #1: sleeping 0.3 seconds

provide nix flake package

brew pull 27283 don't work

Error: Calling brew pull is disabled! Use hub checkout instead.

I’m sorry, 'getopt --test' failed in this environment. Please load GNU getopt.

Fresh install from homebrew as described in the readme

Liams-MBP:istio liam$ retry -s 10 -t 10 'echo "this is a test"; false'
I’m sorry, 'getopt --test' failed in this environment. Please load GNU getopt.

Reimplement in plain bash

Removing ruby dependency for this tool is a good thing.

Fail script will run even if last attempt is successful

If a command fails a few times but then succeeds on the final attempt, retry will still print

Retries exhausted

or

Retries exhaused, running fail script

and then return 0.

I think the line

if [ $attempts -gt $max_tries ]; then

needs to be

if [[ $return_code -ne 0 && $attempts -gt $max_tries ]]; then

to fix this.

sleep: invalid time interval ‘0,6’

I have this error on my host:

Before retry #1: sleeping 0,3 seconds
sleep: invalid time interval ‘0,3’
Try 'sleep --help' for more information.
Before retry #2: sleeping 0,6 seconds
sleep: invalid time interval ‘0,6’
Try 'sleep --help' for more information.
Before retry #3: sleeping 1,2 seconds
sleep: invalid time interval ‘1,2’
Try 'sleep --help' for more information.
Before retry #4: sleeping 2,4 seconds
sleep: invalid time interval ‘2,4’
Try 'sleep --help' for more information.
Before retry #5: sleeping 4,8 seconds
sleep: invalid time interval ‘4,8’
Try 'sleep --help' for more information.

Best regards,
Stéphane

Dependency on awk

The shell implementation requires the ability to do floating point calculations for calculating exponential backoff. Out of the workable options, I found only awk available in the minified system images I was working with. I suppose this dependency could cause problems for some.
Should implementations for others be included?
Others that could work: python, ruby, bc, dc

Default writes to stderr

retry writes all messages to stderr by default. Shouldn't only be the actual failure (e.g. retries exhausted) and the failed command output be written to stderr? The rest could technically just go towards stdout, that would make it easier to pick out the actual failure in a CI result.
I'd be happy to provide an according PR.

Fail script

Allow running something if the final result is a fail after all retries.

Unit test

Add unit testing using roundup for bash implementation.

verify shell being run

Something funny going on.

Retried script has shebang forcing bash, but if the shell uses bash specific features, it looks like dash is being run. Specificially set -o pipefail so the way it's being run might not be honoring the shebang.

support retry processing of stdin input

Would need to read in stdin and buffer it for retry. Similarly buffer stdout until success. Only output final failed result or success.

mac-os not supported

use retry failed bcz longoptions is not supported on mac

$ getopt --test
 --
$ echo $?
0

Maybe there are two solution ways:

1. make --longoptions be optional in retry implement
2. use brew install gnu-getopt instead of getopt

retry N times looking to see if script fails, and stop on that failure?

I am trying to look for a script failing, e.g. flaky test. Can I use this script to retry N times and stop when I hit the flaky test failure? I am sure that this could be done but wondering if there is a nice idiomatic way to use this tool for it