Benchmarking Tool

Linux command-line Benchmarking Tool

A port of the hyperfine Benchmarking Tool to Bash.

Does NOT require installing Rust, downloading dependencies or compiling anything.
Includes the same features (except Linux only), produces the same output (with some improvements) and supports most of the same command line options.
Outputs most of the numbers with greater precision and outputs more information.
Supports outputting in ASCII only (no Unicode characters) to support older terminals.
Slightly faster when interactive output (the progress bar) is disabled, as it does not need to launch intermediate shells.

❤️ Please visit tealdulcet.com to support this script and my other software development.

Benchmark of the GNU factor and uutils factor commands.

Also see the Testing and Benchmarking scripts.

Usage

Run: ./time.sh [OPTIONS] <command(s)>...
All the options can also be set by opening the script in an editor and setting the variables at the top. See Help below for full usage information.

Download the script (time.sh). Run: wget https://raw.github.com/tdulcet/Benchmarking-Tool/main/time.sh.
Execute the script once to make sure there are no errors. For example, run: chmod u+x time.sh and ./time.sh 'sleep 0.3'.
If you want the script to be available for all users, install it. Run: sudo cp time.sh /usr/local/bin/benchmark and sudo chmod +x /usr/local/bin/benchmark.

Help

$ benchmark -h
Usage:  benchmark [OPTION(S)] <command(s)>...
or:     benchmark <OPTION>
All the options can also be set by opening the script in an editor and setting the variables at the top. See examples below.

Options:
    -w <NUM>        Warmup
                        Perform NUM warmup runs before the actual benchmark. This can be used to fill (disk) caches for I/O-heavy programs. Default: 0
    -m <NUM>        Min-runs
                        Perform at least NUM runs for each command. Default: 10
    -M <NUM>        Max-runs
                        Perform at most NUM runs for each command. Default: no limit
    -r <NUM>        Runs
                        Perform exactly NUM runs for each command. If this option is not specified, it will automatically determines the number of runs.
    -s <command>    Setup
                        Execute command before each set of runs. This is useful for compiling your software with the provided parameters, or to do any other work that should happen once before a series of benchmark runs, not every time as would happen with the prepare option.
    -p <command>    Prepare
                        Execute command before each run. This is useful for clearing disk caches, for example. The prepare option can be specified once for all commands or multiple times, once for each command. In the latter case, each preparation command will be run prior to the corresponding benchmark command.
    -c <command>    Cleanup
                        Execute command after the completion of all benchmarking runs for each individual command to be benchmarked. This is useful if the commands to be benchmarked produce artifacts that need to be cleaned up.
    -i              Ignore-failure
                        Ignore non-zero exit codes of the benchmarked programs.
    -u              ASCII
                        Do not use Unicode characters in output.
    -S              Disable interactive
                        Disable interactive output and progress bars.
    -C <FILE>       Export CSV
                        Export the timing summary statistics as CSV to the given FILE.
    -j <FILE>       Export JSON
                        Export the timing summary statistics and timings of individual runs as JSON to the given FILE.
    -o <WHERE>      Output
                        Control where the output of the benchmark is redirected. <WHERE> can be:
                            null: Redirect both stdout and stderr to '/dev/null' (default).
                            pipe: Feed stdout through a pipe before discarding it and redirect stderr to '/dev/null'.
                            inherit: Output the stdout and stderr.
                            <FILE>: Write both stdout and stderr to the given FILE.
    -n <NAME>       Command-name
                        Give a meaningful name to a command. This can be specified multiple times if several commands are benchmarked.
    -h              Display this help and exit
    -v              Output version information and exit

Examples:
    Basic benchmark
    $ benchmark 'sleep 0.3'

    Benchmark two commands
    $ benchmark 'find -iname "*.jpg"' 'fd -e jpg -uu'

    Benchmark piped commands
    $ benchmark 'seq 0 10000000 | factor' 'seq 0 10000000 | uu-factor'

    Warmup runs
    $ benchmark -w 3 'grep -R TODO *'

    Parameterized benchmark
    $ benchmark -p 'make clean' 'make -j '{1..12}
    This performs benchmarks for 'make -j 1', 'make -j 2', … 'make -j 12'.

    Parameterized benchmark with step size
    $ benchmark 'sleep 0.'{3..7..2}
    This performs benchmarks for 'sleep 0.3', 'sleep 0.5' and 'sleep 0.7'.

    Parameterized benchmark with list
    $ benchmark {gcc,clang}' -O3 main.c'
    This performs benchmarks for 'gcc -O3 main.c' and 'clang -O3 main.c'.

Contributing

Pull requests welcome! Ideas for contributions:

Support more of hyperfine's options
- Support disabling color output
Add option to use the GNU time command (/usr/bin/time)
Add more examples
Improve the performance
Add tests

README claims

Hi!

Author of hyperfine here. Glad to see that you liked our tool and decided to port it to bash. I have a few questions regarding the statements in the README. Not because I want to claim they are wrong, but because I'm genuinely curious:

produces the same output (with some improvements)

What would be some of these improvements? Maybe we could profit from these in hyperfine as well?

Outputs most of the numbers with greater precision

hyperfine reports all times in millisecond resolution because I don't know how to make a measurement that would be more precise. There is no sense in showing more digits if the actual measurement is not that precise. The problem is that we are spawning an intermediate shell that takes roughly 5 milliseconds on its own. We subtract that time again, but we can not expect to measure microsecond-resolution execution times on top of a 4.8 ms ± 3.7 ms shell spawning time. This is why hyperfine also shows a warning if commands take less than 5 ms to complete.

To get more precise timings, we would need to get rid of the intermediate shell. This is possible in principle, but would keep us from (easily) running benchmarks like seq 100000 | factor.

and outputs more information.

What kind of information would that be? CPU usage?

Supports outputting in ASCII only (no Unicode characters) to support older terminals.

Nice!

Slightly faster when interactive output (the progress bar) is disabled.

Slightly faster than what?

tdulcet / benchmarking-tool Goto Github PK