fuzzymonkeyco / monkey

@FuzzyMonkeyCo's minion

monkey ~ FuzzyMonkeyCo's minion

FuzzyMonkey is an automated API testing service that behaves as your users would and minimizes sequences of calls that lead to a violation of your software's properties.

monkey is the official open source client that executes the tests FuzzyMonkey generates.

monkey M.m.p go1.18.3 linux amd64

Usage:
  monkey [-vvv]           env [VAR ...]
  monkey [-vvv] [-f STAR] fmt [-w]
  monkey [-vvv] [-f STAR] lint [--show-spec]
  monkey [-vvv] [-f STAR] exec (repl | start | reset | stop)
  monkey [-vvv] [-f STAR] schema [--validate-against=REF]
  monkey [-vvv] [-f STAR] fuzz [--intensity=N] [--seed=SEED]
                               [--label=KV]...
                               [--tags=TAGS | --exclude-tags=TAGS]
                               [--no-shrinking]
                               [--progress=PROGRESS]
                               [--time-budget-overall=DURATION]
                               [--only=REGEX]... [--except=REGEX]...
                               [--calls-with-input=SCHEMA]...  [--calls-without-input=SCHEMA]...
                               [--calls-with-output=SCHEMA]... [--calls-without-output=SCHEMA]...
  monkey        [-f STAR] pastseed
  monkey        [-f STAR] logs [--previous=N]
  monkey [-vvv]           update
  monkey                  version | --version
  monkey                  help    | --help    | -h

Options:
  -v, -vv, -vvv                   Debug verbosity level
  -f STAR, --file=STAR            Name of the fuzzymonkey.star file
  version                         Show the version string
  update                          Ensures monkey is the latest version
  --intensity=N                   The higher the more complex the tests [default: 10]
  --time-budget-overall=DURATION  Stop testing after DURATION (e.g. '30s' or '5h')
  --seed=SEED                     Use specific parameters for the Random Number Generator
  --label=KV                      Labels that can help classification (format: key=value)
  --tags=TAGS                     Only run checks whose tags match at least one of these (comma separated)
  --exclude-tags=TAGS             Skip running checks whose tags match at least one of these (comma separated)
  --progress=PROGRESS             dots, bar, ci (defaults: dots)
  --only=REGEX                    Only test matching calls
  --except=REGEX                  Do not test these calls
  --calls-with-input=SCHEMA       Test calls which can take schema PTR as input
  --calls-without-output=SCHEMA   Test calls which never output schema PTR
  --validate-against=REF          Validate STDIN payload against given schema $ref
  --previous=N                    Select logs from Nth previous run [default: 1]

Try:
     export FUZZYMONKEY_API_KEY=fm_42
  monkey update
  monkey -f fm.star exec reset
  monkey fuzz --only /pets --calls-without-input=NewPet --seed=$(monkey pastseed)
  echo '"kitty"' | monkey schema --validate-against=#/components/schemas/PetKind

Getting started

Recommended way: using the GitHub Action.

Quick install:

curl -#fL https://git.io/FuzzyMonkey | BINDIR=/usr/local/bin sh

# or the equivalent:
curl -#fL https://raw.githubusercontent.com/FuzzyMonkeyCo/monkey/master/.godownloader.sh | BINDIR=/usr/local/bin sh

With Docker:

DOCKER_BUILDKIT=1 docker build -o=/usr/local/bin --platform=local https://github.com/FuzzyMonkeyCo/monkey.git

# or the faster:
DOCKER_BUILDKIT=1 docker build -o=/usr/local/bin --platform=local --build-arg PREBUILT=1 https://github.com/FuzzyMonkeyCo/monkey.git

Or simply install the latest release.

Configuration

monkey uses Starlark as its configuration language: a simple Python-like deterministic language.

Minimal example fuzzymonkey.star file

OpenAPIv3(
  name = "dev_spec",
  file = "openapi/openapi.yaml",
  host = "http://localhost:3000",

  ExecReset = "curl -fsSL -X DELETE http://localhost:3000/api/1/items",
)

Demos

• demo_erlang_cowboy_simpleREST

A more involved fuzzymonkey.star

# Invariants of our APIs expressed in a Python-like language

assert that(monkey.env("TESTING_WHAT", "demo")).is_equal_to("demo")
SPEC = "pkg/runtime/testdata/jsonplaceholder.typicode.comv1.0.0_openapiv3.0.1_spec.yml"
print("Using {}.".format(SPEC))

monkey.openapi3(
    name = "my_spec",
    # Note: references to schemas in `file` are resolved relative to file's location.
    file = SPEC,
    host = "https://jsonplaceholder.typicode.com",
    # header_authorization = "Bearer {}".format(monkey.env("DEV_API_TOKEN")),
)

# Note: exec commands are executed in shells sharing the same environment variables,
# with `set -e` and `set -o pipefail` flags on.

# List here the commands to run so that the service providing "my_spec"
# can be restored to its initial state.
monkey.shell(
    name = "example_resetter",

    # Link to above defined spec.
    provides = ["my_spec"],

    # The following gets executed once per test
    #   so have these commands complete as fast as possible.
    # For best results, tests should start with a clean slate
    #   so limit filesystem access, usage of $RANDOM and non-reproducibility.
    reset = """
echo ${BLA:-42}
BLA=$(( ${BLA:-42} + 1 ))
echo Resetting state...
    """,
)

## Ensure some general property

def ensure_lowish_response_time(ms):
    def responds_in_a_timely_manner(ctx):
        assert that(ctx.response.elapsed_ms).is_at_most(ms)

    return responds_in_a_timely_manner

monkey.check(
    name = "responds_in_a_timely_manner",
    after_response = ensure_lowish_response_time(500),
    tags = ["timings"],
)

## Express stateful properties

def stateful_model_of_posts(ctx):
    """Properties on posts. State collects posts returned by API."""

    # NOTE: response has already been decoded & validated for us.

    url = ctx.request.url

    if all([
        ctx.request.method == "GET",
        "/posts/" in url and url[-1] in "1234567890",  # /posts/{post_id}
        ctx.response.status_code in range(200, 299),
    ]):
        post_id = url.split("/")[-1]
        post = ctx.response.body

        # Ensure post ID in response matches ID in URL (an API contract):
        assert that(str(int(post["id"]))).is_equal_to(post_id)

        # Verify that retrieved post matches local model
        if post_id in ctx.state:
            assert that(post).is_equal_to(ctx.state[post_id])

        return

    if all([
        ctx.request.method == "GET",
        url.endswith("/posts"),
        ctx.response.status_code == 200,
    ]):
        # Store posts in state
        for post in ctx.response.body:
            post_id = str(int(post["id"]))
            ctx.state[post_id] = post
        print("State contains {} posts".format(len(ctx.state)))

monkey.check(
    name = "some_props",
    after_response = stateful_model_of_posts,
)

## Encapsulation: ensure each monkey.check owns its own ctx.state.

def encapsulation_1_of_2(ctx):
    """Show that state is not shared with encapsulation_2_of_2"""
    assert that(ctx.state).is_empty()

monkey.check(
    name = "encapsulation_1_of_2",
    after_response = encapsulation_1_of_2,
    tags = ["encapsulation"],
)

monkey.check(
    name = "encapsulation_2_of_2",
    after_response = lambda ctx: None,
    state = {"data": 42},
    tags = ["encapsulation"],
)

## A test that always fails

def this_always_fails(ctx):
    assert that(ctx).is_none()

monkey.check(
    name = "always_fails",
    after_response = this_always_fails,
    tags = ["failing"],
)

Issues?

Report bugs on the project page or contact us.

License

See LICENSE

monkey's Issues

use fancy term windows for logs, results, ...?

https://github.com/rivo/tview

use platform-tolerant tmp/cache/log paths

https://golang.org/pkg/io/ioutil/#TempFile
golang/go@50bd1c4

badges badges badges

https://shields.io

todo: use dep to lock dependencies

https://github.com/golang/dep

And drop gopkg.in

Provide handy shell utils such as travis_retry / circleci's no_output_timeout

See https://github.com/travis-ci/travis-build/blob/d63c9e95d6a2dc51ef44d2a1d96d4d15f8640f22/lib/travis/build/script/templates/header.sh#L97

call Stop or Reset when testman is interrupted

Cleanup behind testman, stop containers, ... when getting ^C.

open web UI automatically

Here's how: https://github.com/golang/tools/blob/99037e3760ed7d9c772c980caee42b17779b80ce/cmd/cover/html.go#L143-L156

create only ever one instance of the shell command runner

pipe commands to it.
wait on trap EXIT or a argument-provided timeout.
read $? and maybe set a var which name changes often to convey some state if needed.

https://stackoverflow.com/a/2130323/1418165
https://stackoverflow.com/a/9357505/1418165

todo: return err instead of log.Fatal() for defers & cleanliness

todo: ensure builds are reproducible

https://blog.filippo.io/reproducing-go-binaries-byte-by-byte/

Maybe test using CircleCI and racing to compare x64 linux build sha1s.

Support both QC and Hedgehog?

Is it possible to use this service but instead of QuickCheck (QC) use Hedgehog? There's even a way to use QC generators in Hedgehog and vice versa.

@jystic shows how a bug slipped in bos/text library by using QC default arbitraries. It's on this video at 11:50 (up to 13:09).

The solution would be to use Hedgehog's Range type, which offers fine-grained control over the scope and shrinking of generated values.

Being able to support both QC and Hedgehog may actually be very interesting.

use ag -- the silver searcher -- in TravisCI

Cause we have some lint checks like this that require it.

Be SHELL agnostic

Right now bash is required only to be able to restore the set of environment variables on each substep.
If at all possible it would be much faster to just start a non-login shell (overridable through YML shell: /bin/sh) with -s and pipe commands in and out of that.
See http://technosophos.com/2014/07/11/start-an-interactive-shell-from-within-go.html

todo: better handle buffering/marshaling

Have a look at: https://medium.com/stupid-gopher-tricks/streaming-data-in-go-without-buffering-3285ddd2a1e5

We should use buffered marshaling (should be easier on memory on large payloads) instead of things like

marshaled, err := json.Marshal(data)
if err != nil {
	...
}

todo: generate bins for more platforms

Have a look at rclone/rclone@d5b16c8

use pkgr

https://packager.io/pricing free for OSS.
https://github.com/scaleway/scaleway-cli/blob/master/.pkgr.yml

push a container as an alternative installation option

See https://github.com/bcicen/ctop/blob/01a305d32602db395bb99abd4de0b6f4b5f086f1/Dockerfile