open-chaos / experiment-catalog

A public and open source chaos engineering experiment catalog.

Welcome to the Open Chaos Experiment Catalog!

This is a free, public and community-contributed catalog of chaos engineering experiments.

Catalog Layout and Experiment Format

The format of the catalog is relatively flexible, being a resource that will index many different types of experiments and even potentially different experiment-executing tools and systems. The current convention, where applicable, for a new experiment to be added to the catalog are:

1. Every experiment should reside in its own directory below a parent directory that is used to group experiments by target platform context, i.e. Kubernetes, Azure, AWS etc.

2. Top level tags for an experiment are recommended to be platform and service, with a corresponding descriptor after a :.

   Platform specifies the overall platform being targetted or manipulated by the experiment, for example platform:kubernetes. service, where applicable, specifies any sub-service of the platform being employed, such as service:node. Any combination of platform:* and service:* tags may be used for a single experiment so that it is appropriately indexed for catalog searching.

3. A README.md is recommended, following the general template form as show by the file-must-not-disappear experiment sample.

4. Optionally Yaml as well as JSON can be provided as the preferred format of an experiment, according to an author's tastes. However it is advised to submit both formats if possible.

Contributing to this Experiment Catalog

If you wish to contribute to this experiment catalog, you are more than welcome to do so. Please, fork this project, make your changes, and submit a PR for review.

We recommend that any experiments written to use the Chaos Toolkit should use the toolkit's chaos validate command to test the syntax of an experiment before it is submitted for consideration to the catalog.

This project also requires all contributors must sign a Developer Certificate of Origin on each commit they would like to merge into the master branch of the repository. Please, make sure you can abide by the rules of the DCO before submitting a PR.

experiment-catalog's Issues

create aws ec2 stop instance by id experiment

create experiment entry for terminate random ec2 instance by zone

Add Game Day style automated experiment

Bring over initial experiment templates from community playground

Parameterise the simple file-presence-check experiment

The current experiment here is not parameterised. This will involve also updating the execution documentation.

https://github.com/open-chaos/experiment-catalog/tree/master/file-must-not-disappear

Introduce an experiment validation build

I would like us to consider adding a chaos validate build to this repo. This is where each experiment is run through chaos validate, with corresponding dummy env variables where necessary, and where the official docker image of Chaos Toolkit is employed.

This would show:

• How the chaos toolkit can be executed from a build system, perhaps Travis.
• How chaos validate is especially useful when building catalogs of experiments in an organisation.

Does this sound like a good idea?

Create Kubernetes node deleted experiment

Add explanation of experiment documentation and current tag conventions for experiments

Tag conventions should include the platform:service style, as well as an explanation of contribution model.

Update docker instructions to use chaostoolkit official image

Both kill-pods and drain-node need to be updated.

Create Kubernetes kill micorservice experiment

Create a Kubernetes kill microservice experiment

stop random ec2 instance in named availability zone (AZ)

Create aws terminate random asg instance experiment

Should we add a no-warranty declaration to this repo?

It seems it might be useful to have some sort of warranty statement that says that any experiments here are used at the individuals own risk and no warranty or indemnity is made?

Restructure the top level of the catalog for aws experiments

Following discussion, it was decided to organize the catalog using top-level directories to show what platform the experiment would run on. Note on the naming ensure underscores are used s a name separator as opposed to hyphens. so this would use directories such as Kubernetes, aws, azure etc.

Fix http-based experiment execution instructions for docker option

Bring over disruption budget experiment and make a template

create a catalog experiment to stop an aws ec2 instance by a filter

Axon Server Disk Full experiment

OH: This can really happen

Bring across first Chaos Monkey for Spring Boot experiments into the catalog

Original sources here: https://github.com/chaosiq/demos/tree/master/spring-boot-chaos-monkey

Error when running Kubernetes + Kill Micorservice Experiment through HTTP

[2019-09-24 10:18:31 ERROR] Failed to fetch the experiment: 404: Not Found

Create Kubernetes Pod remove service endpoint experiment

AxonIQ GRPC call skipped between nodes in a cluster

Describe content architecture and formats for experiment contributors in README

Add validation sections to experiments

I suggest we add chaos validate instructions alongside chaos run for each of the experiments, to aid in their further development. Sections such as:

Developing this experiment further

Validating the experiment from the catalog using HTTP

(chaostk) $ export APPLICATION_ENTRYPOINT_URL=http://192.168.99.100:32474; \
            export SERVICE_NAME=my-service; \
                   chaos validate https://raw.githubusercontent.com/open-chaos/experiment-catalog/master/kubernetes/kill_microservice/kill_microservice.json

Wdyt @gtfisher

This will be reliant on this PR being in a release of the chaos toolkit first: chaostoolkit/chaostoolkit#130

move spring boot experiments to spring boot folder

Update README for current platform-centric directory nesting

Restructure catalog for azure experiments

Following discussion, it was decided to organize the catalog using top-level directories to show what platform the experiment would run on. Note on the naming ensure underscores are used s a name separator as opposed to hyphens. so this would use directories such as Kubernetes, aws, azure etc.

Bring over first version of world's simplest experiment

Docker commands not correct

currently, the docker commands make reference to /Users/russmiles - this should be changed to ~/...

Also the file-must-not-disappear does not map the file to the docker image

Bring across the OpenFaaS demo experiment and docs as an entry in the catalog

See the following for the original source experiment: https://github.com/chaosiq/demos/tree/master/openfaas/experiments/alerts-should-trigger-scaling

Should use a tree structure to organise experiments or should we keep a flat scheme?

Right now, experiments are using a flat structure and all live at the top-level. This will likely get out of hand quickly and doesn't really help quick discovery.

Should we at least have one level based on the target? Obviously, the question then becomes how to deal with experiments that cover two targets?

Axon Server Node Out of Memory

Add restart ec2 instance with filter and availability zone

Add README and runtime instructions to simple experiment as an example.

ec2_restart_instance_by_filter_and_az should be in the aws folder

Add experiment to kill a Spring Boot application

Remove pod_label from drain node and delete node experiments

I don't believe the pod_label is being used in either of these experiments?

Move Kubernetes experiments to sub folder

Add DCO signing process to this repo?

I've mentioned the DCO currently in the contributing section of this repo's README. Is it applicable, I am guessing it probably would be a good practice, especially when it is opened up to the community?

Document and parameterize kill pods and kill nodes experiments

Following the template now set by this experiment: https://github.com/open-chaos/experiment-catalog/tree/master/file-must-not-disappear

Unreachable network between nodes in Axon Cluster

Create Kubernetes Pod scale_micro-service (up or down)

Correct the driver install steps in azure-aks-stop-random-node

The river install steps refer to the Kubernetes driver these should be corrected to refer to the azure driver

Move samples to flat directory structure, indexing based on tags

Error when running Kubernetes / Delete Random Node Experiment

Here is what Chaos Toolkit sends:

[2019-09-24 10:08:15 INFO] Validating the experiment's syntax
[2019-09-24 10:08:15 ERROR] argument 'random' is not part of the function signature in activity 'drain_nodes'

Add ASL 2

Command to run Experiments from HTTP are not easily pasted from the Read Me

Because of the $ on each line:

export APPLICATION_ENTRYPOINT_URL=http://192.168.99.100/myapp; \
$ export POD_LABEL=myapp; \
$ export NODE_NAME=mynode; \
                   chaos run https://raw.githubusercontent.com/open-chaos/experiment-catalog/master/kubernetes/delete_random_node/delete_random_node_experiment.j
son

should be

export APPLICATION_ENTRYPOINT_URL=http://192.168.99.100/myapp; \
export POD_LABEL=myapp; \
export NODE_NAME=mynode; \
                   chaos run https://raw.githubusercontent.com/open-chaos/experiment-catalog/master/kubernetes/delete_random_node/delete_random_node_experiment.j
son

Create Kubernetes cordon node experiment

move file experiment to the local folder

Create Azure AKS experiments

The Azure driver exposes delete_node, stop_node & restart_node actions create catalog experiments for these