goto / entropy

Entropy is a framework to safely and predictably create, change, and improve modern cloud applications and infrastructure using familiar languages, tools, and engineering practices.

Entropy

Entropy is an extensible infrastructure orchestration and application deployment tool. Entropy provides features required for deploying and managing complex applications like resource versioning, config schema versioning, rollbacks dry runs etc.

Key Features

• No Dependency: Written in Go. It compiles into a single binary with no external dependency.
• Extensible: Entropy provides framework to easily write and deploy applications to your choice of cloud
• Runtime: Entropy can run inside VMs or containers with minimal memory footprint.

Refer docs for more on capabilites, internals, etc.

Installation

Install Entropy on macOS, Windows, Linux, OpenBSD, FreeBSD, and on any machine.

Binary (Cross-platform)

Download the appropriate version for your platform from releases page. Once downloaded, the binary can be run from anywhere. You don’t need to install it into a global location. This works well for shared hosts and other systems where you don’t have a privileged account. Ideally, you should install it somewhere in your PATH for easy use. /usr/local/bin is the most probable location.

Homebrew

# Install entropy (requires homebrew installed)
$ brew install goto/tap/entropy

# Check for installed entropy version
$ entropy version

Usage

Entropy typically runs as a service and requires a Postgres to store its state.

Refer entropy.yaml for sample configuration values.

• You can override the configurations by directly editing the entropy.yaml file or by setting environment variables.
• Environment variable name will be uppercased version of the complete path in YAML along with . replaced with _ character. For example, the service.host can be overriden by setting SERVICE_HOST.
• It is also possible to create a copy of the sample configuration file with different name and provide that path to entropy.

$ entropy serve --config ./my_config.yaml

Development

Running locally

# Clone the repo
$ git clone https://github.com/goto/entropy.git

# Build entropy binary file
$ make build

# Start a MongoDB instance
$ docker-compose up

# Run entropy on a recipe file
$ ./dist/entropy serve

Running tests

# Running all unit tests, excluding extractors
$ make test

Contribute

Development of Entropy happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving Entropy.

Read our contributing guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Entropy.

To help you get your feet wet and get you familiar with our contribution process, we have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started.

This project exists thanks to all the contributors.

License

Entropy is Apache 2.0 licensed.

entropy's Issues

Invalid memory metric name issue

Describe the bug
There is a warning when running entropy with dash in the app name

data-entropy-app-b7b7665-qxxl8 app 2023/09/04 06:17:31 Failed to export to Prometheus: cannot register the collector: descriptor Desc{fqName: "data-entropy_process_gc_sys", help: "Bytes of memory in garbage collection metadatas", constLabels: {}, variableLabels: []} is invalid: "data-entropy_process_gc_sys" is not a valid metric name

To Reproduce
Steps to reproduce the behavior:

1. Use dash in the app name
2. Run entropy

Expected behavior
No warning should be thrown and name should be valid

New kubernetes job module

Firehose has a new dlq job feature that fetch dlq data from a dlq storage and send to a sink. Entropy need to support the orchestration of such job. We could make it generic enough to run kubernetes job without firehose or dlq context.

Requirements

• Run as a kubernetes job
  • Input
    • A set of configs and resource config
    • A dlq job image
  • Telegraf running as a sidecar to send metrics
• Kubernetes job is immutable
• Kubernetes job and pods won't be deleted unless it is over TTL (~ 1 month)
• Actions
  • Start
    • Create jobs and pods if not exist
  • Stop
    • Stopping the pods (not removing the pods since the logs need to be there for more debugging)
• Retention on finished jobs (Complete or Failed)
  • configurable ttl (default 1 month)
• Output
  • Need to get the final-state of jobs
    • Option 1: We could perhaps put the resource in PENDING if the status is not completed or failed so entropy would poll until it is COMPLETE. Entropy will the status as COMPLETE if it is either completed or failed.
    • Option 2: Store output when Get Resource (ref easiest)

Orchestration

There are 2 possible approaches of orchestration, using kubernetes client and using helm.
With kubernetes client, entropy would need to create individual resource one by one, configmap, metadata, etc or just creating 1 job resource and inject the configs and everything there. With helm, we will need to create a chart that would be used by this module.

1. Kubernetes client

Pros

• Simpler if we only create 1 job resource and injecting the configs in the resource

Cons

• Won't support versioning

2. Helm

Pros

• Have versioning

Cons

• Need to uninstall helm if we stop the job to leave no traces in the server

Ideas

• We can pass container spec as a config
• Job would have only 2 containers, 1 for the main and 1 for the telegraf

Decision

• Use kubernetes client and inject the configmap there