dalmena / cstore

CLI for central file or configuration storage and sharing among team members and applications based on project context.

README

The cStore CLI provides commands to push files $ cstore push {{file}} to remote storage. The pushed files are replaced by a catalog file, cstore.yml, that understands folder context, storage location, file encryption, and other details making restoration as simple as $ cstore pull {{file}}.

*.env and *.json files are special file types whose contents can be parameterized with tokens, encrypted, and pushed to storage locations like AWS S3. *.env files can also be pushed to AWS Parameter Store.

TL;DR: cStore encrypts and stores environment configuration remotely using storage like AWS S3 or AWS Parameter Store and restores the configuration anywhere including Docker containers using a catalog file cstore.yml which can be checked into source control or stored with a project without exposing configuration or secrets.

Example

├── project
│   ├── components
│   ├── models
│   ├── main.go
│   ├── Dockerfile 
│   ├── cstore.yml (catalog)
│   └── env
│       └── dev
│       │   └── .env (stored)
│       |   └── .cstore (ghost)
│       |   └── fargate.yml
│       |   └── docker-compose.yml
│       │
│       └── qa
│           └── .env (stored)
│           └── .cstore (ghost)
│           └── fargate.yml
│           └── docker-compose.yml

The cstore.yml catalog and hidden .cstore ghost files take the place of the stored *.env files. The *.env files can be encrypted and stored in AWS S3 or AWS Parameter Store and no longer checked into source control.

When the repository has been cloned or the project shared, running $ cstore pull in the same directory as the cstore.yml catalog file or any of the .cstore ghost replacement files will locate, download and decrypt the *.env files to their respective original paths restoring the project's environment configuration.

How to Use (3 minutes)

Ensure a supported storage location is already set up and available.

Install/Upgrade

mac: $ sudo curl -L -o /usr/local/bin/cstore https://github.com/turnerlabs/cstore/releases/download/v2.6.1-alpha/cstore_darwin_amd64 && sudo chmod +x /usr/local/bin/cstore

linux: $ sudo curl -L -o /usr/local/bin/cstore https://github.com/turnerlabs/cstore/releases/download/v2.6.1-alpha/cstore_linux_386 && sudo chmod +x /usr/local/bin/cstore

win: wget https://github.com/turnerlabs/cstore/releases/download/v2.6.1-alpha/cstore_windows_amd64.exe (add download dir to the PATH environment variable)

The first push creates a catalog file in the same directory that can be checked into source control. Subsequent commands executed in the same directory will use the existing catalog.

By default, cStore will use the AWS credential chain and store configuration in AWS Parameter Store.

Store Files

$ cstore push {{file}} -s aws-paramter # AWS Parameter Store

or

$ cstore push {{file}} -s aws-s3 # AWS S3 Bucket

Multiple files can be discovered and pushed in one command. If needed, replace service with a custom environments folder or . to search all project sub folders.

$ cstore push $(find service -name '*.env')

Restore Files

$ cstore pull {{file}}

Instead of restoring files locally, export environment variables listed inside the files.

$ eval $( cstore pull {{file}} -g terminal-export ) # works for '*.env' files only

Advanced Usage

• Migrate from v1 to v2 (breaking changes)
• Set Up S3 Bucket
• Access Config inside Docker Container
• Access Config inside Lambda Function
• Storing/Injecting Secrets
• Ghost Files (.cstore)
• Tagging Files
• Versioning Files
• Linking Catalogs
• CLI Commands and Flags

Additional Info

• Terms
• Stores
• Vaults
• Terraform State Files

Project

• Goals
• Integration Testing
• Publish Release