kapicorp / kapitan

Generic templated configuration management for Kubernetes, Terraform and other things

Kapitan: advanced configuration management tool

Kapitan aims to be your one-stop tool to help you manage the ever growing complexity of your configurations.

Join the community #kapitan

Official site https://kapitan.dev

Quick Start

Install Kapitan

Docker (recommended)

docker run -t --rm -v $(pwd):/src:delegated kapicorp/kapitan -h

On Linux you can add -u $(id -u) to docker run to preserve file permissions.

Pip

Kapitan needs Python 3.7.

Install Python 3.7

• Linux: sudo apt-get update && sudo apt-get install -y python3.7-dev python3-pip python3-yaml git
• Mac: brew install python3 libyaml git libmagic

Install Kapitan

User ($HOME/.local/lib/python3.7/bin on Linux or $HOME/Library/Python/3.7/bin on macOS):

pip3 install --user --upgrade kapitan

System-wide (not recommended):

sudo pip3 install --upgrade kapitan

Build Kapitan

Docker

To build a docker image for the architecture of your machine, run docker build . -t you-kapitan-image, and to build for a specific platform, add --platform linux/arm64.

To build a multi-platform image (as the CI does), follow the docker multi-platform documentation.

Related projects

• Tesoro - Kubernetes Admission Controller for Kapitan Secrets
• Kapitan Reference - our reference repository to get started with Kapitan

kapitan's Issues

Allow multiple directories for --search-path

Hi :)

As I've already posted in #kapitan on Slack I am trying to use https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus with kapitan.

For that I have fetched the libraries with jsonnet bundler jb

A vendor directory was created containing the libraries.

ls -al vendor
total 0
drwxr-xr-x 11 bud staff 352 Jul 28 06:58 .
drwxr-xr-x 12 bud staff 384 Jul 28 07:04 ..
drwxr-xr-x  2 bud staff  64 Jul 28 06:58 .tmp
drwxr-xr-x  4 bud staff 128 Jul 28 06:58 etcd-mixin
drwxr-xr-x  5 bud staff 160 Jul 28 06:57 grafana
drwxr-xr-x  3 bud staff  96 Jul 28 06:57 grafana-builder
drwxr-xr-x 15 bud staff 480 Jul 28 06:57 grafonnet
drwx------ 20 bud staff 640 Jul 28 06:58 ksonnet
drwxr-xr-x 18 bud staff 576 Jul 28 06:56 kube-prometheus
drwx------ 15 bud staff 480 Jul 28 06:56 kubernetes-mixin
drwxr-xr-x  9 bud staff 288 Jul 28 06:58 prometheus-operator

I have defined a component monitoring that imports kapitan and kube-prometheus from vendor directory

$ cat components/monitoring/main.jsonnet
local kap = import "lib/kapitan.libjsonnet";
local inventory = kap.inventory();

local kp = (import 'vendor/kube-prometheus/kube-prometheus.libsonnet') + {
  _config+:: {
    namespace: inventory.parameters.namespace,
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

The inventory class for the above component.

$ cat inventory/classes/component/monitoring.yml
parameters:
  namespace: "monitoring"
  kapitan:
    vars:
      target: ${target}
      namespace: ${namespace}
    compile:
    - output_path: kubernetes/manifests/monitoring
      output_type: yaml
      input_paths:
        - components/monitoring/main.jsonnet
      input_type: jsonnet

And the target.

$ cat inventory/targets/dev.yml
classes:
  - component.monitoring

parameters:
  a: b
  target: dev

Compile without -J vendor fails with

$  kapitan compile
Jsonnet error: failed to compile /Users/bud/code/valibud/kure/components/monitoring/main.jsonnet:
 RUNTIME ERROR: couldn't open import "grafana/grafana.libsonnet": [Errno 2] No such file or directory: '/Users/bud/code/valibud/kure/vendor/kube-prometheus/grafana/grafana.libsonnet'
    /Users/bud/code/valibud/kure/vendor/kube-prometheus/kube-prometheus.libsonnet:4:2-36    thunk <import>
    /Users/bud/code/valibud/kure/components/monitoring/main.jsonnet:4:13-70    thunk <kp>
    /Users/bud/code/valibud/kure/components/monitoring/main.jsonnet:10:81-83    thunk <o>
    std.jsonnet:1153:24
    std.jsonnet:1153:5-33    function <anonymous>
    /Users/bud/code/valibud/kure/components/monitoring/main.jsonnet:10:64-99    thunk <a>
    /Users/bud/code/valibud/kure/components/monitoring/main.jsonnet:10:1-101    function <anonymous>
    /Users/bud/code/valibud/kure/components/monitoring/main.jsonnet:10:1-101

Compile error: failed to compile target: dev

Import statements from all the libraries are relative.

Then I've ran with -J vendor expecting it to work based on my understanding of the -J flag.

$ kapitan compile -J vendor
Compile error: input_path for target: dev not found in search_path: components/monitoring/main.jsonnet

Copying all the libraries from vendor to repository root alongside components and inventory
and running compile, as expected, works.

$ kapitan compile
Compiled dev (55.54s)

Thank you @ramaro for your patient support on Slack.

GPG encrypt passes binary blob to secrets_file['data'] instead of base64

kapitan secrets --write/--update/--update-targets ... does not base64 the encrypted data from secrets, instead it writes the encrypted binary blob directly to yaml:

data: !!binary |
...

Print target name on compile error

Most compilation errors don't print the target name, making it harder to debug what the issue is.

kapitan init

kapitan init command to create an empty/example project skeleton.

Simpler Example

It may be useful to do a simpler example for kaptian, something like a helloworld/hostname example that builds for multiple namespaces with a cluster.

Validate compile operation using jsonschema lib

Have kapitan use the jsonschema lib (if you enable it in your inventory).
jsonschema lib will validate the compiled file output structure for k8s/terraform/whatever other yaml resources.
links:
https://json-schema.org/implementations.html
https://pypi.org/project/jsonschema/

Secret files containing multiple values / allowing GPG encrypted target files even?

When many secrets are used, gpg will be running / invoked over and over again to decrypt each an every file containing just a little string.
It is certainly inefficient to run GPG for every such little string and it clutters the classes files to add gpg and the secret path for each time.

1. What about allowing referencing sub-variables in secret files like

  mysql:
    storage: 10G
    storage_class: standard
    image: mysql:latest
    users:
      root:
         password: ${gpg:allmysecretsinonefile:mysql:password}

-- This would allow decrypting and then caching a whole file containing multiple values once and allow for much faster compiles when many secrets are used.

2. Much easier / less error-prone:

Allow using whole gpg encrypted target files which are simply decrypted prior to rendering the templates.
this could be as easy as adding ".gpg" to your target yaml file by convention. Inside the classes those variables should be addressable just like any other variable, making the fact that this is actually sourced from an encrypted file transparent.

create kapitan binary (using pyinstaller or similar)

Improve debugging

When jsonnet fails to compile, the output does not easily allow to understand which target caused the jsonnet to fail.

Compiled working-target1 (1.86s)
Compiled working-target2 (1.86s)
Jsonnet error: failed to compile repository/components/cloudsql-proxy/main.jsonnet:
 RUNTIME ERROR: field does not exist: cloudsql
        repository/components/cloudsql-proxy/main.jsonnet:13:26-49       object <anonymous>

One possible solution is to write the target being compiled before attempting the compilation. Output could look like this

Compiling working-target1  (OK 1.86s)
Compiling working-target2  (OK 1.86s)
Compiling failing-target3
Jsonnet error: failed to compile repository/components/cloudsql-proxy/main.jsonnet:
 RUNTIME ERROR: field does not exist: cloudsql
        repository/components/cloudsql-proxy/main.jsonnet:13:26-49       object <anonymous>

Create `|randomstr|sha256` function in secrets backend and support pipes

Similar to the current randomstrb64, but instead sha256 the random string, with an optional parameter as the salt.

Would be great to allow pipes like |randomstr|base64 or randomstr|sha256:salt to make the functions much more generic.

Improve error messages

Currently if something is undefined, the error message is an unhelpful Python stack trace with no indication of where the error actually was. Ideally it would be something more human readable.

GPG fails to write secret when key doesn't have expiry date

When expiry key is set infinity, Kapitan fails to write a secret with GPG. Stack trace below:

2018-11-14 21:11:01,398 kapitan.refs.secrets.gpg DEBUG    Key for recipient: XXX with fingerprint: YYY is expired, skipping
Traceback (most recent call last):
  File "/Library/Python/3.6/bin/kapitan", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/3.6/lib/python/site-packages/kapitan/cli.py", line 279, in main
    secret_write(args, ref_controller)
  File "/Library/Python/3.6/lib/python/site-packages/kapitan/cli.py", line 312, in secret_write
    secret_obj = GPGSecret(data, recipients, encode_base64=args.base64)
  File "/Library/Python/3.6/lib/python/site-packages/kapitan/refs/secrets/gpg.py", line 59, in __init__
    fingerprints = lookup_fingerprints(recipients)
  File "/Library/Python/3.6/lib/python/site-packages/kapitan/refs/secrets/gpg.py", line 188, in lookup_fingerprints
    lookedup_fingerprint = fingerprint_non_expired(name)
  File "/Library/Python/3.6/lib/python/site-packages/kapitan/refs/secrets/gpg.py", line 212, in fingerprint_non_expired
    raise GPGError("Could not find valid key for recipient: %s" % recipient_name)
kapitan.refs.secrets.gpg.GPGError: Could not find valid key for recipient: XXX

kapitan source tarball on PyPI doesn't include requirements.txt

$ pip install --user kapitan
Collecting kapitan
  Downloading https://files.pythonhosted.org/packages/ac/01/613694f1f2eb99f73a6ea9ecc4e4d04492dc918048b438410c2faec88461/kapitan-0.21.0.tar.gz (79kB)
    100% |████████████████████████████████| 81kB 617kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-exgfpmrk/kapitan/setup.py", line 55, in <module>
        pkgs, new_links = install_deps()
      File "/tmp/pip-install-exgfpmrk/kapitan/setup.py", line 41, in install_deps
        with open('requirements.txt', 'r') as f:
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'

Add support for resolving container image digests by tag name

Docker tags are not immutable and the same tag can be assigned to the same image over and over.
However they are easier to deal with than, for instance, sha digests.

In an example scenario, say that you have a deployment with 4 replicas, all referencing "my_image:latest".
If you scale up the deployment, or delete a pod say, a month later, it is not possible to guarantee that the exact same version is running.

The idea is to be able to specify an image by tag (i.e. my_image:latest) but a compile time, turn that tag into an immutable reference (sha)

This way it will be easier to have the guarantee of running the same image consistently for all pods.

Create kubelib in python for Kadet

Based on kube.libjsonnet but using the new kadet/python input type create a new library that provides similar functionality for kubernetes.

FR - Allow filtering of variables on kapitan inventory

e.g. kapitan inventory -t minikube-es -p parameters.secrets would only show parameters.secrets variable struct.

Generate kapitan secrets with multiple encodings

Let's assume the following use case:

A kapitan user wants to encrypt a password as a secret.
This password has to be used in:

1. A secret.jsonnet file where it should be Base64'ed so that K8s loads it as a secret properly
2. A jinja2 file where it should be displayed plaintext for various reasons.
3. Other references in jsonnet files might also exist but they should not be Base64'ed.

Currently there is no nice way to do this with Kapitan.
Either you create a second inventory item manually and base64 it (in which case you don't have auto-generation with |randomstr) or you do |base64 in the inventory item but you have to base64 decode it everywhere you use it after kapitan secrets --reveal happens (which means possibly way more code for developers or use cases where this is quite hard).

---

Suggestion:

• Create a new inventory filter like password: ?{/secrets/mypassword|randomstr|encode:b64:sha256}. This generates multiple secrets files which are encoded with the corresponding algorithm as well as plaintext.
• As a Kapitan user then, you should be able to define an inventory item like `password_b64: ?{/secrets/mypassword} and it will reference the corresponding encoded secret from above.
• Then in your jsonnet/jinja2 you just reference parameters....password_b64 or parameters....password anywhere you want without having to apply other filters during reveal time.

Improve error messages for missing mandatory kapitan inventory keys

For an invalid target like the one below:

parameters:
  target_name: mynamespace
  namespace: ${target_name}
  kapitan:
    compile:
        - output_path: pre-deploy
          input_type: jsonnet
          output_type: yaml
          input_paths:
            - components/namespace/main.jsonnet

Compilation fails with a KeyError, not suggesting that vars: needs to be set:

Traceback (most recent call last):
  File "/kapitan/kapitan/targets.py", line 84, in compile_targets
    [p.get() for p in pool.imap_unordered(worker, target_objs) if p]
  File "/kapitan/kapitan/targets.py", line 84, in <listcomp>
    [p.get() for p in pool.imap_unordered(worker, target_objs) if p]
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
KeyError: 'vars'

A valid target needs at least vars:target, as below:

parameters:
  target_name: mynamespace
  namespace: ${target_name}
  kapitan:
    vars:
      target: ${target_name}
    compile:
        - output_path: pre-deploy
          input_type: jsonnet
          output_type: yaml
          input_paths:
            - components/namespace/main.jsonnet

Create "cleartext" backend for secrets

Secrets with GPG and especially automatic generation of secrets is very powerful.

However for some low-security setups (i.e. minikube or example folder) a user is required to have configured GPG and have the right key to decrypt secrets.

This forces the user to 2 choices:

1. reveal the secret in plaintext by overriding the target file
2. store a "shared" gpg key in the repository to allow for low security situations

The first option will force users to have alternative setups for demo/personal and prod environments
The second option will still force the user to setup gpg, and make more difficult to understand if an encrypted key is "unsecure" and auditing

I would like to encourage a healthy use of secrets that does not force users to diverge from the standard way for convenience.

My proposal is to create a new backend as an alternative to the gpg backend: a plaintext secret.

This enfoces a similar workflow but proving the benefit of a gpg-less setup.

Also, explicit use of this backend are easier to audit

example:

?{gpg:targets/prod/mysql/my_user_password}

?{plain:targets/example/mysql/my_user_password}

ultimately this can easily be templated with

?{${encryption_backend}:targets/${target_name}/mysql/my_user_password}

searchvar only works with leaf variables

e.g. kapitan searchvar parameters.mysql.user works, but kapitan searchvar parameters.mysql doesn't return anything.

Failed to compare release-candidate versions

cat .kapitan
version: 0.20.0-rc.0

kapitan compile

Traceback (most recent call last):
  File "/Users/chifor/Library/Python/3.7/bin/kapitan", line 11, in <module>
    sys.exit(main())
  File "/Users/chifor/Library/Python/3.7/lib/python/site-packages/kapitan/cli.py", line 228, in main
    check_version()
  File "/Users/chifor/Library/Python/3.7/lib/python/site-packages/kapitan/utils.py", line 381, in check_version
    if kapitan_config and kapitan_config["version"] and StrictVersion(kapitan_config["version"]) > StrictVersion(VERSION):
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/version.py", line 40, in __init__
    self.parse(vstring)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/version.py", line 137, in parse
    raise ValueError("invalid version number '%s'" % vstring)
ValueError: invalid version number '0.20.0-rc.0'

Create `|rsa` function to automatically generate private keys

The function should accept parameters like key length

?{gpg:targets/${target_name}/private_key|rsa:4096}

replace jsonnet with go-jsonnet

Validate inventory yaml files before compiling

Access global inventory from Jinja

We already allow for jsonnet to access the full inventory (cross target), and we should allow jinja templates to access that as well.

This is helpful to create docs and canned scripts that need to have a global view across all targets.

Add AWS KMS support

As a secrets encryption backend

Secrets write should remove duplicate recipients

Basic linter functionality

kapitan lint should check for things like:

• Variable definition in target/class but no usage or reference.
• Duplicate variables in the same target.
• Re-definition of the same variable with the same value between targets and classes.

Incomplete ref/secret tag hangs compile

If an incomplete/invalid secret tag is referenced from the inventory, compile will hang indefinitely:

Given the inventory (note the missing } for my_secret):

parameters:
  my_secret: ?{gpg:something_secret
  other_stuff: yoda

and a jsonnet main.jsonnet file:

local kap = import "lib/kapitan.libjsonnet";
local inv = kap.inventory();
{
   my_secret:  inv.parameters.my_secret,
}

Set kapitan default flags in a config file (.kapitan)

For example if we want to compile every time with --indent 4, we could make use of the Makefile, but would be nicer to allow configuration of flag defaults in a config file such as .kapitan, where we already store the latest kapitan version used for compilation.

.kapitan can become something like:

version: 0.16.3
compile:
  indent: 4

would be equivalent with running kapitan compile --indent 4

Search for multiple class & secret paths

Feature request as discussed in #kapitan.

Use Case

We have multiple teams (dev/ops/devops) and one infrastructure team. All of them have dedicated/own Git repositories and CI/CD pipelines to rollout applications on Kubernetes with manifests compile by kapitan. To prevent redundant data in classes we (infrastructure team) want to provide a central CMDB repository that contains shared (reclass) (inventory) classes and secrets.
Git-subtree is used to maintain a local copy of the global/shared CMDB Git repository.

This leads to classes/secrets that are distributed in multiple directories:

$ tree
.
`-- deploy/
    |-- cmdb/
    |   |-- classes/
    |   |   `-- kubernetes/
    |   |       |-- clusters.yml
    |   |       `-- namespaces.yml
    |   `-- secrets/
    |       |-- deploy-key
    |       `-- registry-readonly-key
    `-- kapitan/
        |-- inventory/
        |   |-- classes/
        |   |   `-- abc.yml
        |   `-- targets/
        |       `-- mytarget.yml
        `-- secrets/
            `-- eu-central-1/
                `-- monitoring/
                    `-- oidc_key

Feature Request

We need to tell kapitan to search for multiple classes directories and secrets directories.

Example:

$ cd deploy/kapitan/
$ kapitan compile \
  --class-search-path "classes/:../cmdb/classes/" \
  --secrets-search-path "secrets/:../cmdb/secrets/"

This might obsoletes/collides --search-path (?) and --secrets-path.

jinja2 yaml pretty printer filter

I would like to create a INVENTORY.md to allow for looking at the inventory without running kapitan inventory, so that users can understand what values are set to without running kapitan.

The INVENTORY.md doc contains the following

{{inventory.classes | pprint}}


{{inventory.parameters | pprint }}

This would look much better if there was a proper toyaml jinja filter.

Allow configurable indentation for compiled YAML/JSON files

https://github.com/deepmind/kapitan/blob/d7ba2af70c954f2f8687286bf3a057f03f2adda4/kapitan/targets.py#L347

Intermittent IO issues with jinja2 templates when using docker image to compile on macOS ([Errno 9] Bad file descriptor)

Compilation using the docker image on macOS fails randomly when trying to render jinja2 templates.
Kapitan v0.13.0, macOS 10.13.3, Docker version 17.12.0-ce, build c97c6d6.

...
Jsonnet jinja2 failed to render <path>/file.yml.j2: [Errno 9] Bad file descriptor: '<path>/file.yml.j2'
Jsonnet error: failed to compile <path>/main.jsonnet:
 RUNTIME ERROR: [Errno 9] Bad file descriptor: '<path>/file.yml.j2'
	/kapitan/kapitan/lib/kapitan.libjsonnet:2:32-89	function <anonymous>
	<path>/config.jsonnet:13:17-87	object <anonymous>
	<path>/kube.libjsonnet:208:37-49	
	<path>/kube.libjsonnet:208:28-50	thunk <a>
	<path>/kube.libjsonnet:208:28-62	function <anonymous>
	<path>/kube.libjsonnet:208:28-62	thunk <nonstrings>
	<path>/kube.libjsonnet:209:23-33	
	<path>/kube.libjsonnet:209:12-34	thunk <a>
	<path>/kube.libjsonnet:209:12-39	function <anonymous>
	<path>/kube.libjsonnet:209:12-39	thunk <object_assert>
	<path>/kube.libjsonnet:199:24-42	object <anonymous>
	<path>/kube.libjsonnet:199:16-49	object <anonymous>
	<path>/kube.libjsonnet:24:24-55	thunk <array_element>
	<path>/kube.libjsonnet:170:14-45	object <anonymous>
	<path>/deployment.jsonnet:(64:16)-(86:6)	object <anonymous>
	<path>/deployment.jsonnet:(62:10)-(87:4)	object <anonymous>
	<path>/deployment.jsonnet:(53:58)-(88:2)	object <anonymous>
	<path>/main.jsonnet:7:20-30	object <anonymous>
	During manifestation
...

Dependency management

We could of course use plain git submodules, but what if we only need only one file or folder from an external git repo, http url or even helm chart/templates ?

I would like to introduce the concept of modular dependencies in kapitan, and it would look something like this:

parameters:
  kapitan:
    compile:
    ...

    dependencies:
    - type: git
      output_path: lib
      source: [email protected]:grafana/grafonnet-lib.git
      subdir: grafonnet
      version: 0.1
    - type: http
      output_path: lib
      source: https://raw.githubusercontent.com/bitnami-labs/kube-libsonnet/master/kube.libsonnet
      version: 0.1

In this scenario, kapitan compile will pull github.com/grafana/grafonnet-lib/grafonnet/* and github.com/bitnami-labs/kube-libsonnet/master/kube.libsonnet into lib at the root. Dependencies will be cached and only re-pulled when the version changes, keeps things simple and we don't need to implement an extra kapitan subcommand.

Users have full control of what the source can be (jsonnet, libsonnet, jinja2 templates, k8s manifests etc.) and where it's placed inside the project (output_path), can be per target, class or global.

In terms of modularity, each type can be defined as a separate python module with the same name, as in kapitan/kapitan/dependencies/git.py will control the functionality of type: git; in the example above, subdir is a git module specific variable, we can also add branch, release etc.

@ademariag @gburiola @ramaro What do you think?

GPG fails to load pinentry [RESOLVED]

For posterity. Would be good to have a better/more helpful error message and a troubleshooting section.

Problem [RESOLVED]

I have experienced this on my linux desktop
Basically GPG is unable to call the pinentry-curses (or -tty) to read the password on the command line.

./compiled/my_target/vault/provision.sh
potential problem: ERROR: pkdecrypt_failed 83918950
Traceback (most recent call last):
  File "/usr/local/google/home/ademaria/kapitan/bin/kapitan", line 11, in <module>
    load_entry_point('kapitan==0.14.0', 'console_scripts', 'kapitan')()
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/site-packages/kapitan/cli.py", line 230, in main
    secret_gpg_reveal_raw(gpg_obj, args.secrets_path, None, verify=(not args.no_verify))
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/site-packages/kapitan/secrets.py", line 249, in secret_gpg_reveal_raw
    revealed = re.sub(SECRET_TOKEN_TAG_PATTERN, _reveal_gpg_replace, line)
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/re.py", line 191, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/site-packages/kapitan/secrets.py", line 220, in reveal_gpg_replace
    return secret_gpg_read(gpg_obj, secrets_path, token, **kwargs)
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/site-packages/kapitan/secrets.py", line 83, in secret_gpg_read
    raise GPGError(dec.status)
kapitan.secrets.GPGError: decryption failed

The solution is

export GPG_TTY=`tty`

other helpful commands

sudo apt-get install pinentry-curses
sudo update-alternatives --config pinentry

Debug it with

eval $(gpg-agent --daemon --debug-level 9 --pinentry-program /usr/bin/pinentry-curses | tee /dev/tty)

secrets validation and update only looks at targets

If we have secrets/common for example, kapitan secrets --validate-targets or kapitan secrets --update-targets will ignore it and only look at secrets/targets/.

Secret is wiped if writing aborted

When trying to update a secret e.g. secrets/targets/my-target/secret_sauce, if a GPGError occurs, the content of secrets/targets/my-target/secret_sauce is emptied.

secrets --update crash when -t is used

Kapitan should warn if kapitan.secrets configuration is missing

MacBook-Pro:empty_secret_issue rjmco$ echo -n 'important' | kapitan secrets -w gpg:important -t dummy -f -
Traceback (most recent call last):
  File "/Users/rjmco/.pyenv/versions/3.7.0/bin/kapitan", line 11, in <module>
    load_entry_point('kapitan==0.18.2', 'console_scripts', 'kapitan')()
  File "/Users/rjmco/.pyenv/versions/3.7.0/lib/python3.7/site-packages/kapitan/cli.py", line 279, in main
    secret_write(args, ref_controller)
  File "/Users/rjmco/.pyenv/versions/3.7.0/lib/python3.7/site-packages/kapitan/cli.py", line 311, in secret_write
    recipients = inv['nodes'][args.target_name]['parameters']['kapitan']['secrets']['recipients']
KeyError: 'secrets'

Support multiple markup languages in inventory

The inventory only support YAML right now, but it would be cool to add support for TOML, JSON and XML (would have to serialize back to YAML to feed into reclass).

Pull secrets from Hashicorp Vault

As a Vault user, I would like Kapitan to be able to pull information from Hashicorp Vault so that I can have a single authority on secrets in my organization.

Acceptance Criteria:

• Supports both v1 and v2 KV backends including the ability to specify a version of a value where appropriate.
• Inherits or detects Vault information from the environment/other mechanisms (i.e. reads the token from ~/.vault-token, uses VAULT_ADDR, etc.)
• Has the ability to explicitly specify Vault information to override inherited or detected configurations.
• Supports basic CRUD operations on an arbitrary secret backend
• Supports, at minimum, the userpass, token, and cert authentication mechanisms.

compiled directory not created for jinja target alone

Kapitan is failing to create directories for Jinja targets.

$ kapitan compile -v -t global_tools
2018-04-06 23:54:41,982 gnupg        DEBUG    231982: gpg --status-fd 2 --no-tty --debug ipc --fixed-list-mode --batch --with-colons --version
2018-04-06 23:54:41,983 gnupg        DEBUG    stderr reader: <Thread(Thread-1, initial daemon)>
2018-04-06 23:54:41,983 gnupg        DEBUG    stdout reader: <Thread(Thread-2, initial daemon)>
2018-04-06 23:54:41,985 gnupg        DEBUG    gpg: reading options from '/usr/local/google/home/ademaria/.gnupg/gpg.conf'
2018-04-06 23:54:41,986 gnupg        DEBUG    chunk: b'gpg (GnuPG) 2.2.2\nlibgcrypt 1.8.1\nCopyright (C) 2017 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to t'
2018-04-06 23:54:41,995 kapitan.resources DEBUG    Using reclass inventory config at: ./inventory/reclass-config.yml
2018-04-06 23:54:43,258 kapitan.targets DEBUG    load_target_inventory: found valid kapitan target global_tools
2018-04-06 23:54:43,263 kapitan.resources DEBUG    Using reclass inventory config at: /usr/local/google/home/ademaria/projects/health-templates/inventory/reclass-config.yml
2018-04-06 23:54:44,586 kapitan.utils DEBUG    render_jinja2 rendering /usr/local/google/home/ademaria/projects/health-templates/scripts/global_tools/create_secrets.sh
2018-04-06 23:54:44,594 kapitan.targets DEBUG    Wrote /tmp/tmpe0lyg75a.kapitan/global_tools/scripts/create_secrets.sh with mode 0750
2018-04-06 23:54:44,595 kapitan.targets INFO     Compiled global_tools (1.34s)
2018-04-06 23:54:44,645 kapitan.targets DEBUG    Compile pool terminated
2018-04-06 23:54:44,645 kapitan.targets ERROR
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/site-packages/kapitan/targets.py", line 76, in compile_targets
    shutil.rmtree(compile_path_target)
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/shutil.py", line 471, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/local/google/home/ademaria/kapitan/lib/python3.6/shutil.py", line 469, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: './compiled/global_tools'
2018-04-06 23:54:44,648 kapitan.targets DEBUG    Removed /tmp/tmpe0lyg75a.kapitan

$ mkdir -p compiled/global_tools
$ kapitan compile -v -t global_tools
2018-04-06 23:55:26,086 gnupg        DEBUG    232115: gpg --status-fd 2 --no-tty --debug ipc --fixed-list-mode --batch --with-colons --version
2018-04-06 23:55:26,087 gnupg        DEBUG    stderr reader: <Thread(Thread-1, initial daemon)>
2018-04-06 23:55:26,087 gnupg        DEBUG    stdout reader: <Thread(Thread-2, initial daemon)>
2018-04-06 23:55:26,091 gnupg        DEBUG    gpg: reading options from '/usr/local/google/home/ademaria/.gnupg/gpg.conf'
2018-04-06 23:55:26,091 gnupg        DEBUG    chunk: b'gpg (GnuPG) 2.2.2\nlibgcrypt 1.8.1\nCopyright (C) 2017 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to t'
2018-04-06 23:55:26,106 kapitan.resources DEBUG    Using reclass inventory config at: ./inventory/reclass-config.yml
2018-04-06 23:55:27,380 kapitan.targets DEBUG    load_target_inventory: found valid kapitan target global_tools
2018-04-06 23:55:27,385 kapitan.resources DEBUG    Using reclass inventory config at: /usr/local/google/home/ademaria/projects/health-templates/inventory/reclass-config.yml
2018-04-06 23:55:28,754 kapitan.utils DEBUG    render_jinja2 rendering /usr/local/google/home/ademaria/projects/health-templates/scripts/global_tools/create_secrets.sh
2018-04-06 23:55:28,764 kapitan.targets DEBUG    Wrote /tmp/tmplnho7_0j.kapitan/global_tools/scripts/create_secrets.sh with mode 0750
2018-04-06 23:55:28,764 kapitan.targets INFO     Compiled global_tools (1.38s)
2018-04-06 23:55:28,765 kapitan.targets DEBUG    Copied /tmp/tmplnho7_0j.kapitan/global_tools into ./compiled/global_tools
2018-04-06 23:55:28,765 kapitan.targets DEBUG    Removed /tmp/tmplnho7_0j.kapitan

How to generate duplicated keys?

I want to generate google_iam_policy for terraform, here is a sample code.

data "google_iam_policy" "admin" {
  binding {
    role = "roles/compute.instanceAdmin"

    members = [
      "serviceAccount:[email protected]",
    ]
  }

  binding {
    role = "roles/storage.objectViewer"

    members = [
      "user:[email protected]",
    ]
  }
}

How to generate two binding sections with different values?
Thank you.

save version of kapitan used to compile targets

This is similar to how terraform works.

tf saves the version of the terraform binary used to produce the terraform.tfstate file and fails to run if you try to use an older version:

$ terraform plan

Error:
Terraform doesn't allow running any operations against a state
that was written by a future Terraform version. The state is
reporting it is written by Terraform '0.11.5'

Please run at least that version of Terraform to continue.

we should have a similar feature on kapitan

importing helm charts support

Load inventory only once

Calling reclass and generating the inventory is an expensive process (~2 seconds). We do save the result using the memoise decorator, however we still run it once for each thread.

We should load the inventory once before we create multiple threads and pass the result to each thread.

searchvar doesn't respect variable chain

For example, kapitan searchvar parameters.elasticsearch.replicas will also match parameters.mysql.replicas, as it only seems to look at the latest value.

Processing continues while kapitan waits for gpg unlock key

Currently when running kapitan compile, if gpg password is needed, the compilation for other targets will continue on the background making it difficult for a user to understand that entering the password is needed.

Eventually kapitan will error out with "encryption incomplete"

We should have a strategy to check if keyring is open/closed before compiling targets, or at least make it easier to enter the password.

Search for plaintext secrets in inventory

Maybe check the keys containing

*key*  *pass*  *token*  *secret*  *pin*  *security*  *crypto*

and check the value entropy.

We should ignore gpg references.

Can be integrated as part of kapitan lint