lrstanley / vault-unseal

auto-unseal utility for Hashicorp Vault

🔗 Table of Contents

• Why
• Solution
• Installation
  • Container Images (ghcr)
  • Source
• Usage
• TODO
• Support & Assistance
• Contributing
• License

❔ Why

HashiCorp Vault provides a few options for auto-unsealing clusters:

• Cloud KMS (AWS, Azure, GCP, and others) (cloud only)
• Hardware Security Modules with PKCS11 (enterprise only)
• Transit Engine via Vault (requires another vault cluster)
• Potentially others

However, depending on your deployment conditions and use-cases of Vault, some of the above may not be feasible (cost, network connectivity, complexity). This may lead you to want to roll your own unseal functionality, however, it's not easy to do in a relatively secure manner.

So, what do we need to solve? We want to auto-unseal a vault cluster, by providing the necessary unseal tokens when we find vault is sealed. We also want to make sure we're sending notifications when this happens, so if vault was unsealed unintentionally (not patching, upgrades, etc), possibly related to crashing or malicious intent, a human can investigate at a later time (not 3am in the morning).

✔️ Solution

The goal for this project is to find the best way to unseal vault in a way that doesn't compromise too much security (a good balance between security and ease of use/uptime), without the requirement of Vault Enterprise, or having to move to a cloud platform.

We do this by running multiple instances of vault-unseal (you could run one on each node in the cluster). Each instance of vault-unseal is given a subset of the unseal tokens. You want to give each node just enough tokens, that when paired with another vault-unseal node, they can work together to unseal the vault. What we want to avoid is giving a single vault-unseal instance enough tokens to unseal (to prevent a compromise leading to enough tokens being exposed that could unseal the vault). Let's use the following example:

Explained further:

• cluster-1 consists of 3 nodes:
  • node-1
  • node-2
  • node-3
• cluster-1 is configured with 5 unseal tokens (tokens A, B, C, D, E), but only 3 are required to unseal a given vault node.
• given there are 3 nodes, 3 tokens being required:
  • vault-unseal on node-1 gets tokens A and B.
  • vault-unseal on node-2 gets tokens B and C.
  • vault-unseal on node-3 gets tokens A and C.

With the above configuration:

• Given each vault-unseal node, each node has two tokens.
• Given the tokens provided to vault-unseal, each token (A, B, and C), there are two instances of that token across nodes in the cluster.
• If node-1 is completely hard-offline, nodes node-2 and node-3 should have all three tokens, so if the other two nodes reboot, as long as vault-unseal starts up on those nodes, vault-unseal will be able to unseal both.
• If node-2 becomes compromised, and the tokens are read from the config file (note: vault-unseal will not start if the permissions on the file aren't 600), this will not be enough tokens to unseal the vault.
• vault-unseal runs as root, with root permissions.

💻 Installation

Check out the releases page for prebuilt versions.

🐳 Container Images (ghcr)

$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.5.0
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:latest
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:master
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.4.1
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.4.0
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.3.0
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.2.4
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.2.3
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.2.2
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.2.1
$ docker run -it --rm ghcr.io/lrstanley/vault-unseal:0.2.0

🧰 Source

Note that you must have Go installed (latest is usually best).

git clone https://github.com/lrstanley/vault-unseal.git && cd vault-unseal
make
./vault-unseal --help

⚙️ Usage

The default configuration path is /etc/vault-unseal.yaml when using deb/rpm. If you are not using these package formats, copy the example config file, example.vault-unseal.yaml, to vault-unseal.yaml. Note, all fields can be provided via environment variables (vault-unseal also supports .env files).

$ ./vault-unseal --help
Usage:
  vault-unseal [OPTIONS]

Application Options:
  -v, --version          Display the version of vault-unseal and exit
  -l, --log-path=PATH    Optional path to log output to
  -c, --config=PATH      Path to configuration file (default: ./vault-unseal.yaml)

Help Options:
  -h, --help             Show this help message

☑️ TODO

• add option to use vault token/another vault instance to obtain keys (e.g. as long the leader is online)?
• memory obfuscating/removing from memory right after unseal?

🙋‍♂️ Support & Assistance

• ❤️ Please review the Code of Conduct for guidelines on ensuring everyone has the best experience interacting with the community.
• 🙋‍♂️ Take a look at the support document on guidelines for tips on how to ask the right questions.
• 🐞 For all features/bugs/issues/questions/etc, head over here.

🤝 Contributing

• ❤️ Please review the Code of Conduct for guidelines on ensuring everyone has the best experience interacting with the community.
• 📋 Please review the contributing doc for submitting issues/a guide on submitting pull requests and helping out.
• 🗝️ For anything security related, please review this repositories security policy.

⚖️ License

MIT License

Copyright (c) 2018 Liam Stanley <[email protected]>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Also located here

vault-unseal's Issues

bug: CheckInterval is duplicated

🌧 Describe the problem

In

vault-unseal/daemon.go

Line 26 in ccabcff

errDelay = (30*time.Second)*time.Duration(errCount) + conf.CheckInterval

to the errDelay the CheckInterval is always be added.
So the errDelay is minimum = CheckInterval.

In

vault-unseal/daemon.go

Line 42 in ccabcff

case <-time.After(conf.CheckInterval + errDelay):

The errDelay is added to the CheckInterval so the time between the the checks min. is 2x CheckInterval.
I would guess this isn't intended.

⛅ Expected behavior

The configured CheckInterval is the time between two consecutive checks.

🔄 Minimal reproduction

Set check_interval in yaml:

grep -w check_interval /etc/vault-unseal.yaml
check_interval: 60s

Check the execution times in log:

Aug 29 16:41:11 argos vault-unseal[98884]: timestamp=2022-08-29T16:41:11.250763784+02:00 level=info message="running checks" addr=https://<omit>:8200 environment= version=0.2.3
Aug 29 16:41:11 argos vault-unseal[98884]: timestamp=2022-08-29T16:41:11.272847516+02:00 level=info message="seal status" addr=https://<omit>:8200 environment= version=0.2.3
Aug 29 16:43:11 argos vault-unseal[98884]: timestamp=2022-08-29T16:43:11.273515519+02:00 level=info message="running checks" addr=https://<omit>:8200 environment= version=0.2.3
Aug 29 16:43:11 argos vault-unseal[98884]: timestamp=2022-08-29T16:43:11.313285866+02:00 level=info message="seal status" addr=https://<omit>:8200 environment= version=0.2.3

💠 Version: vault-unseal

vault-unseal version: 0.2.3 [6d0c99e] (linux, amd64), compiled 2022-08-25T00:22:00Z

🖥 Version: Operating system

linux/other

⚙ Additional context

No response

🤝 Requirements

• I believe the problem I'm facing is a bug, and is not intended behavior. Post here if you're not sure.
• I have confirmed that someone else has not submitted a similar bug report.

seal status/running checks should be debug messages

Can these be changed to debug level messages, to avoid spamming the log with "everything is still working" messages every 15 seconds?

timestamp=2021-04-21T16:28:41.593169277-04:00 level=info message="seal status" addr=https://myserver:8200 environment= version=v0.0.6
timestamp=2021-04-21T16:28:51.602402128-04:00 level=info message="running checks" addr=https://myserver:8200 environment= version=v0.0.6

vault-unseal --version output

🌧 Describe the problem

When I issue this command:

vault-unseal --version

I get vault-unseal version: master [latest] (linux, amd64), compiled -

It used to output a version number as well, like 0.3.0. I am using this for my ansible-playbook to see if I need to upgrade vault-unseal.

⛅ Expected behavior

vault-unseal version: 0.4.0 [latest] (linux, amd64), compiled - or at least something that contains the version.

🔄 Minimal reproduction

vault-unseal --version

💠 Version: vault-unseal

v0.40.0

🖥 Version: Operating system

linux/ubuntu

⚙ Additional context

No response

🤝 Requirements

• I believe the problem I'm facing is a bug, and is not intended behavior. Post here if you're not sure.
• I have confirmed that someone else has not submitted a similar bug report.

Add support for additional notification methods

Options:

• https://github.com/containrrr/shoutrrr (easier to configure, just allow users to pass in url's via cli or env)
• https://github.com/nikoksr/notify

This would likely be a breaking change for existing email support.

bug: vault-unseal won't accept read-only config file

🌧 Describe the problem

A config file set to 0400 results in an error message:

timestamp=2022-08-23T17:21:33.858258175Z level=fatal message="error reading config" environment= error="permissions of \"/etc/vault-unseal.yaml\" are insecure: r--------, please use 0600" version=0.2.2

⛅ Expected behavior

Config file with 0400 permissions is loaded by the program.

🔄 Minimal reproduction

echo '---' > vault-unseal.yaml
chmod 0400 vault-unseal.yaml
vault-unseal -c vault-unseal.yaml

💠 Version: vault-unseal

0.2.2

🖥 Version: Operating system

linux/alpine

⚙ Additional context

No response

🤝 Requirements

• I believe the problem I'm facing is a bug, and is not intended behavior. Post here if you're not sure.
• I have confirmed that someone else has not submitted a similar bug report.

readme contains out of date information

🌧 Describe the problem

If you have not, auto-unseal functionality for on-prem is currently only in enterprise

This is not correct anymore. You actually can do auto-unseal in on-prem Vault OSS. But it does require some configuration, and some resources in a cloud provider like GCP/AWS.

I have set up the open source Vault on-prem in kubernetes with auto-unseal configured to use a key and keyring managed in GCP.

To be clear, I still believe this tool has a purpose. I'm actually considering using it over GCP KMS just so I won't also have to maintain some terraform.

⛅ Expected behavior

This text should be updated. The "why" for this kind of a project would now be something more like, "If you want to maintain a Vault cluster on-prem with auto-unseal functionality without relying on any public cloud KMS assets."

🔄 Minimal reproduction

N/A

💠 Version: vault-unseal

master branch

🖥 Version: Operating system

other

⚙ Additional context

N/A

🤝 Requirements

• I believe the problem I'm facing is a bug, and is not intended behavior. Post here if you're not sure.
• I have confirmed that someone else has not submitted a similar bug report.

Where to Install

Hi,
Not an issue, merely a query.
I'm setting up a cluster - three Consul nodes and two Vault nodes.
Does this vault-unseal script have to be on the Vault nodes or can I put it on the three Consul nodes? All instances are running Ubuntu 20.04.

Thanks,
Richie

Support single Vault node

We have a single Vault instance, but validation for vault_nodes parameter require us to provide at least 3 urls

feature: prevent check delay on startup

✨ Describe the feature you'd like

At the moment the first execution of the seal check is delayed for the CheckInterval because of the time.After statement in the "daemon.go".
So it would be cool if this would be executed immediately after startup of vault-unseal.

🌧 Is your feature request related to a problem?

Not really an problem but an question if this is intentional.
Imagine an high value of CheckInterval (say 300s), so the first check would occur 5m after startup.

🔎 Describe alternatives you've considered

As I'm not an programmer I'm not able to provide a solution for this out of the box.
Maybe the main part of the for loop could be an function so that this could be called once
before the loop starts 🤷 but I'm not sure if this works with the select stuff in golang.

⚠ If implemented, do you think this feature will be a breaking change to users?

No

⚙ Additional context

No response

🤝 Requirements

• I have confirmed that someone else has not submitted a similar feature request.
• If implemented, I believe this feature will help others, in addition to solving my problems.
• I have looked into alternative solutions to the best of my ability.
• (optional) I would be willing to contribute to testing this feature if implemented, or making a PR to implement this functionality.

Vault-unseal process high CPU usage

🌧 Describe the problem

Hi,

I have a problem with the vault-unseal process, because it is constantly using 100% of the CPU. Apparently the process works correctly. This service is running in a proxmox linux container (LXC) with centos 7.

The version I have installed of vault-unseal is the latest (0.2.2), but this happened with previous versions.

[root@mypc ~]$ rpm -qa | grep vault-unseal
vault-unseal-0.2.2-1.x86_64
[root@mypc ~]$

This is a top capture where you can see the high CPU usage

If you need anything else to diagnose what's going on, I'll be available

Thanks!!!

⛅ Expected behavior

Reduce the high CPU usage of the process

🔄 Minimal reproduction

No response

💠 Version: vault-unseal

v0.2.2

🖥 Version: Operating system

linux/centos

⚙ Additional context

No response

🤝 Requirements

• I believe the problem I'm facing is a bug, and is not intended behavior. Post here if you're not sure.
• I have confirmed that someone else has not submitted a similar bug report.

feature: Helm chart for Kubernetes deployment

✨ Describe the feature you'd like

It would be cool to deploy vault-unseal via a helm chart in a Kubernetes cluster.

🌧 Is your feature request related to a problem?

No response

🔎 Describe alternatives you've considered

I've considered to write 3 deployments with their respective secret.
With a Helm chart, I will be able to deploy vault-unseal for example in 3 distinct namespaces with a GitOps tool (ArgoCD, Flux...).
In fact, Helm chart is clearly a standard in the Kubernetes ecosystem and it would be a nice-to-have deployment for this very usefull project.

⚠ If implemented, do you think this feature will be a breaking change to users?

No

⚙ Additional context

No response

🤝 Requirements

• I have confirmed that someone else has not submitted a similar feature request.
• If implemented, I believe this feature will help others, in addition to solving my problems.
• I have looked into alternative solutions to the best of my ability.
• (optional) I would be willing to contribute to testing this feature if implemented, or making a PR to implement this functionality.

feature: Make it easy to push config through a shared drive/location

✨ Describe the feature you'd like

I have 5 nodes which are easy to maintain via a shared drive.
Pushing config for each in ./node${i}/config/vault-unseal.yaml on the said shared drive.
Sadly they all remain in the shared drive unless I move them away, which makes the whole process pointless.

It would be nice to:

• load the config from folderA => my case the sahred drive
• save it to folderB (working dir) => some location on the host
• optionally delete it from folderA => or it can be done manually as well, what is important is that it doesn't rely on folderA anymore

Unless you might suggest another way todo.

🌧 Is your feature request related to a problem?

No response

🔎 Describe alternatives you've considered

Manually login into each node and apply the needed changes before launching the cluster.

⚠ If implemented, do you think this feature will be a breaking change to users?

Not sure

⚙ Additional context

It won't affect the others necessarily, it can be made optional.

🤝 Requirements

• I have confirmed that someone else has not submitted a similar feature request.
• If implemented, I believe this feature will help others, in addition to solving my problems.
• I have looked into alternative solutions to the best of my ability.
• (optional) I would be willing to contribute to testing this feature if implemented, or making a PR to implement this functionality.

Occasional Error Emails

Hi,
Everything is working great on my cluster. I have email notifications setup and I occasionally (numerous times a day) get the following:
vault-unseal ran into errors when attempting to check seal status/unseal. here are the errors:

14 Jan 22 11:50 GMT :: error: checking seal status: Error making API request.

URL: GET https://node03:8200/v1/sys/seal-status
Code: 500. Errors:

• Unexpected response code: 500

sent from vault-unseal. version: v0.0.8, compile date: 2021-10-17T19:40:03Z, hostname: consul03

These alerts come from any of the three vault-unseal nodes and also report the issue on any of the Vault nodes - so it's not an issue with any specific node.
Can anything be done to reduce these - some sort of further check, increased timeout?

Thanks,
Richie

feature: Docker image for arm64 platform

✨ Describe the feature you'd like

Hello! Thanks for the cool tool, it would be nice to add a build docker image for the arm64 platform
If nedded i can help with GH Action for this solution

🌧 Is your feature request related to a problem?

Users can run this tool at raspbery pi/orage pi/banaba pi and etc

🔎 Describe alternatives you've considered

⚠ If implemented, do you think this feature will be a breaking change to users?

No

⚙ Additional context

No response

🤝 Requirements

• I have confirmed that someone else has not submitted a similar feature request.
• If implemented, I believe this feature will help others, in addition to solving my problems.
• I have looked into alternative solutions to the best of my ability.
• (optional) I would be willing to contribute to testing this feature if implemented, or making a PR to implement this functionality.

feature: Support reading tokens from file so tokens can be stored safely in docker secrets

✨ Describe the feature you'd like

Hi, I'm trying to run vault-unseal as container in my docker swarm instance.
Then I realize to make it work I need to store my tokens in unencrypted format inside the vault-unseal.yaml file in the host.

I think it would be very nice if the vault-unseal support reading token list from separate file, so I can then use docker secrets to store my tokens securely.

🌧 Is your feature request related to a problem?

No response

🔎 Describe alternatives you've considered

--

⚠ If implemented, do you think this feature will be a breaking change to users?

Yes

⚙ Additional context

To make my approach of using vault-unseal clearer, here is the example.

First I create new secret containing the tokens:
echo "aaaaaaaaaaaaaaaaaa,bbbbbbbbbbbbbbb" | docker secret create vault_unseal_tokens_a_b -

Then I can use the stack compose script below to easily spawn new vault-unseal node:

version: '3.8'
services:
  vault_unseal_1:
    image: ghcr.io/lrstanley/vault-unseal:latest
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    environment:
      - CONFIG_PATH=/vault_unseal_config
      - TOKENS__FILE=/run/secrets/unseal_tokens
    networks:
      - vault-unseal-network
    configs:
      - source: vault_unseal_config
        mode: 0400
    secrets:
      - unseal_tokens

networks:
  vault-unseal-network:
    name: "vault_unseal_network"

configs:
  vault_unseal_config:
    name: "vault_unseal_conf_v1"
    external: true

secrets:
  unseal_tokens:
    name: "vault_unseal_tokens_a_b"
    external: true

🤝 Requirements

• I have confirmed that someone else has not submitted a similar feature request.
• If implemented, I believe this feature will help others, in addition to solving my problems.
• I have looked into alternative solutions to the best of my ability.
• (optional) I would be willing to contribute to testing this feature if implemented, or making a PR to implement this functionality.

feature: support kubernetes proxy verb

✨ Describe the feature you'd like

It would be cool to have the possibility to pass a kubeconfig and use kubernetes proxy verb. It will allow to unseal a kubernetes vault HA cluster from non kubernetes nodes.

🌧 Is your feature request related to a problem?

If you want to run an HA vault cluster exposed only in the cluster.

🔎 Describe alternatives you've considered

• Exposing each pod with an ingress (I don't like the idea to expose something that is not needed).
• Install vault-unseal in the cluster, I don't like the idea to run this software in the same cluster as my vault cluster. For example a compromise node (with a SSH key) can result in discovering vault unseal keys and secret in one time.

⚠ If implemented, do you think this feature will be a breaking change to users?

No

⚙ Additional context

No response

🤝 Requirements

• I have confirmed that someone else has not submitted a similar feature request.
• If implemented, I believe this feature will help others, in addition to solving my problems.
• I have looked into alternative solutions to the best of my ability.
• (optional) I would be willing to contribute to testing this feature if implemented, or making a PR to implement this functionality.