pbs / gorson

experiment with go and aws ssm parameter store

Warning: experimental

This is an experimental library, and is currently unsupported.

Usage

gorson loads parameters from AWS ssm parameter store, and adds them as shell environment variables.

Download parameters from parameter store as a json file

gorson get /a/parameter/store/path/ > ./example.json

$ cat ./example.json

{
    "alpha": "the_alpha_value",
    "beta": "the_beta_value",
    "delta": "the_delta_value"
}

There's also a --format flag to pass in which format you want the parameters to export as.

gorson get --format yaml /a/parameter/store/path/ > ./example.yml

$ cat ./example.yml

alpha: "the_alpha_value"
beta: "the_beta_value"
delta: "the_delta_value"

gorson get --format env /a/parameter/store/path/ > ./.env

$ cat ./.env

alpha="the_alpha_value"
beta="the_beta_value"
delta="the_delta_value"

Load parameters as environment variables from a json file

source <(gorson load ./example.json)

$ env | grep 'alpha\|beta\|delta'
alpha=the_alpha_value
delta=the_delta_value
beta=the_beta_value

Upload parameters to parameter store from a json file

gorson put /a/parameter/store/path/ --file=./new-values.json

Delete parameter difference on put

$ gorson put /a/parameter/store/path/ --file=./different-values.json --delete

The following are not present in the file, but are in parameter store:
/a/parameter/store/path/gamma
Are you sure you'd like to delete these parameters?
Type yes to proceed:

Auto-approve prompts

If you would like to answer 'yes' to any prompts that require it, append --auto-approve.

Deactivate color

If you would prefer the output of commands to be colorless, append --no-color.

Installation

Currently gorson ships binaries for MacOS and Linux 64bit systems. You can download the latest release from GitHub

MacOS

wget https://github.com/pbs/gorson/releases/latest/download/gorson-darwin-amd64

Linux

Download the binary

wget https://github.com/pbs/gorson/releases/latest/download/gorson-linux-amd64

ARM

For either MacOS or Linux, replace the amd64 with arm64 above.

Install the binary

Move the binary to an installation path, make it executable, and add to path

mkdir -p /opt/gorson/bin
mv gorson-linux-amd64 /opt/gorson/bin/gorson
chmod +x /opt/gorson/bin/gorson
export PATH="$PATH:/opt/gorson/bin"

asdf

Install using asdf

Add asdf plugin

asdf plugin add gorson https://github.com/pbs/asdf-pbs.git

List available versions

asdf list-all gorson

Install a particular version

asdf install gorson 13

Make a particular version your default

asdf global gorson 13

Notes

These environment variables will affect the AWS session behavior:

https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

• AWS_PROFILE: use a named profile from your ~/.aws/config file (see https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html)
• AWS_REGION: use a specific AWS region (see https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html)

AWS_PROFILE=example-profile AWS_REGION=us-east-1 gorson get /a/parameter/store/path/

Development

See docs/development.md

gorson's Issues

Value with space and colon doubly quoted with single quotes

With gorson get --format env /DEV/PBSORG > .env running against pbs-digi-preprod, I get a KV pair that looks like this:
PBSORG_BADGE_CTA_HEADLINE=''COMPLETE COVERAGE:''

That causes bash to choke when sourcing the file:
/etc/pbsorg/.env: line 29: COVERAGE:: command not found.

The full value in the parameter store is COMPLETE COVERAGE:. Odd that Gorson is putting two pairs of single quotes around it, as there are many other values with whitespace that aren't treated as such.

Remove -buildvcs flag from release script

The -buildvcs=false flag seems to do an end-run around protections deemed advisable by the Go community. We should modify the way we build the release such that this is not necessary.

Advisable to do this after the 1.20 release, as there are patches included in said release that would seem to affect go bulid behavior: golang/go@bb39656

Add ARM Support

We should add aarm64 support to support for ARM machines like the M1 Macbooks, and the Graviton processors.

Cannot retrieve individual parameters

[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:32:02]» gorson get --format json /STAGING/SVP/SVP_REDIS_DATABASE
{}
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:34:32]» gorson get --format env /STAGING/SVP/SVP_REDIS_DATABASE

[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:34:35]» gorson get --format env /STAGING/SVP/SVP_REDIS_DATABASE > /tmp/foo
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:34:45]» cat /tmp/foo
───────┬─────────────────────────────────────────────────────────────────────────────────────
       │ File: /tmp/foo
───────┼─────────────────────────────────────────────────────────────────────────────────────
   1   │ 
───────┴─────────────────────────────────────────────────────────────────────────────────────
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:34:48]» gorson get --format env /STAGING/SVP/SVP_REDIS_DATABASE/ > /tmp/foo
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:35:07]» cat /tmp/foo                                                       
───────┬─────────────────────────────────────────────────────────────────────────────────────
       │ File: /tmp/foo
───────┼─────────────────────────────────────────────────────────────────────────────────────
   1   │ 
───────┴─────────────────────────────────────────────────────────────────────────────────────
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:35:29]» gorson get --format json /STAGING/SVP/SVP_REDIS_DATABASE/ > /tmp/foo
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:35:31]» cat /tmp/foo                                                        
───────┬─────────────────────────────────────────────────────────────────────────────────────
       │ File: /tmp/foo
───────┼─────────────────────────────────────────────────────────────────────────────────────
   1   │ {}
   2   │ 
───────┴─────────────────────────────────────────────────────────────────────────────────────
[ RWEB-8083-no-more-memcached][  system][🅰 pbs-digi-preprod][RWEB-8083-no-more-memcached]
[2024-03-05 17:35:37]❮ gorson get --format json /STAGING/SVP/ > /tmp/foo   | wc -l 
84              

Add `--soft-delete` Flag

Instead of directly deleting parameters that show up in a delta between a local JSON and SSM parameter store, having the capability to tag parameters for future deletion, and treat them as missing during gets would allow for a more complicated, external process to handle their removal safely.

Readme typo

On line 107 of the readme, what is meant to be the linux binary in the url is actually the darwin binary. I was going to PR that change but I got the error ERROR: Permission to pbs/gorson.git denied to pbs-jmo.

ThrottlingException: Rate exceeded

When working with a particularly large json file to be put into parameter store, throttling can be encountered due to the way parameters are being put.

I suspect that resolution of this problem lies in getting this updated with some exponential back off implemented, but I don't know to detect this throttling error (ThrottlingException: Rate exceeded), as these seem to be the errors that can be returned by the PutParameter: https://docs.aws.amazon.com/sdk-for-go/api/service/ssm/#SSM.PutParameter

Some investigation will be necessary.

As a workaround, AWS accounts can be adjusted to increase throttling limits, but it costs money and it might benefit free tier users to have the retry when throttled.

Create `--delete` Flag

It might be useful to have a -d/--delete flag that will do a destructive sync (delete any parameters that are not present in the JSON being used).

@cmac1000 @joshfinnie what do you guys think about this? Worth it?

[Security] Workflow release.yml is using vulnerable action actions/checkout

The workflow release.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

Mocking AWS SDK Calls

SDK calls can be mocked like this https://aws.amazon.com/blogs/developer/mocking-out-then-aws-sdk-for-go-for-unit-testing/. This should be implemented here.

Implement Github Actions for test running/binary building

I think it would be useful to leverage Github Actions to run the tests and build the binaries.

Put discrete parameters

Users shouldn't have to create a JSON file to insert a single parameter in the store.

Proposed command line would look something like this:

gorson put /a/parameter/path/toasingleparameter "<quoted value"

Add format flag to gorson load

The usage for the gorson load command states:

Usage:
  gorson load ./example.json

gorson get allows the user to retrieve parameter store entries and format them in a variety of ways. I'm proposing that a flag for load be implemented such that the user can load any of these potential formats (JSON, .env, or YAML).

Create version agnostic artifact for Latest release

GitHub allows for a redirect such that a URL in the form of https://github.com//owner/name/releases/latest/download/asset-name.zip is redirected to the release tagged Latest. However, if there is no artifact named asset-latest.zip (or something similarly version agnostic), then that request will fail.

It would be nice to avoid hard-coding a version in URLs used to retrieve gorson. For example, in pbsorg Dockerfile, we have this:
RUN wget https://github.com/pbs/gorson/releases/download/4.2.0/gorson-4.2.0-linux-amd64 (now woefully out-of-date). This would be avoided with the kind of indirection suggested.

Add Confirmation for Puts

Now that gorson is being used more widely, it might be nice to have a little confirmation page with a diff on parameter updates before completing the put request.