ake-persson / etcdtool

Export/Import/Edit etcd directory as JSON/YAML/TOML and validate directory using JSON schema

etcdtool

Export/Import/Edit etcd directory as JSON/YAML/TOML and validate directory using JSON schema.

Use cases

• Backup/Restore in a format which is not database or version specific.
• Migration of data from production to testing.
• Store configuration in Git and use import to populate etcd.
• Validate directory entries using JSON schema.

Build

git clone https://github.com/mickep76/etcdtool.git
cd etcdtool
make

Build RPM

Make sure you have Docker configured.

git clone https://github.com/mickep76/etcdtool.git
cd etcdtool
make rpm

Install using Homebrew on Mac OS X

brew tap mickep76/funk-gnarge
brew install etcdtool

Update

brew update
brew upgrade --all

Example

Make sure you have Docker configured.

Start etcd:

./init-etcd.sh start
eval "$(./init-etcd.sh env)"

Import some data:

cd examples/host/
etcdtool import /hosts/test1.example.com test1.example.com.json
etcdtool import /hosts/test2.example.com test2.example.com.json

Inspect the content:

etcdtool tree /
etcdtool export /

Export the content and infer numbers lists to keep original json:

etcdtool export / --num-infer-list

Validate data with different routes:

etcdtool -d -c etcdtool.toml validate /
etcdtool -d -c etcdtool.toml validate /hosts
etcdtool -d -c etcdtool.toml validate /hosts/test2.example.com
etcdtool -d -c etcdtool.toml validate /hosts/test2.example.com/interfaces
etcdtool -d -c etcdtool.toml validate /hosts/test2.example.com/interfaces/eth0

Import with validation:

etcdtool -d -c etcdtool.toml import -v /hosts/test3.example.com test2.example.com.json

Fix validation error:

etcdtool -d -c etcdtool.toml edit -v -f toml /hosts/test2.example.com

---
    gw = "1.192.168.0.1"
+++
    gw = "192.168.0.1"

Re-validate data:

etcdtool -d -c etcdtool.toml validate /hosts

Authentication

These commands will prompt you for the password for the user. Alternatively, you can pass the password in a file with --password-file or -F:

cat /path/to/passwordfile
passwordstring

etcdtool -password-file /path/to/passwordfile validate /hosts

Caveats

• etcd doesn't support list's, this is handled by using the index as the key:

JSON Input:

{
    "users": [
        { "username": "jblack", "first_name": "John", "last_name": "Blackbeard" },
        { "username": "ltrier", "first_name": "Lars", "last_name": "Von Trier" }
    ]
}

Result in etcd:

users/0/username: jblack
users/0/first_name: John
users/0/last_name: Blackbeard
users/1/username: ltrier
users/1/first_name: Ludwig
users/1/last_name: Von Treimer

TODO

• Add detection of format for import based on file type

etcdtool's Issues

Update to using new Etcd API

Password support

We would love to use this tool programmatically, but running into issues with passing the password. Would you be open to a PR that added support for using a flag or environment variables or something else? I'm willing to work on it, just wanted to get your thoughts on the best strategy.

Thanks!

Create missing directories on import

Create missing directories on import, maybe -create switch.

Add stdin support

Import doesn't work with YAML files

When I try to import a YAML into etcd, the imported content is broken.
Digging into the code, I have found that the problem lies in etcdmap.Create.

It seems, json and toml return some sort of map[string]interface{} whereas yaml returns a map[interface{}]interface{}. This is causing a lot of trouble and an issue has been opened for go-yaml (see go-yaml/yaml#139, someone posted a workaround for this)

Here is an example that shows the problem:

package main

import (
  "fmt"
  "reflect"
  "encoding/json"
  "gopkg.in/yaml.v2"
)

func test(doc interface{}) {
  fmt.Println(doc)
  v := reflect.ValueOf(doc)
  fmt.Printf("Kind: %v\n", v.Kind())
  for _, k := range v.MapKeys() {
    fmt.Println(k.String())
  }
}


func main() {

  var doc interface{}

  fmt.Println("=== YAML ===")
  yaml.Unmarshal([]byte("a: b"), &doc)
  test(doc)

  fmt.Println("=== JSON ===")
  json.Unmarshal([]byte(`{"a":"b"}`), &doc)
  test(doc)
}

Run this and you should see:

=== YAML ===
map[a:b]
Kind: map
<interface {} Value>     <===== !!!!
=== JSON ===
map[a:b]
Kind: map
a

Add Mac OS X packaged version

Cannot import empty directories

Importing json with an empty directory skips that directory. For example, importing the following json:

{"empty_dir":{},"dir":{"key":"value"}}

results in:

/
└── dir/
    └── key

I thought this might be related to matching Go's default value if a dir is unmarshalled into an empty map, but I'm not sure because keys with empty string values are imported correctly.

Problem with list convertions

Hi there!

I'm facing a problem trying to import/export some data into etcd using etcdtool. Since lists are converted into json objects creating directories using the index as the key, when I load a json file with a list and then try to export the destination key, both json files differ.

Example:

# cat test.json 
{
  "mylist": [
    { "one": "uno"},
    {"two": "dos"}
  ]
}
# ./etcdtool-3.4 --peers="https://myetcd:2379" import -y /test ./test.json
# ./etcdtool-3.4 --peers="https://myetcd:2379" export /test
{
  "mylist": {
    "0": {
      "one": "uno"
    },
    "1": {
      "two": "dos"
    }
  }
}

It would be great to export the directory so the resulting json file is the same that was loaded (expected behavior: when all the keys in a directory are whole numbers, then convert them into a list). Maybe it can be added as an optional parameter (e.g. --keep-lists)?.

Refactor code into one binary with positional arg.

• Major cleanup and normalization of code

Allow for JSON schema reference to Etcd key

Like:

{
    "ntp": {
        "$ref": "etcd://etcd1.example.com:5001/#/schemas/ntp"
    }
}

macOS : Build error during brew install

Environment

macOS Mojave 10.14.6
go 1.13.1

Description

Here is the output I got when I ran brew install etcdtool :

[0][0][benjy@benjy-mac:~/local/var/lib]
$ brew install etcdtool
==> Installing etcdtool from mickep76/funk-gnarge
==> Downloading https://github.com/mickep76/etcdtool/archive/3.3.2.tar.gz
Already downloaded: /Users/benjy/Library/Caches/Homebrew/downloads/2c5ed62a0fe74cb017f87d9ed730b0a90463edf47cdcb9aebb409aa712d33e3b--etcdtool-3.3.2.tar.gz
==> ./build
Last 15 lines from /Users/benjy/Library/Logs/Homebrew/etcdtool/01.build:
github.com/xeipuuv/gojsonschema
runtime.main_main·f: function main is undeclared in the main package
# github.com/constabulary/gb/cmd/gb-vendor
github.com/constabulary/gb/cmd/gb-vendor
runtime.main_main·f: function main is undeclared in the main package
# github.com/constabulary/gb/cmd/gb
github.com/constabulary/gb/cmd/gb
github.com/ugorji/go/codec
github.com/coreos/etcd/client
github.com/mickep76/etcdmap
github.com/mickep76/etcdtool/command
runtime.main_main·f: function main is undeclared in the main package
# github.com/mickep76/etcdtool
github.com/mickep76/etcdtool
FATAL: command "build" failed: exit status 2

If reporting this issue please do so at (not Homebrew/brew or Homebrew/core):
  https://github.com/mickep76/homebrew-funk-gnarge/issues

Option to recursively delete data for a directory before import.

Mac M1 can't install etcdtool through brew

eden@edendeMacBook-Pro etcdtool % brew tap mickep76/funk-gnarge
fatal: not in a git directory
Warning: No remote 'origin' in /opt/homebrew/Library/Taps/homebrew/homebrew-services, skipping update!
==> Tapping mickep76/funk-gnarge
Cloning into '/opt/homebrew/Library/Taps/mickep76/homebrew-funk-gnarge'...
Username for 'https://github.com':
Password for 'https://github.com':
remote: Repository not found.
fatal: Authentication failed for 'https://github.com/mickep76/homebrew-funk-gnarge/'
Error: Failure while executing; git clone https://github.com/mickep76/homebrew-funk-gnarge /opt/homebrew/Library/Taps/mickep76/homebrew-funk-gnarge --origin=origin --template= exited with 128.

Mac can't install etcdtool through brew, why do I need a github account and password to install it, and I can't use my own github account and password

Add support for TLS

support for etcd v3

Hi,

Are there any plans for supporting etcd v3?

Kind regards,
Peter

Tool cannot be built any more

Unfortunately tool cannot be built any more, because it is referring to a package which is not available any more:
github.com/mickep76/iodatafmt

I understood, the successor of this github repo would be
github.com/mickep76/encoding

But how to change that in the source code, that package is building again?

Build failed

github.com/mickep76/iodatafmt is not available now and build does not work

go: github.com/mickep76/[email protected]: invalid version: git ls-remote -q origin in /Users/maxim-konovalov/go/pkg/mod/cache/vcs/4efce029de38da2ac7d547b49ef84924855725f87afda474db214842c5723ab9: exit status 128:
remote: Repository not found.
fatal: repository 'https://github.com/mickep76/iodatafmt/' not found
go: github.com/mickep76/[email protected]: invalid version: git ls-remote -q origin in /Users/maxim-konovalov/go/pkg/mod/cache/vcs/4efce029de38da2ac7d547b49ef84924855725f87afda474db214842c5723ab9: exit status 128:
remote: Repository not found.
fatal: repository 'https://github.com/mickep76/iodatafmt/' not found

Add detection of file format based on ext on import

Add inotify support for monitoring conf dir

Add inotify support for monitoring conf dir to upload files

Add resume as a choice after failed edit

Currently resume is only offered as a choice when running the same command again and there is a cached file.

RPM Build Process Broken?

I noticed the latest released version 3.3.2 doesn't have the fix made here: #20 So I'm attempting to build my own RPM - Following the instructions doesn't seem to work:

[jlang1@antman etcdtool]$ make rpm
docker pull mickep76/centos-golang:latest
Trying to pull repository docker.io/mickep76/centos-golang ...
latest: Pulling from docker.io/mickep76/centos-golang
Digest: sha256:4d413b367327364bea4d3860eafc8f70d8424dc144894f2ab191198a4d80a1b6
Status: Image is up to date for docker.io/mickep76/centos-golang:latest
docker run --rm -it -v "$PWD":/go/src/github.com/mickep76/etcdtool -w /go/src/github.com/mickep76/etcdtool mickep76/centos-golang:latest make build-rpm
make: *** No rule to make target `make'. Stop.
make: *** [rpm] Error 2

Alternatively, if you could build a 3.4 RPM and "release it" I'd be happy with that too :)

Add support for importing list without an encapsulating hash

[
    { "username": "jblack", "first_name": "John", "last_name": "Blackbeard" },
    { "username": "ltrier", "first_name": "Lars", "last_name": "Von Trier" }
]

This will now generate:

json: cannot unmarshal array into Go value of type map[string]interface {}

Cannt install etcdtool

env

mac: 10.15.5 (19F101)

install by brew

when I install by brew

 ~ brew tap mickep76/funk-gnarge
==> Tapping mickep76/funk-gnarge
Cloning into '/usr/local/Homebrew/Library/Taps/mickep76/homebrew-funk-gnarge'...
remote: Repository not found.
fatal: repository 'https://github.com/mickep76/homebrew-funk-gnarge/' not found
Error: Failure while executing; `git clone https://github.com/mickep76/homebrew-funk-gnarge /usr/local/Homebrew/Library/Taps/mickep76/homebrew-funk-gnarge` exited with 128.

install by code

step1: I run git clone pull code and cd etcdtool

step2: I run make

~/etcdtool   master  make
rm -rf build release
mkdir build || true
go build -o build/etcdtool
go: github.com/mickep76/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/dacheng.sun/go/pkg/mod/cache/vcs/4efce029de38da2ac7d547b49ef84924855725f87afda474db214842c5723ab9: exit status 128:
	remote: Repository not found.
	fatal: 仓库 'https://github.com/mickep76/iodatafmt/' 未找到
make: *** [build] Error 1

Build with CGO_ENABLED=0 and netgo?

quote from: https://dominik.honnef.co/posts/2015/06/statically_compiled_go_programs__always__even_with_cgo__using_musl/

The second solution, which most people go for, is to avoid cgo altogether. They use the netgo tag to build a pure-Go version of the net package, and they set CGO_ENABLED=0. Of course this has the drawback of not being able to use cgo. That means you can’t even use os/user to get the name of the current user.

remove dependency on glibc for this tool?

yes/no is ignored on `import --replace --yes`

When doing an import with --replace --yes, you still get asked if you want to overwrite. Looking at the code, I can't tell if this is intentional or an oversight. I can override this behavior by piping the word "yes" to the program, but I think that it would make more sense for --yes to "Answer yes to any questions".

Context: goal is to store files representing specific directories in git, going through PR review process, and on change, commit those to etcd.

Error: SHA256 mismatch

$ brew tap mickep76/funk-gnarge
$ brew install etcdtool
==> Installing etcdtool from mickep76/funk-gnarge
==> Downloading https://github.com/mickep76/etcdtool/archive/3.3.2.tar.gz

Error: SHA256 mismatch
Expected: f9f4aea1e1459e1f187e8f74faec537b97cb2519bdead9f12e4c1bf93a69df6b
Actual: e3e6964e33e4cade95303b642f5b160225e61eac1fe754da606983a4c0edf06e
Archive: /Users/andy/Library/Caches/Homebrew/etcdtool-3.3.2.tar.gz
To retry an incomplete download, remove the file above.