lyraproj / hiera Goto Github PK
View Code? Open in Web Editor NEWHiera in golang - Elegant, powerful hierarchical key-value lookups
License: Apache License 2.0
Hiera in golang - Elegant, powerful hierarchical key-value lookups
License: Apache License 2.0
In addition to the --facts
flag (which needs an input file in yaml or json), it'd be cool if the lookup
command had a way to pass fact name/value on the command line. This would make it easy and fast to test hierarchy or data changes. I'd suggest a syntax like
lookup --fact name1=value1 --fact name2=value2
hi,
literal('%') is not functional
usecase: test to lookup a var with the following definition
---
test: "%{literal('%'}{TEST}"
Results:
it outputs an empty string, as if the resulting string were interpolated twice.
to make it work you have to escape the following way:
---
test: "%{literal('%')}{literal('%{literal('%')}')}{TEST}"
I think I'm seeing something similar to this - for some lookups I get back __ref: 26
or similar on the first try, but retrying the lookup returns the correct value. I can't figure out exactly what's triggering it, but I think it's only when using strict_alias
. Also passing --dialect pcore
seems to make it work first time.
Currently, it seems to be necessary to specify data_dir
per heirarchy key; if it is only used in the defaults
key, hiera does not prepend it and fails to find the data files.
Given a hierarchy like:
---
version: 5
defaults:
data_dir: hiera
data_hash: yaml_data
hierarchy:
- name: "Environmental overrides"
path: "environments/%{environment}.yaml"
- name: "Fall through defaults"
path: "defaults.yaml"
The debug messages show:
019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="deep merge strategy" provider="data_hash function 'yaml_
data'" location="path{ original:environments/%{environment}.yaml, resolved:/Users/eric/Sandbox/lyra-local/environments/.yaml, exist:false}"
2019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="deep merge strategy" provider="data_hash function 'yaml_
data'" location="path{ original:defaults.yaml, resolved:/Users/eric/Sandbox/lyra-local/defaults.yaml, exist:false}"
2019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="first found strategy" provider="data_hash function 'yaml
_data'" location="path{ original:environments/%{environment}.yaml, resolved:/Users/eric/Sandbox/lyra-local/environments/.yaml, exist:false}" 2019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="first found strategy" provider="data_hash function 'yaml_data'" location="path{ original:defaults.yaml, resolved:/Users/eric/Sandbox/lyra-local/defaults.yaml, exist:false}"
2019-05-20T06:17:35.597-0700 [ERROR] lyra: apply failed: Error="lookup() did not find a value for the name 'wordpress.location' (file: /private/tmp/lyr
a-20190520-79576-p2kmxa/lyra-0.2.0/.brew_home/go/pkg/mod/github.com/lyraproj/[email protected]/internal/init.go, line: 106)
hiera.yaml
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: global
path: global.yaml
global.yaml
empty_map: {}
lookup empty_map --render-as json
Error: json: unsupported type: map[interface {}]interface {}
After the changes in #60 , I cannot start hieraserver either standalone or in a container.
I think the problem is this bit:
hiera/hieraserver/rest/rest.go
Lines 167 to 171 in a04027d
Because makeTLSConfig()
creates a new tls.Config
instance before it checks whether ssl keys and certs are passed in, the variable is never nil
; it contains an instantiated but non-working tls configuration.
One of the super helpful things about puppet-hiera is the puppet lookup --explain
functionality, that shows the provenance of a value as it's looked up in the hierarchy. It's be great to have something similar in go-hiera. For reference, explain output looks like this (we don't need to exactly match this, IMO it's a bit too "busy" and could stand some simplification and maybe colorisation):
Searching for "openssl::package_version"
Global Data Provider (hiera configuration version 5)
No such key: "openssl::package_version"
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Hierarchy entry "Classifier Configuration Data"
No such key: "openssl::package_version"
Hierarchy entry "Datacenter"
Path "/etc/puppetlabs/code/environments/production/hieradata/datacenter/.yaml"
Original path: "datacenter/%{::datacenter}.yaml"
Path not found
Hierarchy entry "PE instance secrets"
Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/pe-kearney.p9.puppet.net.eyaml"
Original path: "nodes/%{fqdn}.eyaml"
Path not found
Hierarchy entry "common-e"
Path "/etc/puppetlabs/code/environments/production/hieradata/common.eyaml"
Original path: "common.eyaml"
No such key: "openssl::package_version"
Hierarchy entry "common"
Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml"
Original path: "common.yaml"
Found key: "openssl::package_version" value: "1.0.1e-60.el7_3.1"
Since the latest update the lookup command seems to be dumping the plugin meta information to stderr:
& lookup.exe testy
2020/03/04 14:18:30 {"version":1,"network":"unix","address":"C:\\Users\\HENRY~1.BUC\\AppData\\Local\\Temp\\terraform_backend.exe-8280.socket","functions":{"data_hash":{"terraform_backeend"}}}
2020/03/04 14:18:30 {"backend":"local","root_key":"root","config":{"path":"one.tfstate"}}
2020/03/04 14:18:30 {"root":{"two":"two","three":"two","one":"one"}}
2020/03/04 14:18:30 value
value
I've tried debugging the lookup command but I can't see what's writing it
I need to do the following in golang code:
mykey
, module_a::mykey
, module_b::mykey
- I am not expecting hiera to do that for me).hiera.yaml
files involved to look exactly the same as in the Ruby version (to allow the ruby version of my application to use exactly the same configuration and data files).Trying out hiera 0.3 and getting an error with plugins
version: 5
defaults:
plugindir: C:\Temp\hiera\plugins
hierarchy:
- name: one
data_hash: terraform_backend
options:
backend: local
config:
path: C:\Temp\hiera\one.tfstate
lookup.exe test
Error: unable to start plugin /C:/Temp/hiera/plugins/terraform_backend.exe: exec: "/C:/Temp/hiera/plugins/terraform_backend.exe": file does not exist
Looks like (dgo.Loader).AbsoluteName()
is returning the path with /
prefixed
Comes from #78
Given a facts.yaml
file:
---
inventory: 'a'
For people like me coming from puppet, it's a bit weird/consufing that using --facts facts.yaml
results in a different interpolation when using lookup
(inventory=a
) and when using puppet lookup
(facts.inventory=a
).
I feel that it should be at least a little documented somewhere (maybe it is and I haven't found it) or, for an easiest transition between tools, may current go implementation copy all facts to make them also available as facts map subkeys.
To give some background in our use case, we come from using Puppet + some scripting with hiera as our main source of truth and now we're replacing our scripting with tools like terraform but want to keep our hiera hierarchy as much as posible "as is".
Thanks for alll your fantastic work
/cc @thallgren
The key to hiera's operation is the hiera.yaml
which lets users configure their own hierarchies. This implementation should support a minimal variant of the Ruby hiera version - path names with (optional) variable interpolation in order, falling through to a common baseline.
Any chance of tagging up some releases for this and the plugins?
Thanks
When getting an error like this:
panic: reflect: call of reflect.Value.SetFloat on int32 Value [recovered]
panic: reflect: call of reflect.Value.SetFloat on int32 Value
When calling vf.FromValue(v, target)
it is time consuming to figure out where the problem is when there are multiple int32 values in the target structure.
Suggest adding the name of the type/member that it is failing on in the error message. There should probably be a general catch that amends all error messages.
The workflow to build and publish docker image runs on every push, regardless of repository. In my thallgren fork, the build fails because I don't have the correct credentials to publish (which is probably for the good in this case).
Example of failure: https://github.com/thallgren/hiera/runs/284812108
hiera.yaml
version: 5
hierarchy:
- name: yaml
path: yaml1.yaml
yaml1.yaml
yaml1: "%{lookup('nothing')}"
> lookup yaml1
Error: runtime error: invalid memory address or nil pointer dereference
Previously this used to return undef
Add a standalone command line tool similar to puppet lookup.
I created a an AWS SSM Parameter Store plugin. It works, so I can write secrets to the parameter store and reference them from Hiera. An example hiera.yaml
configuration that uses the plugin looks like this:
---
version: 5
defaults:
plugindir: ./plugins
datadir: ./data
data_hash: yaml_data
hierarchy:
- name: region
path: region/%{region}.yaml
- name: environment
path: environment/%{environment}.yaml
- name: common
glob: defaults/*.yaml
- name: secrets
lookup_key: "aws_ssm_parameter"
options:
aws_profile_name: production.AdministratorAccess
aws_region: "us-west-1"
What I had hoped to do was to use interpolation of %{environment}
and %{region}
in the options section, like this:
- name: secrets
lookup_key: "aws_ssm_parameter"
options:
aws_profile_name: "%{environment}.AdministratorAccess"
aws_region: "%{region}"
This would enable Hiera to seamlessly switch between AWS accounts and regions based on these "facts" (or in the case of the Terraform Hiera provider, based on provider scope). Unfortunately, interpolation doesn't work for options so I must hard-code them. This forces me to store all parameters within a single AWS account and region.
Hiera finds configured lookup functions (providers) using a loader. New functions can be added using the pcore px.NewGoFunction()
. This will however require that the function is compiled together with the rest of the Hiera binary (#34 is a good example). A better approach would be if such functions were reachable using some plug-in framework that could use a Hiera SDK and then made available as Go plugins (shared libraries / same address space) or as RESTful binaries (see #32).
Adding the Azure lookup function alone increased the go.sum file from 75 to 227 lines, corresponding to a transitive dependency tree that is three times as large. It can be expected that other types of lookup functions will cause a similar increase in dependencies.
The current git action for testing tests on linux only. We need to run tests on windows too.
As Go doesn't install modules by default, the documentation doesn't work as shown.
$ go get github.com/lyraproj/hiera
package github.com/lyraproj/hiera: no Go files in /Users/jrhett/go/src/github.com/lyraproj/hiera
$ go install github.com/lyraproj/hiera/lookup
../../go/src/github.com/lyraproj/hiera/internal/location.go:8:2: cannot find package "github.com/bmatcuk/doublestar" in any of:
/usr/local/Cellar/go/1.12.6/libexec/src/github.com/bmatcuk/doublestar (from $GOROOT)
/Users/jrhett/go/src/github.com/bmatcuk/doublestar (from $GOPATH)
I suspect many potential users are not aware enough of Go's module management to do the clone of the repo and install the modules, so at least mentioning it is probably a good idea. Or outright mention GO111MODULE=on go get github.com/lyraproj/hiera
as the quick fix ;)
Was looking at possibly using this instead of the gem counterpart. Love the direction so far, but think I'm blocked as we use eyaml as a way of encrypting secrets.
Do you have a mechanism that accomplishes this or is there plans to support such a plugin?
Hello! We are on hiera version v0.2.0.
We are parsing a fact into a JSON output format and noticed a parsing error.
When parsing a yaml list like
myList:
- typeA: a
- typeB: b
- typeC: c
- tupeD: d
hiera will parse it as
[{"typeA":"a"},{"typeB":"b"},{"typeC":"c"}:{"tupeD":"d"}]
Note that what should have been a comma delimiter is now a :
!
We have the go code to parse from the yaml file:
var variables []string
for _, fact := range facts {
variables = append(variables, fact.String())
}
cmdOpts.Merge = "deep"
cmdOpts.Variables = variables
cmdOpts.RenderAs = formatToFormatString(format)
configOptions := map[string]px.Value{
provider.LookupKeyFunctions: types.WrapRuntime([]hieraapi.LookupKey{provider.ConfigLookupKey, provider.Environment})}
configOptions[hieraapi.HieraConfig] = types.WrapString(path.Join(directory, "hierav5.yaml"))
dest := bytes.NewBuffer(nil)
err := hiera.TryWithParent(context.TODO(), provider.MuxLookupKey, configOptions, func(c px.Context) error {
hiera.LookupAndRender(c, &cmdOpts, []string{key}, dest)
return nil
})
I'm trying to merge multiple yaml files. Since I haven't found out how I can read everything in and process it, I process everything section to section. That goes well so far, but now I'm slowly getting problems that I don't understand. That's why I would like to know how it is intended and how it works.
I'll explain (even if it can be easier or more elegant) how I do it and where my problem is.
I have a struct, in which I would like to save the result later in order to be able to process it further.
hiera.DoWithParent(context.Background(), provider.ConfigLookupKey, configOptions, func(hs api.Session) {
result := hiera.Lookup(hs.Invocation(nil, nil), section, nil, map[string]string{`merge`: `deep`})
s := streamer.New(nil, nil)
b := bytes.Buffer{}
s.Stream(result, streamer.JSON(&b))
if err = json.Unmarshal(b.Bytes(), &c.Hosts); err != nil { ...... }
}
Do I only have one host e.g. in the yaml, it's fine. However, if I have anchors, I sometimes get references that are lost when restoring or cause panic because they do not match my struct.
{
"host2":
{
....
"data":[{"__ref":9}]
....
}
}
panic: json: cannot unmarshal object into Go struct field Host.Data of type string
If my investigations are correct, then this reference comes from dgo, I have seen in streamer there is a function unmarshalJSON that requires a type dialect, I have not yet understood what this dialect is and whether I can convention dgo.Value after dialect.
Can someone please explain to me the best way to look up my files. Get the full change back?
At the moment I am just doing a lookup on a certain value and have it returned to me and read this into the struct. It would of course be ingenious if it were possible to receive the complete change over the files and to write them to the struct in one go.
Thank you in advance for your help.
Hi there, we'd like to be able to either:
Why?
Because we don't know what keys will exist in the various yaml files ahead of time.
Example:
Let's say we have files:
\hiera\hiera.yaml:
defaults:
datadir: data
hierachy:
- name: per_environment
path: per_environment/%{environment}.yaml
\hiera\data\global.yaml:
key1: abc
key2: def
\hiera\data\per_environment\staging.yaml
key2: xyz
\hiera\data\per_environment\production.yaml
key2: pqr
We'd like to say something like:
lookup --get-dump-of-all-values --render-as json --vars facts.json
If we specify an environment of production we'd expect to get output:
{
key1: abc
key2: pqr
}
If we don't specify any environment we'd expect to get output:
{
key1: abc
key2: def
}
If you can suggest a way we can do this with existing functionality that'd be fab, otherwise would it be possible to add this as a feature?
Thanks very much.
global.yaml:
mynull: null
mymap:
mykey: null
.\lookup.exe mynull
null
.\lookup.exe mymap
Error: reflect: call of reflect.Value.Type on zero Value
Many teams who use Puppet to manage server-based infrastructure are also starting to manage cloud-native application delivery including the needed infrastructure. Usually, those teams have codified configuration data in hiera for the Puppet node-based configuration processes, but they cannot use this company-wide configuration data in a cloud-native deployment pipeline. If Hiera would be available as a service, hiera lookups could be easily done outside the "traditional Puppet context", for example from a delivery pipeline, and enable teams to define configuration data only once - inside hiera.
Note: HAAS means "hare" in Dutch
vf.FromValue does not handle a target with map[string]*X
It will crash with an error that it cannot do reflection on nil.
Me again :)
I can't seem to get the lookup interpolation function to work when using the terraform backend plugin, however lookups using the command line are working. This works using the pre-plugin single binary version
hiera.yaml
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: tfstate
data_hash: terraform_backend
options:
backend: azurerm
config:
storage_account_name: xxx
container_name: tfstate
access_key: xxx
key: xxx
- name: global
path: global.yaml
global.yaml
test: "%{lookup('vnet_name')}"
.\lookup.exe vnet_name --explain
Searching for "vnet_name"
data_hash function 'terraform_backend'
Found key: "vnet_name" value: 'testlemur'
.\lookup.exe test --explain
Searching for "test"
Merge strategy "first found strategy"
data_hash function 'terraform_backend'
No such key: "test"
data_hash function 'yaml_data'
Path "C:\Temp\hiera\data\global.yaml"
Original path: "global.yaml"
Interpolation on "%{lookup('vnet_name')}"
Searching for "vnet_name"
Merge strategy "first found strategy"
data_hash function 'terraform_backend'
No such key: "vnet_name"
data_hash function 'yaml_data'
Path "C:\Temp\hiera\data\global.yaml"
Original path: "global.yaml"
No such key: "vnet_name"
Found key: "test" value: 'undef'
Merged result: 'undef'
For compatibility with the puppet hiera docs, data_dir
needs to be renamed to datadir
.
https://puppet.com/docs/puppet/5.0/hiera_config_yaml_5.html#the-defaults-key
It would be very convenient if LookupInto
could handle time.Duration
such that an integer value is interpreted as a nanosecond timestamp and that a string is interpreted in the time.ParseDuration
string format (for example "14h52m"). At present this ends with a panic panic: reflect: call of reflect.Value.SetString on int64 Value
.
The only current work around is to use string
fields and post process the looked up result.
A url based lookup_key function should be added so that one Hiera configuration can include lookups in Hiera servers that resides on other hosts. The options should include what variables that should be passed from the current scope to the remote scope (identified using patterns) and also possibly new variables to add to the remote scope.
If the datadir
option specifies an absolute path, then the lookup command will append that absolute path to your current working directory
Hi - I wanted to add a new lookup key provider (for Azure Key Vault). I've created the function in the provider package but the options
map is empty when the function is called. Is this functionality implemented yet?
func AzureKeyVaultLookupKey(c hieraapi.ProviderContext, key string, options map[string]px.Value) px.Value {
vaultName, ok := options[`vault_name`]
if !ok {
panic(px.Error(hieraapi.MissingRequiredOption, issue.H{`option`: `vault_name`}))
}
When running this always fails with Missing required provider option 'vault_name'
The hiera.yaml is
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: secret
lookup_key: azure_key_vault
options:
vault_name: xxx
If a plugin backend has 2 entries in the hiera.yaml then lookups from the second one fail
hiera.yaml
version: 5
hierarchy:
- name: one
data_hash: terraform_backend
options:
backend: local
config:
path: C:\Temp\hiera\one.tfstate
- name: two
data_hash: terraform_backend
options:
backend: local
config:
path: C:\Temp\hiera\two.tfstate
one.tfstate
{
"version": 3,
"serial": 1,
"modules": [
{
"path": [
"root"
],
"outputs": {
"one": {
"sensitive": false,
"type": "string",
"value": "one"
}
},
"resources": {},
"depends_on": []
}
]
}
two.tfstate
{
"version": 3,
"serial": 1,
"modules": [
{
"path": [
"root"
],
"outputs": {
"two": {
"sensitive": false,
"type": "string",
"value": "two"
}
},
"resources": {},
"depends_on": []
}
]
}
lookup.exe one
>one
lookup.exe two
>Error: lookup() did not find a value for the name 'two'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.