In addition to the --facts flag (which needs an input file in yaml or json), it'd be cool if the lookup command had a way to pass fact name/value on the command line. This would make it easy and fast to test hierarchy or data changes. I'd suggest a syntax like

lookup --fact name1=value1 --fact name2=value2

hi,

literal('%') is not functional

usecase: test to lookup a var with the following definition

---
test: "%{literal('%'}{TEST}"

Results:

• within the unit tests inside the hiera package, it's working
• within the unit tests inside the "lookup" package, it's broken

it outputs an empty string, as if the resulting string were interpolated twice.

to make it work you have to escape the following way:

---
test: "%{literal('%')}{literal('%{literal('%')}')}{TEST}" 

• it seems that it

I think I'm seeing something similar to this - for some lookups I get back __ref: 26 or similar on the first try, but retrying the lookup returns the correct value. I can't figure out exactly what's triggering it, but I think it's only when using strict_alias. Also passing --dialect pcore seems to make it work first time.

Currently, it seems to be necessary to specify data_dir per heirarchy key; if it is only used in the defaults key, hiera does not prepend it and fails to find the data files.

Given a hierarchy like:

---
version: 5
defaults:
  data_dir: hiera
  data_hash: yaml_data

hierarchy:
  - name: "Environmental overrides"
    path: "environments/%{environment}.yaml"
  - name: "Fall through defaults"
    path: "defaults.yaml"

The debug messages show:

019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="deep merge strategy" provider="data_hash function 'yaml_
data'" location="path{ original:environments/%{environment}.yaml, resolved:/Users/eric/Sandbox/lyra-local/environments/.yaml, exist:false}"
2019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="deep merge strategy" provider="data_hash function 'yaml_
data'" location="path{ original:defaults.yaml, resolved:/Users/eric/Sandbox/lyra-local/defaults.yaml, exist:false}"
2019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="first found strategy" provider="data_hash function 'yaml
_data'" location="path{ original:environments/%{environment}.yaml, resolved:/Users/eric/Sandbox/lyra-local/environments/.yaml, exist:false}"           2019-05-20T06:17:35.596-0700 [DEBUG] lyra: location not found: key=wordpress.location context="first found strategy" provider="data_hash function 'yaml_data'" location="path{ original:defaults.yaml, resolved:/Users/eric/Sandbox/lyra-local/defaults.yaml, exist:false}"
2019-05-20T06:17:35.597-0700 [ERROR] lyra: apply failed: Error="lookup() did not find a value for the name 'wordpress.location' (file: /private/tmp/lyr
a-20190520-79576-p2kmxa/lyra-0.2.0/.brew_home/go/pkg/mod/github.com/lyraproj/[email protected]/internal/init.go, line: 106)

hiera.yaml

version: 5
defaults:
  data_hash: yaml_data
  datadir: data
hierarchy:
  - name: global
    path: global.yaml

global.yaml

empty_map: {}

lookup empty_map --render-as json
Error: json: unsupported type: map[interface {}]interface {}

After the changes in #60 , I cannot start hieraserver either standalone or in a container.

I think the problem is this bit:

Because makeTLSConfig() creates a new tls.Config instance before it checks whether ssl keys and certs are passed in, the variable is never nil; it contains an instantiated but non-working tls configuration.

One of the super helpful things about puppet-hiera is the puppet lookup --explain functionality, that shows the provenance of a value as it's looked up in the hierarchy. It's be great to have something similar in go-hiera. For reference, explain output looks like this (we don't need to exactly match this, IMO it's a bit too "busy" and could stand some simplification and maybe colorisation):

Searching for "openssl::package_version"
  Global Data Provider (hiera configuration version 5)
    No such key: "openssl::package_version"
  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Hierarchy entry "Classifier Configuration Data"
      No such key: "openssl::package_version"
    Hierarchy entry "Datacenter"
      Path "/etc/puppetlabs/code/environments/production/hieradata/datacenter/.yaml"
        Original path: "datacenter/%{::datacenter}.yaml"
        Path not found
    Hierarchy entry "PE instance secrets"
      Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/pe-kearney.p9.puppet.net.eyaml"
        Original path: "nodes/%{fqdn}.eyaml"
        Path not found
    Hierarchy entry "common-e"
      Path "/etc/puppetlabs/code/environments/production/hieradata/common.eyaml"
        Original path: "common.eyaml"
        No such key: "openssl::package_version"
    Hierarchy entry "common"
      Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml"
        Original path: "common.yaml"
        Found key: "openssl::package_version" value: "1.0.1e-60.el7_3.1"

Since the latest update the lookup command seems to be dumping the plugin meta information to stderr:

& lookup.exe testy
2020/03/04 14:18:30 {"version":1,"network":"unix","address":"C:\\Users\\HENRY~1.BUC\\AppData\\Local\\Temp\\terraform_backend.exe-8280.socket","functions":{"data_hash":{"terraform_backeend"}}}
2020/03/04 14:18:30 {"backend":"local","root_key":"root","config":{"path":"one.tfstate"}}
2020/03/04 14:18:30 {"root":{"two":"two","three":"two","one":"one"}}
2020/03/04 14:18:30 value
value

I've tried debugging the lookup command but I can't see what's writing it

I need to do the following in golang code:

• perform a lookup of keys as directed by an "environment" level hiera.yaml (my app to provide the path)
• then, perform a lookup of a module specific key in each module on the modulepath - need to loop over the lookups so happy to accept having to set up a new instance per module. (My app to provide the path)
• In each lookup I need to be able to set a number of facts/variables for hiera to interpolate.
• I want to be able to do a deep merge of what I looked up in the same way as hiera does this. (My app will make these calls as it is looking up different keys that it merges - for example mykey, module_a::mykey, module_b::mykey - I am not expecting hiera to do that for me).
• I want to be able to (optionally) turn on "explain" output to allow users to debug these lookups
• I only need standard backends (yaml)
• I would like the hiera.yaml files involved to look exactly the same as in the Ruby version (to allow the ruby version of my application to use exactly the same configuration and data files).

Trying out hiera 0.3 and getting an error with plugins

version: 5
defaults:
  plugindir: C:\Temp\hiera\plugins
hierarchy:
  - name: one
    data_hash: terraform_backend
    options:
      backend: local
      config:
        path: C:\Temp\hiera\one.tfstate

lookup.exe test
Error: unable to start plugin /C:/Temp/hiera/plugins/terraform_backend.exe: exec: "/C:/Temp/hiera/plugins/terraform_backend.exe": file does not exist

Looks like (dgo.Loader).AbsoluteName() is returning the path with / prefixed

Comes from #78

Given a facts.yaml file:

---
inventory: 'a'

For people like me coming from puppet, it's a bit weird/consufing that using --facts facts.yaml results in a different interpolation when using lookup (inventory=a) and when using puppet lookup (facts.inventory=a).

I feel that it should be at least a little documented somewhere (maybe it is and I haven't found it) or, for an easiest transition between tools, may current go implementation copy all facts to make them also available as facts map subkeys.

To give some background in our use case, we come from using Puppet + some scripting with hiera as our main source of truth and now we're replacing our scripting with tools like terraform but want to keep our hiera hierarchy as much as posible "as is".

Thanks for alll your fantastic work

/cc @thallgren

The key to hiera's operation is the hiera.yaml which lets users configure their own hierarchies. This implementation should support a minimal variant of the Ruby hiera version - path names with (optional) variable interpolation in order, falling through to a common baseline.

Any chance of tagging up some releases for this and the plugins?
Thanks

When getting an error like this:

panic: reflect: call of reflect.Value.SetFloat on int32 Value [recovered]
	panic: reflect: call of reflect.Value.SetFloat on int32 Value

When calling vf.FromValue(v, target) it is time consuming to figure out where the problem is when there are multiple int32 values in the target structure.
Suggest adding the name of the type/member that it is failing on in the error message. There should probably be a general catch that amends all error messages.

The workflow to build and publish docker image runs on every push, regardless of repository. In my thallgren fork, the build fails because I don't have the correct credentials to publish (which is probably for the good in this case).

Example of failure: https://github.com/thallgren/hiera/runs/284812108

hiera.yaml

version: 5
hierarchy:
  - name: yaml
    path: yaml1.yaml

yaml1.yaml

yaml1: "%{lookup('nothing')}"

> lookup yaml1
Error: runtime error: invalid memory address or nil pointer dereference

Previously this used to return undef

Add a standalone command line tool similar to puppet lookup.

I created a an AWS SSM Parameter Store plugin. It works, so I can write secrets to the parameter store and reference them from Hiera. An example hiera.yaml configuration that uses the plugin looks like this:

---
version: 5

defaults:
  plugindir: ./plugins
  datadir: ./data
  data_hash: yaml_data

hierarchy:
  - name: region
    path: region/%{region}.yaml

  - name: environment
    path: environment/%{environment}.yaml

  - name: common
    glob: defaults/*.yaml

  - name: secrets
    lookup_key: "aws_ssm_parameter"
    options:
      aws_profile_name: production.AdministratorAccess
      aws_region: "us-west-1"

What I had hoped to do was to use interpolation of %{environment} and %{region} in the options section, like this:

  - name: secrets
    lookup_key: "aws_ssm_parameter"
    options:
      aws_profile_name: "%{environment}.AdministratorAccess"
      aws_region: "%{region}"

This would enable Hiera to seamlessly switch between AWS accounts and regions based on these "facts" (or in the case of the Terraform Hiera provider, based on provider scope). Unfortunately, interpolation doesn't work for options so I must hard-code them. This forces me to store all parameters within a single AWS account and region.

Hiera finds configured lookup functions (providers) using a loader. New functions can be added using the pcore px.NewGoFunction(). This will however require that the function is compiled together with the rest of the Hiera binary (#34 is a good example). A better approach would be if such functions were reachable using some plug-in framework that could use a Hiera SDK and then made available as Go plugins (shared libraries / same address space) or as RESTful binaries (see #32).

Adding the Azure lookup function alone increased the go.sum file from 75 to 227 lines, corresponding to a transitive dependency tree that is three times as large. It can be expected that other types of lookup functions will cause a similar increase in dependencies.

The current git action for testing tests on linux only. We need to run tests on windows too.

As Go doesn't install modules by default, the documentation doesn't work as shown.

$ go get github.com/lyraproj/hiera
package github.com/lyraproj/hiera: no Go files in /Users/jrhett/go/src/github.com/lyraproj/hiera

$ go install github.com/lyraproj/hiera/lookup
../../go/src/github.com/lyraproj/hiera/internal/location.go:8:2: cannot find package "github.com/bmatcuk/doublestar" in any of:
	/usr/local/Cellar/go/1.12.6/libexec/src/github.com/bmatcuk/doublestar (from $GOROOT)
	/Users/jrhett/go/src/github.com/bmatcuk/doublestar (from $GOPATH)

I suspect many potential users are not aware enough of Go's module management to do the clone of the repo and install the modules, so at least mentioning it is probably a good idea. Or outright mention GO111MODULE=on go get github.com/lyraproj/hiera as the quick fix ;)

Was looking at possibly using this instead of the gem counterpart. Love the direction so far, but think I'm blocked as we use eyaml as a way of encrypting secrets.

Do you have a mechanism that accomplishes this or is there plans to support such a plugin?

Hello! We are on hiera version v0.2.0.
We are parsing a fact into a JSON output format and noticed a parsing error.

When parsing a yaml list like

myList:
  - typeA: a
  - typeB: b
  - typeC: c
  - tupeD: d

hiera will parse it as

[{"typeA":"a"},{"typeB":"b"},{"typeC":"c"}:{"tupeD":"d"}]

Note that what should have been a comma delimiter is now a :!

We have the go code to parse from the yaml file:

	var variables []string
	for _, fact := range facts {
		variables = append(variables, fact.String())
	}

	cmdOpts.Merge = "deep"
	cmdOpts.Variables = variables
	cmdOpts.RenderAs = formatToFormatString(format)

	configOptions := map[string]px.Value{
		provider.LookupKeyFunctions: types.WrapRuntime([]hieraapi.LookupKey{provider.ConfigLookupKey, provider.Environment})}
	configOptions[hieraapi.HieraConfig] = types.WrapString(path.Join(directory, "hierav5.yaml"))

	dest := bytes.NewBuffer(nil)
	err := hiera.TryWithParent(context.TODO(), provider.MuxLookupKey, configOptions, func(c px.Context) error {
		hiera.LookupAndRender(c, &cmdOpts, []string{key}, dest)
		return nil
	})

I'm trying to merge multiple yaml files. Since I haven't found out how I can read everything in and process it, I process everything section to section. That goes well so far, but now I'm slowly getting problems that I don't understand. That's why I would like to know how it is intended and how it works.

I'll explain (even if it can be easier or more elegant) how I do it and where my problem is.

I have a struct, in which I would like to save the result later in order to be able to process it further.

hiera.DoWithParent(context.Background(), provider.ConfigLookupKey, configOptions, func(hs api.Session) {
    result := hiera.Lookup(hs.Invocation(nil, nil), section, nil, map[string]string{`merge`: `deep`})
    s := streamer.New(nil, nil)
    b := bytes.Buffer{}
    s.Stream(result, streamer.JSON(&b))
    if err = json.Unmarshal(b.Bytes(), &c.Hosts); err != nil { ...... }
}

Do I only have one host e.g. in the yaml, it's fine. However, if I have anchors, I sometimes get references that are lost when restoring or cause panic because they do not match my struct.

{
    "host2":
            {
             ....
             "data":[{"__ref":9}]
             ....
             }
}

panic: json: cannot unmarshal object into Go struct field Host.Data of type string

If my investigations are correct, then this reference comes from dgo, I have seen in streamer there is a function unmarshalJSON that requires a type dialect, I have not yet understood what this dialect is and whether I can convention dgo.Value after dialect.

Can someone please explain to me the best way to look up my files. Get the full change back?
At the moment I am just doing a lookup on a certain value and have it returned to me and read this into the struct. It would of course be ingenious if it were possible to receive the complete change over the files and to write them to the struct in one go.
Thank you in advance for your help.

Hi there, we'd like to be able to either:

• Query Hiera to get a list all the (top level) keys it knows about.
• Or ideally get a dump of all Hiera values as Json.

Why?
Because we don't know what keys will exist in the various yaml files ahead of time.

Example:
Let's say we have files:

\hiera\hiera.yaml:
  defaults:
    datadir: data
 hierachy:
  - name: per_environment
    path: per_environment/%{environment}.yaml

\hiera\data\global.yaml:
  key1: abc
  key2: def

\hiera\data\per_environment\staging.yaml
  key2: xyz

\hiera\data\per_environment\production.yaml
  key2: pqr

We'd like to say something like:
lookup --get-dump-of-all-values --render-as json --vars facts.json

If we specify an environment of production we'd expect to get output:
{
key1: abc
key2: pqr
}

If we don't specify any environment we'd expect to get output:
{
key1: abc
key2: def
}

If you can suggest a way we can do this with existing functionality that'd be fab, otherwise would it be possible to add this as a feature?
Thanks very much.

global.yaml:

mynull: null

mymap:
  mykey: null

.\lookup.exe mynull
null

.\lookup.exe mymap
Error: reflect: call of reflect.Value.Type on zero Value

Summary

Many teams who use Puppet to manage server-based infrastructure are also starting to manage cloud-native application delivery including the needed infrastructure. Usually, those teams have codified configuration data in hiera for the Puppet node-based configuration processes, but they cannot use this company-wide configuration data in a cloud-native deployment pipeline. If Hiera would be available as a service, hiera lookups could be easily done outside the "traditional Puppet context", for example from a delivery pipeline, and enable teams to define configuration data only once - inside hiera.

Proposed approach

1. Finalize clean room implementation of hiera including pluggable backend system and a yaml backend
2. Develop a network front-end to hiera (essentially introducing hiera-server)
3. Develop a pluggable RBAC system to the hiera server

Note: HAAS means "hare" in Dutch

vf.FromValue does not handle a target with map[string]*X

It will crash with an error that it cannot do reflection on nil.

Me again :)
I can't seem to get the lookup interpolation function to work when using the terraform backend plugin, however lookups using the command line are working. This works using the pre-plugin single binary version

hiera.yaml

version: 5
defaults:
  data_hash: yaml_data
  datadir: data
hierarchy:
  - name: tfstate
    data_hash: terraform_backend
    options:
      backend: azurerm
      config:
        storage_account_name: xxx
        container_name: tfstate
        access_key: xxx
        key: xxx
  - name: global
    path: global.yaml

global.yaml

test: "%{lookup('vnet_name')}"

.\lookup.exe vnet_name --explain
Searching for "vnet_name"
  data_hash function 'terraform_backend'
    Found key: "vnet_name" value: 'testlemur'

.\lookup.exe test --explain
Searching for "test"
  Merge strategy "first found strategy"
    data_hash function 'terraform_backend'
      No such key: "test"
    data_hash function 'yaml_data'
      Path "C:\Temp\hiera\data\global.yaml"
        Original path: "global.yaml"
        Interpolation on "%{lookup('vnet_name')}"
          Searching for "vnet_name"
            Merge strategy "first found strategy"
              data_hash function 'terraform_backend'
                No such key: "vnet_name"
              data_hash function 'yaml_data'
                Path "C:\Temp\hiera\data\global.yaml"
                  Original path: "global.yaml"
                  No such key: "vnet_name"
        Found key: "test" value: 'undef'
    Merged result: 'undef'

For compatibility with the puppet hiera docs, data_dir needs to be renamed to datadir.

https://puppet.com/docs/puppet/5.0/hiera_config_yaml_5.html#the-defaults-key

It would be very convenient if LookupInto could handle time.Duration such that an integer value is interpreted as a nanosecond timestamp and that a string is interpreted in the time.ParseDuration string format (for example "14h52m"). At present this ends with a panic panic: reflect: call of reflect.Value.SetString on int64 Value.

The only current work around is to use string fields and post process the looked up result.

A url based lookup_key function should be added so that one Hiera configuration can include lookups in Hiera servers that resides on other hosts. The options should include what variables that should be passed from the current scope to the remote scope (identified using patterns) and also possibly new variables to add to the remote scope.

If the datadir option specifies an absolute path, then the lookup command will append that absolute path to your current working directory

Hi - I wanted to add a new lookup key provider (for Azure Key Vault). I've created the function in the provider package but the options map is empty when the function is called. Is this functionality implemented yet?

func AzureKeyVaultLookupKey(c hieraapi.ProviderContext, key string, options map[string]px.Value) px.Value {
	vaultName, ok := options[`vault_name`]
	if !ok {
		panic(px.Error(hieraapi.MissingRequiredOption, issue.H{`option`: `vault_name`}))
	}

When running this always fails with Missing required provider option 'vault_name'

The hiera.yaml is

---
version: 5
defaults:
  datadir: data
  data_hash: yaml_data
hierarchy:
  - name: secret
    lookup_key: azure_key_vault
    options:
      vault_name: xxx

If a plugin backend has 2 entries in the hiera.yaml then lookups from the second one fail

hiera.yaml

version: 5
hierarchy:
  - name: one
    data_hash: terraform_backend
    options:
      backend: local
      config:
        path: C:\Temp\hiera\one.tfstate
  - name: two
    data_hash: terraform_backend
    options:
      backend: local
      config:
        path: C:\Temp\hiera\two.tfstate

one.tfstate

{
  "version": 3,
  "serial": 1,
  "modules": [
    {
      "path": [
        "root"
      ],
      "outputs": {
        "one": {
          "sensitive": false,
          "type": "string",
          "value": "one"
        }
      },
      "resources": {},
      "depends_on": []
    }
  ]
}

two.tfstate

{
  "version": 3,
  "serial": 1,
  "modules": [
    {
      "path": [
        "root"
      ],
      "outputs": {
        "two": {
          "sensitive": false,
          "type": "string",
          "value": "two"
        }
      },
      "resources": {},
      "depends_on": []
    }
  ]
}

lookup.exe one
>one
lookup.exe two
>Error: lookup() did not find a value for the name 'two'