[data/values] Ability to read in multiple separate "values" files about ytt

[cppforlife]

current thinking is to mark overlay documents with @data/values as well

#@ load("@ytt:overlay", "overlay")
#@ load("@github.com:blah", "blah")
#@data/values
#@overlay/match ...
---
#@overlay/match missing_ok=True
key: foo
#@overlay/match missing_ok=True
blah: #@ blah.something

not sure though how to indicate it's an overlay if @overlay/match doesnt take any args.

marking overlays with data/values helps to identify which files should be templated to obtain data.values.

[cppforlife]

after chatting extensively with @nimakaviani we've come to the following solution to this problem that we are planning to implement as a first cut:

• multiple @data/values could be specified
• data/values are overlayed on top of another, with following ordering rules:
  • follow -f order provided on the command line from left to right
  • if -f specifies a directory, then use existing alpha numeric file name order

ill try to implement first cut of this tomorrow. example from the issue description would look something like this:

ytt -f template.yml -f envs/values.yml -f envs/aws/values.yml -f envs/aws/dev/values.yml

[cppforlife]

v0.13.0 is shipped: https://github.com/k14s/ytt/releases/tag/v0.13.0. it implements above comment.

[cppforlife]

@nimakaviani and i earlier on were discussing offering ability to specify overlay configuration for data/values. it could look something like this:

values.yml

#@data/values
---
val1: one
val2: two

aws/dev/values.yml

#@ load("@ytt:data", "data")
#@overlay/match by=data.values_document
---
val1: two
#@overlay/remove
val2:

which would ultimately result in @data/values:

val1: two

before processing rest of the templates.

wdyt? it sounds like it would allow you to

[cppforlife]

reminder to self: ordering of overlays was in question here.

[MrTrustworthy]

Some comments from Slack moved into here for discussion purposes:

---

The way i’d assumed it would work at the beginning was that i could “name” my various data files. Example: my globals file would declare #@data/globals and my app config file would declare #@data/myapp, and in the templates i could #@ load("@ytt:data", "globals") and #@ load("@ytt:data", "myapp") as needed, with the order of loading determining overlay order behaviour

having non-overlapping files would, in this implementation, be straightforward. in case of overlapping configs, this allows the user to modify order-of-overwriting on a template-by-template case

benefit of having one automatic way to merge data/values is that from the templates perspecive data comss from one place instead of leaking notion of global vs app into templates

True. You could even circumvent the entire problem by just preventing overlaying in those cases completely: make the #@ load("@ytt:data", "globals") and #@ load("@ytt:data", "myapp") statements inject the values directly into a sub-namespace so you’d have to call data.values.globals.x or data.values.myapp.x explicitly. then you’d have to specify in your templates on a statement-by-statement basis how you want overwriting, ex with default(data.values.globals.x, data.values.myapp.x) (sorry if the syntax is wrong, not familiar with ytt syntax yet).

---

IMHO, there are two general approaches to solve decomposition of value files:

1. Handle overlays explicitly, either via order-of-imports, additional CLI arguments, special annotations, dedicated functions (ex. merge(values.globals, values.myapp)) or otherwise. Upside: this solves both decomposition and overlaying with one stroke, which makes it easy for users. Downside: Users must be aware of the behaviour or risk running into surprises.
2. Explicitly don't handle overlaying, only focus on decomposition. This could be done by injecting the different value files into different namespaces based on their declaration (ex. declaring a file with #@data/myglobals and loading it with #@ load("@ytt:data", "myglobals") injects the values of this file into values.myglobals). Upsides: This is a somewhat natural behaviour based on imports in programming languages that people may naturally expect. Downsides: As it doesn't handle overlaying on a per-template or per-deployment level, users must explicitly use something like IF value.myglobals.x THEN value.myglobals.x ELSE value.myapp.x each time in case they want to have overlay behaviour.

[stereobutter]

Any progress on getting values from multiple files via namespaces as suggested in this comment above? I actually think this is rather more useful that the current implementation which seems to be somewhat redundant with overlays?

[cppforlife]

@SaschaSchlemmer

Any progress on getting values from multiple files via namespaces as suggested in this comment above? I

we have no plans at the moment to change how data values are ingested. we currently support usage of any number of data values files (or other variations like env vars, etc.) per this doc description: https://github.com/k14s/ytt/blob/develop/docs/ytt-data-values.md#splitting-data-values-into-multiple-files

I actually think this is rather more useful that the current implementation which seems to be somewhat redundant with overlays?

key thing to data values is that they are commonly modified, overriden by various "layers" (common, dev, prod, etc.) so there need to be a mechanism for merging them. overlays provide that mechanism. with regard to separation them into "namespaces" you can easily achieve that by adding top level keys under data values and nesting child keys under those. in that regard ytt allows you, the user, to structure data values in whatever way it makes sense.