carvel-dev / ytt Goto Github PK

Input:

x: #@ { "first": "is included"}
  second: is ignored

The second line is silently ignored.

Also here:

--- #@ { "first": "is included"}
this line: is ignored too

If this is not valid, I'd expect an error message instead of silently ignoring part of the input.

in example-overlay-files, ops2.yml adds multiple map items (under annotations key) [1]. it has to specify missing_ok=True for per key because to indicate that it's ok to create-or-update. need to come up with a way to allow skipping such annotation as it's pretty verbose.

few options:

• somehow annotate multiple nodes together
• annotate that children of a particular node are ok with missing_ok=True (probably an easier and better option)

[1] https://github.com/vmware-tanzu/carvel-ytt/blob/develop/examples/playground/basics/example-overlay-files/ops2.yml#L7-L10

there might be cases where instead of removing, merging or replacing it might be necessary to modify yaml node. one simple example might be:

#@ def incr_by_2(val): return val + 2

spec:
  #@overlay/modify incr_by_2
  replicas:

to increment replicas by two in a particular environment. this assumes that base value is meaningful and relative values (or modification of base values in general) makes "business" sense.

@scothis brought another example where he was overlaying some data on top of knative manifests (https://github.com/knative/serving/blob/d7f29bd1a436bd651b2fa5ecf042ece6bf7cb09d/config/config-network.yaml#L105) and wanted to escape {{ as they would not be compatible with helm (he was trying to create helm chart on the fly).

originally @patricknelson added as a comment to related issue: #14 (comment):

It's not always essential for the resulting YAML to have comments for me, but it certainly would be very helpful!

What if we were to use a double # to work as an escape method when at the beginning of a line, so that the resulting YAML will contain a standard comment? For example, taken from the main demo at https://get-ytt.io:

#@ def labels():
#! This is a ytt comment
## This is a yaml comment
app: echo
org: test
#@ end

kind: Pod
apiVersion: v1
metadata:
  name: test-comments
  labels: #@ labels()

With the output resulting in:

kind: Pod
apiVersion: v1
metadata:
  name: echo-app
  labels:
    # This is a yaml comment
    app: echo
    org: test

I'm brand new to ytt (just doing research right now), so I'm asking just in case this isn't already available with first class syntax support without any special flags.

currently following works

$ ytt --data-values-inspect -f values.yml=<(echo -e "#@data/values\n---\nstores: [1, 2, 3]")
stores:
- 1
- 2
- 3

however, it would be nice to have following work as well:

$ ytt --data-values-inspect --data-values-file-yaml <(echo "stores: [1, 2, 3]")
stores:
- 1
- 2
- 3

Related Conversations:

• https://kubernetes.slack.com/archives/CH8KCCKA5/p1567035583060800
• https://kubernetes.slack.com/archives/CH8KCCKA5/p1605718554315200

Hi!

The current documentation on @overlay/replace says that it is "valid for both map and array items", however I was able to successfully apply it to a whole document as well. For example:

#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({'foo': 'bar'}), expects="0+"
#@overlay/replace
---
foo: "baz"
note: "I am replacing the YAML document!"

The ability of replacing the YAML document is useful to me. Can I safely use it? If so, I think it would be nice to update the documentation.

In ytt, there is the possibility to load values from files, but it does not seem that it is possible to load another YAML file from the main ytt file.

Could this be considered as an addition?

Something like this:

main.yaml

---
test: #@ include("hello.yaml")

hello.yaml

---
#@ def world()
"world"
#@ end

hello: #@ world()

the result would be:

---
test:
  hello: "world"

Having a way to pass variables to the included file would be nice. What this is is basically a #@ load() but with the function definition implicit in the included file.

I often use process substitution, especially if I'm bringing down secrets using a vault tool like lpass/credhub.

I want to use the command: ytt template -f pipeline.yml -f <(..cmd-that-gets-secrets..) but then I hit this:

https://github.com/k14s/ytt/blob/4b7b67ef6d88cc4777d38e9fbd310daa567961fa/pkg/files/file.go#L203

Eg:

$ ytt template -f <(echo "---")
Error: Expected file '/dev/fd/63' to be a regular file, but was not

Is this really just guarding against directories?

consider mentioning:

• https://github.com/bitnami-labs/sealed-secrets
• https://github.com/k14s/ytt/blob/develop/docs/ytt-data-values.md

Hi there!

According to Starlak's spec on the load statement, the first "argument" is left to be interpreted by the application (ytt itself in our case). I think it might be a good idea to add a note in the docs explaining how ytt interprets that.

I'm raising this issue because I couldn't find in the documentation and examples information regarding a problem I faced while using load. My use case requires passing yml files individually via ytt -f, and it took me a while to figure that I have to pass the included file via -f as well.

For example, if I go to example-load and run the following command, it would fail:

$ ytt -f config.yml
Error: 
- cannot load funcs.lib.yml: Expected to find file funcs.lib.yml
    config.yml:5 in <toplevel>
     L #@ load("funcs.lib.yml", "func1", "func2")

Now, if I include the files to the list, it works properly:

$ ytt -f config.yml -f funcs.star  -f funcs.lib.yml 
func1_key:
  name: max
  cities:
  - SF
  - LA
func2_key:
  name: joanna
  cities:
  - SF
func3_key:
- 1
- 2
- key: value

It would be nice if there was documentation on the load function (and maybe on ytt CLI as well 😃).

Note: it seems that #9 is a related issue, but it talks about improving the error message but not about the project's documentation.

Best regards,
Gustavo Sousa

It'd be nice to allow ytt expressions in the keys of a yaml dict. E.g., see the file below, the direct attempt fails miserably, to get the example to work I have to jump through hoops with functions and template.replace.

#! This doesn't work, but would be convenient

#@ for x in [1]:
? #@ "key" + str(x)
:
  - #@ x+1
  - #@ x+2
#@ end

---

#! This does work, but it's kinda clunky.

#@ load("@ytt:template", "template")

#@ def f(x):
- #@ x+1
- #@ x+2
#@ end

---

#@ for x in 1, 2, 3, 4:
_: #@ template.replace({ "key" + str(x) : f(x) })
#@ end

based on @ctaymor's feedback i think we should find out better content that explains why someone should use ytt instead of erb/jinja/etc on the home page.

For example as part of linkerd installation I wanted to remove strategy and status fields on Deployments. To be more conservative we should probably allow value matching as well to avoid cases when key is being removed but it has an unexpected value (in this example i expected values to be {}).

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"kind":"Deployment"}),expects="1+"
---
spec:
  #@overlay/remove
  strategy: null
#@overlay/remove
status: null

#! Trying to use the multiline yaml ? : notation for map entries doesn't give the expected results:

#! This works
? ac
: bd

#! But this does not (it gives ee:null not ee:fg)
? ee
: #@ 'f' + 'g'

#! Nor this (it gives null:ax not ax:bd)
? #@ 'a' + 'x'
: #@ 'b' + 'd'

#! This one gives an error if you uncomment it.
#! ? #@ 'y' + 'z'
#! : asdf

When trying to use a multiline comment in an anchor, ytt panics:

sth: &use 
  path:
  args:
  - |
    foo
here: *use

error: panic: expected len of sequence children to match len of children line nums [recovered]
	panic: expected len of sequence children to match len of children line nums

goroutine 1 [running]:
github.com/k14s/ytt/pkg/yamlmeta/internal/yaml%2ev2.handleErr(0xc00011d6a8)
	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/yamlmeta/internal/yaml.v2/yaml.go:297 +0x9a
panic(0x8acac0, 0xa81900)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/k14s/ytt/pkg/yamlmeta/internal/yaml%2ev2.(*decoder).sequence(0xc000094a40, 0xc0001322a0, 0x8c8a40, 0xc0001089a0, 0x194, 0x8c8a40)
	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/yamlmeta/internal/yaml.v2/decode.go:565 +0xa71
github.com/k14s/ytt/pkg/yamlmeta/internal/yaml%2ev2.(*decoder).unmarshal(0xc000094a40, 0xc0001322a0, 0x8c8a40, 0xc0001089a0, 0x194, 0x194)
	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/yamlmeta/internal/yaml.v2/decode.go:344 +0x105
github.com/k14s/ytt/pkg/yamlmeta/internal/yaml%2ev2.(*decoder).mappingSlice(0xc000094a40, 0xc0001320e0, 0x8bfee0, 0xc00007acc0, 0x197, 0x197)
	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/yamlmeta/internal/yaml.v2/decode.go:696 +0x4bd
github.com/k14s/ytt/pkg/yamlmeta/internal/yaml%2ev2.(*decoder).mapping(0xc000094a40, 0xc0001320e0, 0x8c8a40, 0xc000108910, 0x194, 

the ytt user may wish to provide data values in the form of environment variables or by passing values through the command line. this is particularly beneficial when providing secrets.

As a user of ytt I expect it to provide me with either of the options above to provide or override values.

It would come in very handy to be able to call a function for url encoding strings.

Use Case:
A liveness probe queries a rest endpoint which requires a parameter that contains speical characters.

Example:

#@load("@ytt:url", "url")

livenessProbe:
    httpGet:
        path: #@ "/query?param={}".format(url.escape("I'm URL encoded!"))

Of course exact naming doesn't matter.

Should result in

livenessProbe:
    httpGet:
        path: "/query?param=I%27m%20URL%20encoded%21"

The following example is causing a crash:

#@ def func1():
hello: world
---
new: document
#@ end

--- #@ func1()

The error:

- (p) expected to find node tag 1 when unwinding (backtrace: goroutine 1 [running]:
    config.yml:3 in func1
     L ---
    config.yml:7 in <toplevel>
     L --- #@ func1()

    reason:
     runtime/debug.Stack(0x8d80a0, 0x8a6240, 0xc000116610)
     	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1.1(0xc00012e3f0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/core/errs.go:20 +0x1c4
     panic(0x8a6240, 0xc000116610)
     	/usr/local/go/src/runtime/panic.go:522 +0x1b5
     github.com/k14s/ytt/pkg/template.(*EvaluationCtx).unwindToTag(0xc000100690, 0x1)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/evaluation_ctx.go:237 +0x10d
     github.com/k14s/ytt/pkg/template.(*EvaluationCtx).startNode(0xc000100690, 0x7, 0x7, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/evaluation_ctx.go:189 +0x163
     github.com/k14s/ytt/pkg/template.(*EvaluationCtx).TplStartNode(0xc000100690, 0xc0000e3050, 0xc0000e3170, 0xc000116600, 0x1, 0x1, 0x0, 0x0, 0x0, 0x8, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/evaluation_ctx.go:157 +0x179
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).tplStartNode(...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/compiled_template.go:241
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1(0xc0000e3050, 0xc0000e3170, 0xc000116600, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/core/errs.go:25 +0xe9
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Builtin).CallInternal(0xc0000e3170, 0xc0000e3050, 0xc000116600, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc000116600, 0xdbfca0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/value.go:622 +0x8b
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.Call(0xc0000e3050, 0xa8f7c0, 0xc0000e3170, 0xc000116600, 0x1, 0x1, 0x0, 0x0, 0x0, 0xa90080, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:968 +0x154
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.call(0xc0000e3050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0xc00012eb20, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/interp.go:292 +0x405a
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Function).CallInternal(0xc000100620, 0xc0000e3050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xa90080, 0xdbfca0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/interp.go:39 +0x1c1
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.Call(0xc0000e3050, 0xa8f840, 0xc000100620, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xa90080, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:968 +0x154
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.call(0xc0000e3050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x70, 0xc0001004d0, 0xc00012f240, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/interp.go:292 +0x405a
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Function).CallInternal(0xc0001004d0, 0xc0000e3050, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x77c4b9, 0x9245a0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/interp.go:39 +0x1c1
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.Call(0xc0000e3050, 0xa8f840, 0xc0001004d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:968 +0x154
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Program).Init(0xc00000e240, 0xc0000e3050, 0xc0000e30b0, 0x0, 0x0, 0xc000066840)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:340 +0x9a
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).eval(0xc000066780, 0xc0000e3050, 0xc0000e30b0, 0x0, 0x0, 0x0, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/compiled_template.go:140 +0x203
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).Eval(0xc000066780, 0xc0000e3050, 0xa8e100, 0xc0000e27b0, 0xc0000e3050, 0xc0000e3020, 0xc0000d3f80, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/compiled_template.go:101 +0x719
     github.com/k14s/ytt/pkg/workspace.(*TemplateLoader).EvalYAML(0xc0000e27b0, 0xc00001e460, 0xc0000628c0, 0xc000027328, 0xc0000e2780, 0x0, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/workspace/template_loader.go:160 +0x66d
     github.com/k14s/ytt/pkg/cmd/template.(*TemplateOptions).RunWithFiles(0xc00009a2c0, 0xc00000e0d0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/template/cmd.go:109 +0xfef
     github.com/k14s/ytt/pkg/cmd/template.(*TemplateOptions).Run(0xc00009a2c0, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/template/cmd.go:81 +0x292
     github.com/k14s/ytt/pkg/cmd/template.NewCmd.func1(0xc000091680, 0xc0000e21e0, 0x0, 0x3, 0xb, 0xb)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/template/cmd.go:54 +0x2a
     github.com/k14s/ytt/pkg/cmd.reconfigureLeafCmd.func1(0xc000091680, 0xc0000e21e0, 0x0, 0x3, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/ytt.go:90 +0x1b7
     github.com/k14s/ytt/vendor/github.com/cppforlife/cobrautil.WrapRunEForCmd.func1.1(0xc000091680, 0xc0000e21e0, 0x0, 0x3, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/cppforlife/cobrautil/misc.go:25 +0xaf
     github.com/k14s/ytt/vendor/github.com/spf13/cobra.(*Command).execute(0xc000091680, 0xc0000e21b0, 0x3, 0x3, 0xc000091680, 0xc0000e21b0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/spf13/cobra/command.go:762 +0x465
     github.com/k14s/ytt/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000091180, 0xc000091180, 0xdf47e0, 0xc000055f58)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/spf13/cobra/command.go:852 +0x2ec
     github.com/k14s/ytt/vendor/github.com/spf13/cobra.(*Command).Execute(...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/spf13/cobra/command.go:800
     main.main()
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/cmd/ytt/ytt.go:17 +0x10b
     )

per discussion in slack (https://kubernetes.slack.com/archives/CH8KCCKA5/p1562863952459400) @hfjn needed a way to encode yaml fragments (chunks) as yaml so that applications can read that configuration from a ConfigMap.

it would look something like this: https://gist.github.com/cppforlife/395dfa60c4e753cd04d2a26eb1d88bf2

currently yaml.encode fails as it only supports encoding simple values (non yaml fragment).

- (p) Unexpected marshaling of map (backtrace: goroutine 1 [running]:
    demo.yml:9 in <toplevel>
     L   config1.yml: #@ yaml.encode(config1())

as a side note, should allow yaml.decode to return yaml fragments if there are multiple docs (alternatively add yaml.decode_stream so that return types are not different based on input types)?

Problem: it's quite cumbersome to prefix each template command by #@. Also the YAML inside the function body cannot (?) indented, which is inconsistent with programming best practices, at odd with YAML itself and starlark conventions.

Proposal: allow function definitions to be defined without the #@ prefix, as part of the template prologue, before the YAML. The starlark syntax should be extended to allow YAML values, eg:

def some_yaml():
  name: max
  cities:
  - SF
  - LA
end

I don't have the whole proposal fleshed out yet. I just putting this here to see if there is some interest in this strawman proposal. If yes I can expand and give more details.

This following example is failing with an error:

#@ def some_data():
- "hello world"
#@ end
---
simple_key: #@ some_data()[0]

with the following error:

- unhandled index operation yamlfragment[int]
    config.yml:5 in <toplevel>
     L simple_key: #@ some_data()[0]

but it works with starlark literals:

#@ def some_data():
#@   return ["hello world"]
#@ end
---
simple_key: #@ some_data()[0]

Same for map indexing, this does not work:

#@ def some_data():
hello: world
#@ end
---
simple_key: #@ some_data()["hello"]

with the following error:

- unhandled index operation yamlfragment[string]
    config.yml:5 in <toplevel>
     L simple_key: #@ some_data()["hello"]

but this works:

#@ def some_data():
#@   return { "hello": "world" }
#@ end
---
simple_key: #@ some_data()["hello"]

I happened to find this tool and tried it, but stumbled upon some unexpected behavior.
Hope the following improvements would help new users.

• Document that at least one YAML document with @data/values annotation should be included in the input when --data-value option is given.
• Document that --output-directory option will empty the target directory.

Is there a reason standard YAML comments are not supported? Since ytt seems to try and sort of follow YAML syntax, I expected it to be ignored by ytt.

$ cat > example.yml <<EOF
earlier: former
# later: latter # TODO someday
EOF

$ ytt template -f example.yml
Error: Unknown comment syntax at line example.yml:2: ' later: latter # TODO someday': Unknown metadata format (use '#@' or '#!')

I could imagine #something without a following space to be something ytt doesn't support due to theoretical ambiguity, but #spacesomething seems reasonable for ytt to ignore as standard YAML convention and non-ytt directive-style.

I understand #! is an alternative, but that's just more of a migration/difference and don't think I understand why it should be needed.

(meant to submit as @dpb587 from personal side-project learnings)

(Apologies, this will be a vague github issue-- about to leave for vacation)

Any thoughts on if/how ytt should/could be leveraged as a templating engine for helm charts as an "official" pluggable backend to be used for a helm chart template? Noticed there weren't any github issues logged about it. And I'm under the impression the helm engine: used by a chart swappable, though is gotpl by default (and mostly only gotpl appears used?)

Could be completely wrong about all of this, and I have done zero research into how helm templating backends work, let alone if they're swappable at all. Just figured I'd put the question out there. Feel free to close anytime.

Apologies if this is something already covered with the documentation, but:

Is there a way to pass in multiple values files into a template? I'm trying to separate out configuration for values into multiple files rather than one big-honkin' single file, though I'm unsure if this is supported or how to properly reference multiple data files. With that said, having a single values.yml isn't horrible and may be preferable. But what I'm trying to do is hierarchical overriding, e.g.:

.
├── envs
│   ├── aws
│   │   ├── dev
│   │   │   └── values.yml
│   │   ├── sandbox
│   │   │   └── values.yml
│   │   └── values.yml
│   ├── values.yml
│   └── vsphere
│       ├── dev
│       │   └── values.yml
│       ├── sandbox
│       │   └── values.yml
│       └── values.yml
└── template.yml

... and extract out "common" values into the appropriate hierarchical level, e.g. global values defined under envs/, "vsphere"-related values in envs/vsphere, etc. and the values are overridden as you get closer to the specific values file you're trying to use.

Advice and/or feedback is appreciated, thank you for your time.

Hi!

It would be nice to be able to pass expressions (that use variables) to the load() statement. Currently only string literals are allowed. Is that a limitation of starlark or was it a design choice of ytt?

My case
I have a project where I'll be using functions for creating parts of the configuration and those functions have a different implementation depending on the "configuration variant" I'm using. In order to import the correct version of the function I wanted to use something load(variant + '.star', 'my_function'), where variant is a variable storing the name of my configuration variant.

this would be an improvement to existing error messages to indicate why ytt isnt allowing to load a file that was not explicitly included via -f/-R flags. outcome of #7

using a data field that is among the list of reserved keywrods, makes ytt fail with the following error:

- keyword pass is not supported
    demo.yml:32
     L   annotation: #@ data.values.pass

The error message should be improved and make it clear why the reserved keywords cannot be used.

When there is a symlink in a library evaluated by ytt, ytt errors out with an error similar to:

Expected file 'pkg2/dir2' to be a regular file, but was not

Symlinks should be handled:

• if the symlink is internal to the directory tree that is initially loaded, the symlink should be followed
• if the symlink is external to the directory tree initially loaded:
  • if the symlink is not needed, it should be silently ignored
  • else, the symlink should trigger an error telling which target directory is missing

Additionally, ytt should allow from command-line to load additional directories. If a symlink points to this additional directory, the symlink should be followed.

Example:

$ ytt -f main.ytt/ -o main.out
Error: Evaluating main.ytt/symlink.yaml points to ../lib.ytt but this file is not loaded.
Use --load /home/user/Projects/lib.ytt to allow access
$ ytt -f main.ytt/ -o main.out --load lib.ytt
$

If we can agree on how this should be implemented, I can implement it

Implementation notes:

• At load time, symlinks should be loaded as symlinks and their target should be stored in memory in the File object as well as the input directory that lead to the discovery of that symlink.
• in library.go NewRootLibrary() should add symlinks to the Library objects. Symlinks should be considered for both file access and child access (it will depend on the file type of the symlink target).
• Library objects should be created for extra loaded directories
• we need to be able to resolve ../ paths that can happen in symlinks, so the Library object must reference its parent.
• For each input directory (-f flag, main.ytt in the example), a relative path should be computed to go from the input directory to the extra loaded directories (--load, lib.ytt):

  // Resolve symlinks
  uniqueLoadedDir, err := filepath.EvalSymlinks(loadedDir)
  uniqueInputDir, err := filepath.EvalSymlinks(inputDir)
  // Make the path absolute so there is no error in filepath.Rel()
  absLoadedDir, err := filepath.Abs(uniqueLoadedDir)
  absInputDir, err := filepath.Abs(uniqueInputDir)
  // Compute relative path
  InputToLoaded, err := filepath.Rel(absInputDir, absLoadedDir)

• Parent Library objects should be created as needed from these relative paths from the root Library.
• When following symlinks, the Library object should be used to navigate (and not the filesystem). When at the root library, there might be different parent depending on which input directory was used to discover the symlink. The proper parent Library should be used.

chatting with @cdutra, it might be interesting to generate json to template some of the more advanced terraform configuration

This work:

#@ def some_data():
hello: world
#@ end

simple_key: #@ some_data()

This does't work

#@ def some_data():
"hello"
#@ end

simple_key: #@ some_data()

neither this:

#@ def some_data():
- hello: world
#@ end

simple_key: #@ some_data()

The error is not really understandable either: Unmarshaling YAML template: yaml: line 2: did not find expected

ytt errors when parsing a string with the character sequence (@. I noticed this when trying to load a file that contained a JSON path expression.

$ echo 'JSONPath: .status.conditions[?(@.type=="Ready")].status' | ytt -f -
Error: Compiling YAML template 'stdin.yml': Missing code closing '@)' at line 1 col 46

Hi,

I get an error when running ytt against example-load-custom-library when I specify the files one by one:

$ ytt -f _ytt_lib/github.com/k14s/test-lib/funcs.star -f config.yml
Error: 
- cannot load @github.com/k14s/test-lib:funcs.star: Could not find private library (directory '_ytt_lib' missing?)
    config.yml:11 in <toplevel>
     L #@ load("@github.com/k14s/test-lib:funcs.star", "testfunc")

currently some top level errors do not include file location:

pivotals-iMac-2:serving pivotal$ ytt -f config/ --file-mark '**/*:type=yaml-plain'
Error: Unmarshaling YAML template: yaml: line 6: did not find expected <document start>

comparison comes up pretty often so it would be nice to document some common comparisons.

Currently, load(0 works by detecting the file type depending on the file name suffix. In case of yaml libraries, especially, it must end up with .lib.yml which is not always desired (you might just want to have the standard yaml suffix).

I suggest this limitation could be lifted by explicitly defining the type we want to load at load time. This could be achieved by separate functions such as loadYaml() or loadStarlark().

Another alternative would be to add a parameter to load() but this would be a backwards incompatible change.

example-load-custom-library does not seem to be working. I renamed the directory as instructed but I still get this error:

Error: 
- cannot load @github.com/k14s/test-lib:funcs.star: Could not find private library (directory '_ytt_lib' missing?)
    config.yml:11 in <toplevel>
     L #@ load("@github.com/k14s/test-lib:funcs.star", "testfunc")

tree . :

.
├── _ytt_lib
│   └── github.com
│       └── k14s
│           └── test-lib
│               └── funcs.star
└── config.yml

Is it possible to also load a file in the same directory, eg:

#@ load("funcs.lib.yml", "source")

tree .:

.
├── config.yml
└── funcs.lib.yml

ytt version: 0.3.0

An "app" is actually a micro-service in my Kubernetes application
My motivation to use ytt is to be able to build all the YAML files on the fly from templates, data and code.
I would like to run ytt only once and get all the necessary YAML files for a deployment.
At the moment, I'm almost there
But I need the separate files
So I could have more control over the deployment process
WDYT?
It would be awesome to have something like this:

#@ load("@ytt:data", "data")
#@ load("@ytt:template", "template")
#@ load("@functions:config.star", "get_app_list")

#@ load("k8s/service.lib.yaml", "k8s_service")
#@ load("k8s/secret.lib.yaml", "k8s_secret")
#@ load("k8s/config-map.lib.yaml", "k8s_config_map")
#@ load("k8s/deployment.lib.yaml", "k8s_deployment")

#@ for app in get_app_list():
#@ output-to-file ("{}.yaml".format(app.name)):
#@ print ("Creating manifests for '{}'".format(app.name))
#@ print ("-- Creating K8s Service")
--- #@ template.replace(k8s_service(app))
#@ print ("-- Creating K8s Secret")
--- #@ template.replace(k8s_secret(app))
#@ print ("-- Creating K8s ConfigMap")
--- #@ template.replace(k8s_config_map(app))
#@ print ("-- Creating K8s Deployment")
--- #@ template.replace(k8s_deployment(app))
#@ end
#@ end

output-to-file ([FILE-NAME]) ... end - just an idea (edited)

via slack

As of 0.17.0, text templating requires the annotation for all parent nodes.

An example where there is tedious is the construction of concourse pipelines: we use loops to design the jobs, and each job uses text templating (i.e. fill in a release name to build). Upgrading from prior to 0.17.0 to latest as of this writing requires the annotation in many places for us.

Would it be possible to have a CLI option or doc-wide annotation to enable? Is there a perceived/real performance cost to this?

Hi,

We (CF Release Integration) use the hasattr helper function as part of our tooling to generate a Concourse pipeline from a template and an input file. One of the things we do is add debug jobs if the "debug" attribute is set for a release in the input file for the pipeline. For example, given this template snippet (greatly simplified):

#@ load("@ytt:data", "data")
---
groups:
- name: #@ r.name
  jobs:
  #@ for r in data.values.releases:
  - #@ "update-" + r.name
  #@ if hasattr(r, "debug"):
  - #@ "destroy-" + r.name + "-debug-env"
  #@ end
  #@ end

and this input file:

#@data/values
---
releases:
- name: binary-buildpack
  repository: cloudfoundry/binary-buildpack-release
  debug: true

running ytt -f . generates the following pipeline yml:

groups:
- name: binary-buildpack
  jobs:
  - update-binary-buildpack
  - destroy-binary-buildpack-debug-env

During normal operation, we do not need to run in "debug" mode, so we simply remove the debug attribute from the release in the input file and those jobs are not rendered in the pipeline manifest:

groups:
- name: binary-buildpack
  jobs:
  - update-binary-buildpack

However, since v0.19.0 of ytt was released, we started receiving the following error instead of the expected generated pipeline yml:

Error:
- interface conversion: interface is nil, not starlark.Value

If I add the debug: x attribute to the input file, then everything works as expected with the latest version of ytt (v0.20.0 as of the time of writing), but we were hoping to not have to do this.

Please let me know if you have any questions about this issue.

Thanks,
Dave

Seems like there's a problem with {{}}.
We're experimenting with using ytt for templating our concourse pipelines.
YTT blows up with the following error:

Error:
- runtime error: hash of unhashable type yaml.MapSlice (backtrace: goroutine 1 [running]:
    stacks.yml:9 in <toplevel>
     L       uri: {{buildpacks-ci-git-uri-public}}

Error:
- runtime error: hash of unhashable type yaml.MapSlice (backtrace: goroutine 1 [running]:
    stacks.yml:9 in <toplevel>
     L       uri: {{buildpacks-ci-git-uri-public}}

    reason:
     runtime/debug.Stack(0x14ec820, 0x14e69c0, 0xc0000b8090)
     	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1.1(0xc0004e6ba0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/core/errs.go:18 +0x9d
     panic(0x14e69c0, 0xc0000b8090)
     	/usr/local/go/src/runtime/panic.go:522 +0x1b5
     github.com/k14s/ytt/pkg/yamltemplate.MapItemOverride.Apply(0xc0004085f0, 0xc000408690, 0xc0004e6908, 0xb)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/yamltemplate/map_key_overrides.go:26 +0x138
     github.com/k14s/ytt/pkg/yamltemplate.EvaluationCtx.PrepareNode(0x16ad880, 0xc0004085f0, 0x16ad8e0, 0xc000408690, 0x12, 0xc000096300)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/yamltemplate/evaluation_ctx.go:24 +0x71
     github.com/k14s/ytt/pkg/template.(*EvaluationCtx).startNode(0xc000128000, 0x13, 0x13, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/evaluation_ctx.go:191 +0x1bc
     github.com/k14s/ytt/pkg/template.(*EvaluationCtx).TplStartNode(0xc000128000, 0xc0000dcff0, 0xc000143a40, 0xc000155ff0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x8, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/evaluation_ctx.go:157 +0x179
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).tplStartNode(...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/compiled_template.go:241
     github.com/k14s/ytt/pkg/template/core.ErrWrapper.func1(0xc0000dcff0, 0xc000143a40, 0xc000155ff0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/core/errs.go:25 +0xe9
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Builtin).CallInternal(0xc000143a40, 0xc0000dcff0, 0xc000155ff0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x16ab320, 0x10, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/value.go:653 +0x8b
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.Call(0xc0000dcff0, 0x16aaa20, 0xc000143a40, 0xc000155ff0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x16ab320, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:1028 +0x19f
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Function).CallInternal(0xc0004d3200, 0xc0000dcff0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13e88b7, 0x152ac40, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/interp.go:305 +0x4550
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.Call(0xc0000dcff0, 0x16aaaa0, 0xc0004d3200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:1028 +0x19f
     github.com/k14s/ytt/vendor/go.starlark.net/starlark.(*Program).Init(0xc0004ba018, 0xc0000dcff0, 0xc000143950, 0x0, 0x0, 0xc0004d2e80)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/go.starlark.net/starlark/eval.go:358 +0x96
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).eval(0xc0004b2060, 0xc0000dcff0, 0xc000143950, 0x0, 0x0, 0x0, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/compiled_template.go:140 +0x203
     github.com/k14s/ytt/pkg/template.(*CompiledTemplate).Eval(0xc0004b2060, 0xc0000dcff0, 0x16a9360, 0xc0001fca20, 0xc0000dcff0, 0xc0001438f0, 0xc0001299d0, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/template/compiled_template.go:101 +0x719
     github.com/k14s/ytt/pkg/workspace.(*TemplateLoader).EvalYAML(0xc0001fca20, 0xc0000dc3c0, 0xc0000dc370, 0x0, 0xc00015ae60, 0x1, 0x1)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/workspace/template_loader.go:172 +0x8be
     github.com/k14s/ytt/pkg/workspace.(*LibraryLoader).eval(0xc0004e7b68, 0x14d6ee0, 0xc0001fc9f0, 0x0, 0x0, 0xc00020f090, 0x0, 0x0, 0x100dca8)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/workspace/library_loader.go:126 +0x6c4
     github.com/k14s/ytt/pkg/workspace.(*LibraryLoader).Eval(0xc0004e7b68, 0x14d6ee0, 0xc0001fc9f0, 0x14d6ee0, 0xc0001fc9f0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/workspace/library_loader.go:82 +0x5d
     github.com/k14s/ytt/pkg/cmd/template.(*TemplateOptions).RunWithFiles(0xc000150000, 0xc0000b2048, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/template/cmd.go:118 +0x20d
     github.com/k14s/ytt/pkg/cmd/template.(*TemplateOptions).Run(0xc000150000, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/template/cmd.go:83 +0x255
     github.com/k14s/ytt/pkg/cmd/template.NewCmd.func1(0xc000152000, 0xc00014a2c0, 0x0, 0x2, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/pkg/cmd/template/cmd.go:53 +0x2a
     github.com/k14s/ytt/vendor/github.com/cppforlife/cobrautil.WrapRunEForCmd.func1.1(0xc000152000, 0xc00014a2c0, 0x0, 0x2, 0x0, 0x0)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/cppforlife/cobrautil/misc.go:25 +0xaf
     github.com/k14s/ytt/vendor/github.com/spf13/cobra.(*Command).execute(0xc000152000, 0xc0000b6010, 0x2, 0x2, 0xc000152000, 0xc0000b6010)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/spf13/cobra/command.go:762 +0x465
     github.com/k14s/ytt/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000152000, 0xc000152000, 0x1a227e0, 0xc0000f3f58)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/spf13/cobra/command.go:852 +0x2ec
     github.com/k14s/ytt/vendor/github.com/spf13/cobra.(*Command).Execute(...)
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/vendor/github.com/spf13/cobra/command.go:800
     main.main()
     	/Users/argonaut/workspace/k14s-go/src/github.com/k14s/ytt/cmd/ytt/ytt.go:17 +0x10b

And excertpy of the YAML in question:

#@ load("@ytt:data", "data")

---
resources:
  #! Github Repos
  - name: buildpacks-ci
    type: git
    source:
      uri: {{buildpacks-ci-git-uri-public}}
      branch: master

now that yaml.v3 is out and supports preserving comments on yaml nodes, it would be nice to have ytt fmt -R -f . command that would format yaml files in place.

alternatively, as @jbeda suggested, provide YAML linting facilities.

How to get the index in the for loop?

For example:

#@ for product in data.values.products:
- #@ "product-{}-{}".format(#@i, product)
#@ end

I'd like to be able to use ytt to apply overlays to semi-trusted yaml files. These files should not be compiled by ytt, but be matched/transformed by other templates. If the file contains a ytt directive, it should be ignored.

In order to make ytt accessible via homebrew package management, it would be nice if the ytt binary were made available in a brew repository.

/cc @drnic I don't mean to be presumptive of starkandwayne's ci/cd process for binaries like this, but maybe this is something you all might consider adding into one of the homebrew-* repository lineups.

It's common to extract some keys and/or values from existing yaml files.

For example, given a product config file:

#! products.yml
products:
  product-1: xxx
  product-2: xxx

It'd be cool to extract all products out as an array, here might be ["product-1", "product-2"], so that I can do something like:

#@ load("@ytt:extract", "extract")

product-info:

#@ for product in (extract("products.yml", "products")):
- name: #@ product
#@ end