two files in one dict? about pyyaml-include

[tanbro]

Thanks for the question.

I thinks, such a YAML:

foo:
  !include foo-1.yml
  !include foo-2.yml

is illegal, casue it's neither a mapping nor a sequence.

If want it to be a sequence, we shall write the YAML as:

foo:
  - !include foo-1.yml
  - !include foo-2.yml

it wall be loaded to Python dictionary object:

{
  "foo":
    [
        {"one": "a"},
        {"two": "b"}
    ]
}

[grauschnabel]

But there "foo" is not a sequence, its a list of sequences. That is exactly my problem.
The result I want is

{
   "foo":
      {
         "one": "a",
         "two": "b"
      }
}

[tanbro]

@grauschnabel Do you mean that:

we have two YAML files to be included:

• file foo-1.yml:

  one: a

• file foo-2.yml:

  two: b

and when we write :

foo:
  !include foo-1.yml
  !include foo-2.yml

the excepted data is:

{
   "foo":
      {
         "one": "a",
         "two": "b"
      }
}

Is that?

But the YAML including constructor can not union two mappings into one.

Because the constructor will first parse a including file into a YAML node, then PyYAML puts the node into the doc tree.

So, the including parts of above example are two separated mappings, and can not be joined into one mapping as what they are in the included files literally

i think they will be parsed to something like:

{
   "foo":
      {"one": "a"}
      {"two": "b"}
}

That's illegal.

What's more, PyYAML takes the fragment

!include foo-1.yml
!include foo-2.yml

like:

include("foo-1.yml !include foo-2.yml")

i did not expect that.

[grauschnabel]

I think that

foo:
   bar: 1
   baz: 2

is absolutly legal in yaml. But now I understand how you have built this tool and the I think that it's a bit like a design issue. Maybe a colon could solve the problem, but that would possibly break compatibility to like its now. I thought about

foo:
   !include: bar.yml
   !include: baz.yml

Would do the trick to make pyyaml parse it. Otherwise

include("foo-1.yml !include foo-2.yml")

Could be reparsed. by str.split("!include").

In both cases: We could load bar and baz and then combine it via dict.update() so that it is clear that later loaded keys overwrite earlyer definitions of the same key.

What do you think about that?

[tanbro]

Thanks for @grauschnabel

foo:
   bar: 1
   baz: 2

is a valid YAML.

i'll try to avoid pyyaml parsing

foo:
   !include: bar.yml
   !include: baz.yml

to include("foo-1.yml !include foo-2.yml")

...

[smurfix]

You should be able to do it with merging, if you can tolerate some cruft lying around.

one: &one
    !include bar.yml
two: &two
    !include baz.yml
foo:
    <<: [ *one, *two ]

Disclaimer: untested.

[tanbro]

Thanks @smurfix

But i think that won't work.
Because what returns by the !include constructor is a python object, not a YAML mapping, and can't inherit

[tanbro]

I think it's not easy to do such a thing in PyYAML's constructor.
Because we can't make it return a document literally.

Perhaps a template engine (eg: Jinja2) is better.

For this issue, we can write a main YAML file with Jinja2 templates in it, and include other files using the template engine.

Here we use Jinja2 together with jinjyaml:

• main.yml:

  foo: !j2 |
      {% include "child-1.yml" %}
      {% include "child-2.yml" %}

• child-1.yml:

  '1.1': one
  '1.2': two

• child-2.yml:

  '2.1':
    2.1.1: three
    2.1.2: four

then parse and render it:

from pprint import pprint
import jinja2
import yaml
from jinjyaml import JinjyamlConstructor, jinjyaml_render

TAG = 'j2'
j2_env = jinja2.Environment(
    loader=jinja2.FileSystemLoader('path/of/the/yaml/files')
)
constructor = JinjyamlConstructor(env=j2_env)
yaml.add_constructor('!{}'.format(TAG), constructor)

with open('main.yml') as fp:
    doc = yaml.load(fp, yaml.Loader)
print(doc)

data = jinjyaml_render(doc, yaml.Loader)
pprint(data)

we'll get

{'foo': <jinjyaml.tagobject.JinjyamlObject object at 0x0000027AB80F0250>}
{'foo': {'1.1': 'one',
         '1.2': 'two',
         '2.1': {'2.1.1': 'three', '2.1.2': 'four'}}}