tanbro / pyyaml-include

yaml include other yaml

pyyaml-include

An extending constructor of PyYAML: include other YAML files into current YAML document.

In version 2.0, fsspec was introduced. With it, we can even include files by HTTP, SFTP, S3 ...

⚠️ Warning
“pyyaml-include” 2.0 is NOT compatible with 1.0

Install

pip install "pyyaml-include"

Since we are using fsspec to open including files from v2.0, an installation can be performed like below, if want to open remote files:

• for files on website:

  pip install "pyyaml-include" fsspec[http]

• for files on S3:

  pip install "pyyaml-include" fsspec[s3]

• see fsspec's documentation for more

🔖 Tip
“pyyaml-include” depends on fsspec, it will be installed no matter including local or remote files.

Basic usages

Consider we have such YAML files:

├── 0.yml
└── include.d
    ├── 1.yml
    └── 2.yml

• 1.yml 's content:

  name: "1"

• 2.yml 's content:

  name: "2"

To include 1.yml, 2.yml in 0.yml, we shall:

1. Register a yaml_include.Constructor to PyYAML's loader class, with !inc as it's tag:

   import yaml
   import yaml_include
   
   # add the tag
   yaml.add_constructor("!inc", yaml_include.Constructor(base_dir='/your/conf/dir'))

2. Write !inc tags in 0.yaml:

   file1: !inc include.d/1.yml
   file2: !inc include.d/1.yml

3. Load it

   with open('0.yml') as f:
      data = yaml.full_load(f)
   print(data)

   we'll get:

   {'file1':{'name':'1'},'file2':{'name':'2'}}

4. (optional) the constructor can be unregistered:

   del yaml.Loader.yaml_constructors["!inc"]
   del yaml.UnSafeLoader.yaml_constructors["!inc"]
   del yaml.FullLoader.yaml_constructors["!inc"]

Include in Mapping

If 0.yml was:

file1: !inc include.d/1.yml
file2: !inc include.d/2.yml

We'll get:

file1:
  name: "1"
file2:
  name: "2"

Include in Sequence

If 0.yml was:

files:
  - !inc include.d/1.yml
  - !inc include.d/2.yml

We'll get:

files:
  - name: "1"
  - name: "2"

Advanced usages

Wildcards

File name can contain shell-style wildcards. Data loaded from the file(s) found by wildcards will be set in a sequence.

That is, a list will be returned when including file name contains wildcards. Length of the returned list equals number of matched files:

If 0.yml was:

files: !inc include.d/*.yml

We'll get:

files:
  - name: "1"
  - name: "2"

• when only 1 file matched, length of list will be 1
• when there are no files matched, an empty list will be returned

We support **, ? and [..]. We do not support ^ for pattern negation. The maxdepth option is applied on the first ** found in the path.

❗ Important

• Using the ** pattern in large directory trees or remote file system (S3, HTTP ...) may consume an inordinate amount of time.
• There is no method like lazy-load or iteration, all data of found files returned to the YAML doc-tree are fully loaded in memory, large amount of memory may be needed if there were many or big files.

Work with fsspec

In v2.0, we use fsspec to open including files, thus we can include files from many different sources, such as local file system, S3, HTTP, SFTP ...

For example, we can include a file from website in YAML:

conf:
  logging: !inc http://domain/etc/app/conf.d/logging.yml

In such situations, when creating a Constructor constructor, a fsspec filesystem object shall be set to fs argument.

For example, if want to include files from website, we shall:

1. create a Constructor with a fsspec HTTP filesystem object as it's fs:

   import yaml
   import fsspec
   import yaml_include
   
   http_fs = fsspec.filesystem("http", client_kwargs={"base_url": f"http://{HOST}:{PORT}"})
   
   ctor = yaml_include.Constructor(fs=http_fs, base_dir="/foo/baz")
   yaml.add_constructor("!inc", ctor, yaml.Loader)

2. then, write a YAML document to include files from http://${HOST}:${PORT}:

   key1: !inc doc1.yml    # relative path to "base_dir"
   key2: !inc ./doc2.yml  # relative path to "base_dir" also
   key3: !inc /doc3.yml   # absolute path, "base_dir" does not affect
   key3: !inc ../doc4.yml # relative path one level upper to "base_dir"

3. load it with PyYAML:

   yaml.load(yaml_string, yaml.Loader)

Above YAML snippet will be loaded like:

• key1: pared YAML of http://${HOST}:${PORT}/foo/baz/doc1.yml
• key2: pared YAML of http://${HOST}:${PORT}/foo/baz/doc2.yml
• key3: pared YAML of http://${HOST}:${PORT}/doc3.yml
• key4: pared YAML of http://${HOST}:${PORT}/foo/doc4.yml

🔖 Tip
Check fsspec's documentation for more

---

ℹ️ Note
If fs argument is omitted, a "file"/"local" fsspec filesystem object will be used automatically. That is to say:

data: !inc: foo/baz.yaml

is equivalent to (if no base_dir was set in Constructor()):

data: !inc: file://foo/baz.yaml

and

yaml.add_constructor("!inc", Constructor())

is equivalent to:

yaml.add_constructor("!inc", Constructor(fs=fsspec.filesystem("file")))

Parameters in YAML

As a callable object, Constructor passes YAML tag parameters to fsspec for more detailed operations.

The first argument is urlpath, it's fixed and must-required, either positional or named. Normally, we put it as a string after the tag(eg: !inc), just like examples above.

However, there are more parameters.

• in a sequence way, parameters will be passed to python as positional arguments, like *args in python function. eg:

  files: !inc [include.d/**/*.yaml, {maxdepth: 1}, {encoding: utf16}]

• in a mapping way, parameters will be passed to python as named arguments, like **kwargs in python function. eg:

  files: !inc {urlpath: /foo/baz.yaml, encoding: utf16}

But the format of parameters has multiple cases, and differs variably in different fsspec implementation backends.

• If a scheme/protocol(“http://”, “sftp://”, “file://”, etc.) is defined, and there is no wildcard in urlpath, Constructor will invoke fsspec.open directly to open it. Which means Constructor's fs will be ignored, and a new standalone fs will be created implicitly.

  In this situation, urlpath will be passed to fsspec.open's first argument, and all other parameters will also be passed to the function.

  For example,

  • the YAML snippet

    files: !inc [file:///foo/baz.yaml, r]

    will cause python code like

    with fsspec.open("file:///foo/baz.yaml", "r") as f:
        yaml.load(f, Loader)

  • and the YAML snippet

    files: !inc {urlpath: file:///foo/baz.yaml, encoding: utf16}

    will cause python code like

    with fsspec.open("file:///foo/baz.yaml", encoding="utf16") as f:
        yaml.load(f, Loader)

• If urlpath has wildcard, and also scheme in it, Constructor will:

  Invoke fsspec's open_files function to search, open and load files, and return the results in a list. YAML include statement's parameters are passed to open_files function.

• If urlpath has wildcard, and no scheme in it, Constructor will:

  1. invoke corresponding fsspec implementation backend's glob method to search files,
  2. then call open method to open each found file(s).

  urlpath will be passed as the first argument to both glob and open method of the corresponding fsspec implementation backend, and other parameters will also be passed to glob and open method as their following arguments.

  In the case of wildcards, what need to pay special attention to is that there are two separated parameters after urlpath, the first is for glob method, and the second is for open method. Each of them could be either sequence, mapping or scalar, corresponds single, positional and named argument(s) in python. For example:

  • If we want to include every .yml file in directory etc/app recursively with max depth at 2, and open them in utf-16 codec, we shall write the YAML as below:

    files: !inc ["etc/app/**/*.yml", {maxdepth: !!int "2"}, {encoding: utf16}]

    it will cause python code like:

    for file in local_fs.glob("etc/app/**/*.yml", maxdepth=2):
        with local_fs.open(file, encoding="utf16") as f:
            yaml.load(f, Loader)

  • Since maxdepth is the seconde argument after path in glob method, we can also write the YAML like this:

    files: !inc ["etc/app/**/*.yml", [!!int "2"]]

    The parameters for open is omitted, means no more arguments except urlpath is passed.

    it will cause python code like:

    for file in local_fs.glob("etc/app/**/*.yml", 2):
        with local_fs.open(file) as f:
            yaml.load(f, Loader)

  • The two parameters can be in a mapping form, and name of the keys are "glob" and "open". for example:

    files: !inc {urlpath: "etc/app/**/*.yml", glob: [!!int "2"], open: {encoding: utf16}}

  ❗ Important
  PyYAML sometimes takes scalar parameter of custom constructor as string, we can use a ‘Standard YAML tag’ to ensure non-string data type in the situation.

  For example, following YAML snippet may cause an error:

  files: !inc ["etc/app/**/*.yml", open: {intParam: 1}]

  Because PyYAML treats {"intParam": 1} as {"intParam": "1"}, which makes python code like fs.open(path, intParam="1"). To prevent this, we shall write the YAML like:

  files: !inc ["etc/app/**/*.yml", open: {intParam: !!int 1}]

  where !!int is a ‘Standard YAML tag’ to force integer type of maxdepth argument.

  ℹ️ Note
  BaseLoader, SafeLoader, CBaseLoader, CSafeLoader do NOT support ‘Standard YAML tag’.

  ---

  🔖 Tip
  maxdepth argument of fsspec glob method is already force converted by Constructor, no need to write a !!int tag on it.

• Else, Constructor will invoke corresponding fsspec implementation backend's open method to open the file, parameters beside urlpath will be passed to the method.

Absolute and Relative URL/Path

When the path after include tag (eg: !inc) is not a full protocol/scheme URL and not starts with "/", Constructor tries to join the path with base_dir, which is a argument of Constructor.__init__(). If base_dir is omitted or None, the actually including file path is the path in defined in YAML without a change, and different fsspec filesystem will treat them differently. In local filesystem, it will be cwd.

For remote filesystem, HTTP for example, the base_dir can not be None and usually be set to "/".

Relative path does not support full protocol/scheme URL format, base_dir does not effect for that.

For example, if we register such a Constructor to PyYAML:

import yaml
import fsspec
import yaml_include

yaml.add_constructor(
    "!http-include",
    yaml_include.Constructor(
        fsspec.filesystem("http", client_kwargs={"base_url": f"http://{HOST}:{PORT}"}),
        base_dir="/sub_1/sub_1_1"
    )
)

then, load following YAML:

xyz: !http-include xyz.yml

the actual URL to access is http://$HOST:$PORT/sub_1/sub_1_1/xyz.yml

Serialization

When load YAML string with include statement, the including files are default parsed into python objects. Thant is, if we call yaml.dump() on the object, what dumped is the parsed python object, and can not serialize the include statement itself.

To serialize the statement, we shall first create an yaml_include.Constructor object whose autoload is False:

import yaml
import yaml_include

ctor = yaml_include.Constructor(autoload=False)

then add both Constructor for Loader and Representer for Dumper:

yaml.add_constructor("!inc", ctor)

rpr = yaml_include.Representer("inc")
yaml.add_representer(yaml_include.Data, rpr)

Now, the including files will not be loaded when call yaml.load(), and yaml_include.Data objects will be placed at the positions where include statements are.

continue above code:

yaml_str = """
- !inc include.d/1.yaml
- !inc include.d/2.yaml
"""

d0 = yaml.load(yaml_str, yaml.Loader)
# Here, "include.d/1.yaml" and "include.d/2.yaml" not be opened or loaded.
# d0 is like:
# [Data(urlpath="include.d/1.yaml"), Data(urlpath="include.d/2.yaml")]

# serialize d0
s = yaml.dump(d0)
print(s)
# ‘s’ will be:
# - !inc 'include.d/1.yaml'
# - !inc 'include.d/2.yaml'

# de-serialization
ctor.autoload = True # re-open auto load
# then load, the file "include.d/1.yaml" and "include.d/2.yaml" will be opened and loaded.
d1 = yaml.load(s, yaml.Loader)

# Or perform a recursive opening / parsing on the object:
d2 = yaml_include.load(d0) # d2 is equal to d1

autoload can be used in a with statement:

ctor = yaml_include.Constructor()
# autoload is True here

with ctor.managed_autoload(False):
    # temporary set autoload to False
    yaml.full_load(YAML_TEXT)
# autoload restore True automatic

Include JSON or TOML

We can include files in different format other than YAML, like JSON or TOML -- custom_loader is for that.

📑 Example
For example:

import json
import tomllib as toml
import yaml
import yaml_include

# Define loader function
def my_loader(urlpath, file, Loader):
    if urlpath.endswith(".json"):
        return json.load(file)
    if urlpath.endswith(".toml"):
        return toml.load(file)
    return yaml.load(file, Loader)

# Create the include constructor, with the custom loader
ctor = yaml_include.Constructor(custom_loader=my_loader)

# Add the constructor to YAML Loader
yaml.add_constructor("!inc", ctor, yaml.Loader)

# Then, json files will can be loaded by std-lib's json module, and the same to toml files.
s = """
json: !inc "*.json"
toml: !inc "*.toml"
yaml: !inc "*.yaml"
"""

yaml.load(s, yaml.Loader)

Develop

1. clone the repo:

   git clone https://github.com/tanbro/pyyaml-include.git
   cd pyyaml-include

2. create then activate a python virtual-env:

   python -m venv .venv
   .venv/bin/activate

3. install development requirements and the project itself in editable mode:

   pip install -r requirements.txt

Now you can work on it.

Test

read: tests/README.md

pyyaml-include's Issues

Yaml anchor support for include yaml files

Hi tanbro, I thought it would be nice if this library also supported anchors when including a yaml file so I wrote this PR #32.

Jinja2 support

Hello,

How likely are you to support jinja templates?

I have YAML templates that have jina logic in them. For example:

 containers:
    - name: "{{application_name}}"
      image: "{{business_unit}}-docker-{{environment}}.artifactory.svc.aws.gartner.com/{{application_name}}:{{version}}"
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 8080
          protocol: TCP
      resources:
        limits:
          memory: {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}

The last line of the code, is a jinja2 expression.

The probably is, I run the template through the pyyaml-include, to get a full single document from multiple templates. Then, I run the single template through the template engine.

If this:

memory:" {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}"

when pyyaml-include operates on this line, you get:

memory: '{{ vars.eks.memory_limits if vars.eks.memory_limits else ''512mb'' }}'

And the jinja2 engine can't understand it. What jinja2 is expecting is this:

memory: {{ vars.eks.memory_limits if vars.eks.memory_limits else '512mb' }}

Are you willing to support a jinja2 template language?

Thanks

Wildcard with Hash/Dict/Map

More of a question, but also a request if it is possible. Can i include wildcard and instead of a list fill a object with filename (with/without suffix) as the key and loaded value as the value?

files: !include {path_name: include.d/*.yml, map: full_name/without_prefix/custom_function}

files: !include {path_name: include.d/*.yml, map: without_prefix}

This should be:

files:
  1:
    name: 1
 2:
    name: 2

files: !include {path_name: include.d/*.yml, map:  true}

This should be:

files:
  1.yaml:
    name: 1
 2.yaml:
    name: 2

def custom_function(file:str, value:any):
  return Path(file).name

How to make the package work with setuptools lower than 61.0.0

I'm building the package from scratch.

For some reason, I work on a system with an old setuptools which does not support PEP621 and is unable to extract name and version metadata in pyproject.toml as pypa/setuptools#3269 (comment).

Is there any way to achieve the compatibility with the old setuptools? Thanks!

Python 3.6 unsupported after releasing `1.2.post1`

Hi,
re.Pattern is Python 3.7+

On your latest release, 1.2.post1, you've included:

...
  File "/usr/local/lib/python3.6/site-packages/yamlinclude/__init__.py", line 7, in <module>
    from .constructor import *
  File "/usr/local/lib/python3.6/site-packages/yamlinclude/constructor.py", line 24, in <module>
    class YamlIncludeConstructor:
  File "/usr/local/lib/python3.6/site-packages/yamlinclude/constructor.py", line 46, in YamlIncludeConstructor
    reader_map: Collection[Tuple[Union[str, re.Pattern], Reader]] = None
AttributeError: module 're' has no attribute 'Pattern'

I guess either by mistake, or you removed support for 3.6 and didn't include that in your release notes anywhere :)

FYI

two files in one dict?

I'm trying to get it mapped with more files, but it doesn't work for me.

I have

file foo-1.yml
one: a
file foo-2.yml
two: b

using

foo: 
  !include foo-1.yml

Works fine, I get a {'foo' :{one: a}} - thats what I want. But adding another line, like

foo:
  !include foo-1.yml
  !include foo-2.yml

gives an error: FileNotFoundError: [Errno 2] No such file or directory: '/home/martin/Projects/toki/toki/data/yaml/items-area-1.yml !include items-area-2.yml'

And using

foo:
  !include foo-*.yml

makes a list of two dicts..
what can I doo to have just {foo: {'one': a, 'two': b}} as a result here?

is it currently supported to set an anchor for the included mapping?

I was trying to have some inheritance, by including the shared yaml-file:

`
file1: !include include.d/1.yaml & shared

section1:
<<: *shared
foo: bar
`

The library looks to by typed, but the `py.typed` file is missing

See: https://peps.python.org/pep-0561/

Ability for default value

Ability to use default value if file doesn't exist.

Example:

Somelist: !include { pathname: path.yaml, default: [] }

How to concatenate two yaml files?

Hello, I want to concatenate two yaml files, what should I do?

for example, I have two different yaml files,

# file1.yaml
name: 1

# file2.yaml
class: 2

I want to concatenate these two files into one file after using pyyaml-include, as shown below

# merge_file.yaml
name: 1
class: 2

I would like to ask, what should I do to get this result?

Thank you very much.

yaml writer?

This is an interesting project that I'd like to integrate into my tools. I'd like to know if there's any support or interest for a writer class that respects includes.

As an example I'll refer to the first example in the README. Once the data has been read and potentially changed I may want to change the value of file2 name to be "3".

When I use yaml to dump the new dictionary to disk it will ignore the file structure and overwrite the values into one file.

I'd like to know if there is a way to handle this such that the writer respects the existing structure. If not, is that something this project would be interested in?

Use tag with anchors across a single yaml config file

Hi,

I've started playing around with this interesting library and I have a question. I'm not sure if this can be an issue or maybe I missed the point. So let me explain what I want to build.

My environment:
python=3.10.10
PyYAML=6.0
pyyaml-include=1.3

What I've tried so far:
I want to create some defaults that would be use across the YAML config files without need to copy it multiple times in different blocks with possibly with anchors. So, if I have /test/path/custom-defaults/person.yaml file:

.default_description: &default-description
    hair_color: grey
    eye_color: blue
    age: 123
    gender: N/A

And I'd like to include it in other config files, e.g. in /test/path/defaults/describe-people.yaml:

!include ../custom-defaults/person.yaml

persons:
   person.with.custom.defaults: 
       <<: *default-description
   person.with.overrided.defaults: 
       <<: *default-description
       hair_color: blue

When try this I'm getting an error: raise ParserError(None, None, yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>'

On the other side, I can fix it by:

1. Updating person.yaml file a bit, by removing anchors
2. Use !include tag only in blocks, like person.with.custom.defaults: !include ../custom-defaults/person.yaml

Questions:
Is it even possible to use it as a document tag and if yes, can someone provide an example?

Thank you.

Recursive includes do not work with nested directories

Here is the test case:

        os.makedirs("foo/bar")
        with open("foo.yml","w") as fd:
            fd.write("!include foo/bar.yml\n")
        with open("foo/bar.yml","w") as fd:
            fd.write("!include bar/zot.yml\n")
        with open("foo/bar/zot.yml","w") as fd:
            fd.write("foo: 42\n")
        YamlIncludeConstructor.add_to_loader_class(yaml.FullLoader)
        with open("foo.yml") as fd:
            self.assertEqual({"foo":42}, yaml.load(fd, Loader=yaml.FullLoader))
        os.remove("foo/bar/zot.yml")
        os.remove("foo/bar.yml")
        os.remove("foo.yml")
        os.removedirs("foo/bar")

it would work if I write the full path foo/bar/zot.yml into foo/bar.yml, but this is counterproductive (what if bar.yml is included from another place or used separately).

Would it be possible for include to take the path relative to the directory of the node that does the include? (cf. https://gist.github.com/joshbode/569627ced3076931b02f).
Or maybe it is already possible?
Thanks!

Multiple YAML documents in a single file

Is there support forr !include a single yaml file that has three yaml documents? A new document in YAML starts with:

Example

---
name: doc-a

---
name: doc-b

---
name: doc-c

This would be extremely useful if it is not supported. I read through the docs but did not see that this feature is supported.

If it is supported, could you provide an example?

Does not support top level include with other keys

According to this answer

Till now, !include constructor can NOT be used at the TOP level, unless it's the only one node in whole YAML document.

The question is why?

#7

Could you update the PyPI to new version to add PyYAML 6.0 support?

Thanks for the good plugin to support include YAML file, it's very useful.

But the latest version in https://pypi.org/project/pyyaml-include/, is still the old version, which meets the problem to support PyYAML6.0, could you update it to a new version? Thanks.

It seems that I can't rewrite the inherited argments from base `yaml` file.

When I use a test case like:
base.yaml:

# base.yaml
name: John Doe
age: 30
address:
  city: New York
  state: NY

derived.yaml:

# derived.yaml
!include base.yaml
age: 35
address:
  state: CA

test.py

import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='./')

with open('derived.yaml') as f:
    data = yaml.load(f, Loader=yaml.FullLoader)

print(data)

I expect the output is {'name': 'John Doe', 'age': 35, 'address': {'city': 'New York', 'state': 'CA'}} but there is a error occured. The traceback is as follows:

Traceback (most recent call last)
[d:\SelfDatas\SJTU\Projects\Labs\yaml\read_yaml.ipynb](file:///D:/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb) Cell 1 in 7
      [4](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=3) YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='[./](https://file+.vscode-resource.vscode-cdn.net/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/)')
      [6](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=5) with open('derived.yaml') as f:
----> [7](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=6)     data = yaml.load(f, Loader=yaml.FullLoader)
      [9](vscode-notebook-cell:/d%3A/SelfDatas/SJTU/Projects/Labs/yaml/read_yaml.ipynb#W0sZmlsZQ%3D%3D?line=8) print(data)

File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\__init__.py:81, in load(stream, Loader)
     79 loader = Loader(stream)
     80 try:
---> 81     return loader.get_single_data()
     82 finally:
     83     loader.dispose()

File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\constructor.py:49, in BaseConstructor.get_single_data(self)
     47 def get_single_data(self):
     48     # Ensure that the stream contains a single document and construct it.
---> 49     node = self.get_single_node()
     50     if node is not None:
     51         return self.construct_document(node)

File [d:\Program](file:///D:/Program) Files\Anaconda\miniconda\envs\main\lib\site-packages\yaml\composer.py:36, in Composer.get_single_node(self)
     34 document = None
     35 if not self.check_event(StreamEndEvent):
...
    583 # the parser.
    584 if not self.flow_level:

ScannerError: mapping values are not allowed here
  in "derived.yaml", line 4, column 4

Relative include: relativeto the current file?

I think (IMHO) it would make more sense if relative include searches for the file in the current file's directory rather than current working directory.

Use case: when I pass config files in the command line I might have them in different directories. And if they're independent - they could load their own local extensions.

The solution to not break backward compatibility would be, for example to add a new parameter to the constructor or to initialize base_dir=None instead of default empty string.

EDIT: perhaps this is not possible to get information about current file from the node object passed to the constructor?
But as soon as we can redefine reader - I see that yaml.Reader class can handle stream names for file-like objects.

Include and achors

I've got the following files:

# main.yaml
included: &included
    !include include.yaml

rendering:
  - << : *included

and

# include.yaml
-
    output: ignore.txt
    template: ignore.tpl

The following code doesn't work:

In [1]: import yaml 
   ...: from yamlinclude import YamlIncludeConstructor                                                                                                                       

In [2]: YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir='/tmp')                                                                            
Out[2]: <yamlinclude.constructor.YamlIncludeConstructor at 0x7f4c3adcfbb0>

In [3]: import pathlib                                                                                                                                                       

In [4]: data = yaml.load(pathlib.Path('/tmp/main.yaml').open(), Loader=yaml.FullLoader)                                                                                      
---------------------------------------------------------------------------
ConstructorError                          Traceback (most recent call last)
<ipython-input-4-f98f7c4c3bd9> in <module>
----> 1 data = yaml.load(pathlib.Path('/tmp/main.yaml').open(), Loader=yaml.FullLoader)

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/__init__.py in load(stream, Loader)
    112     loader = Loader(stream)
    113     try:
--> 114         return loader.get_single_data()
    115     finally:
    116         loader.dispose()

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in get_single_data(self)
     49         node = self.get_single_node()
     50         if node is not None:
---> 51             return self.construct_document(node)
     52         return None
     53 

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_document(self, node)
     58             self.state_generators = []
     59             for generator in state_generators:
---> 60                 for dummy in generator:
     61                     pass
     62         self.constructed_objects = {}

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_yaml_map(self, node)
    414         data = {}
    415         yield data
--> 416         value = self.construct_mapping(node)
    417         data.update(value)
    418 

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in construct_mapping(self, node, deep)
    218     def construct_mapping(self, node, deep=False):
    219         if isinstance(node, MappingNode):
--> 220             self.flatten_mapping(node)
    221         return super().construct_mapping(node, deep=deep)
    222 

~/work/OnixS/opcg/master/.venv/lib/python3.9/site-packages/yaml/constructor.py in flatten_mapping(self, node)
    205                         merge.extend(value)
    206                 else:
--> 207                     raise ConstructorError("while constructing a mapping", node.start_mark,
    208                             "expected a mapping or list of mappings for merging, but found %s"
    209                             % value_node.id, value_node.start_mark)

ConstructorError: while constructing a mapping
  in "/tmp/main.yaml", line 5, column 5
expected a mapping or list of mappings for merging, but found scalar
  in "/tmp/main.yaml", line 1, column 11

include specific components from other yamls?

Does this package let me include specific components from other yamls? For cases where I don't want to include the entire content of the yaml file. E.g something like

a.yaml

param:
  - name: A
    type: A1
  - name: B
    type: B1

stuff: 123

b.yaml

other:
   - param: !include a.yaml/param[0]
  
stuff: !include a.yaml/stuff

such that reading in b.yaml would result in

{
 "other": [{"param": {"name": 'A', "type": 'A1'}}]
 "stuff": 123 
}

Publish sdist to PyPI

Could you please publish 1.2.post2 sdist to PyPI just like 1.2 (pyyaml-include-1.2.tar.gz)? Thanks.

Does not support top level include with other keys

Here is my setup:

• PyYAML: 5.1
• pyyaml-include: 1.1.1
• python: 2.7.15

The python script (from the README):

import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader)

with open('0.yml') as f:
     data = yaml.load(f, Loader=yaml.FullLoader)

print(data)

0.yml:

!include 1.yml
foo: bar

1.yml:

name: "1"

It seems that mixing toplevel include and other value is not (yet?) supported.

I got this error:

Traceback (most recent call last):
  File "../test.py", line 32, in <module>
    data = yaml.load(f, Loader=yaml.FullLoader)
  File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 43, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python2.7/site-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python2.7/site-packages/yaml/composer.py", line 58, in compose_document
    self.get_event()
  File "/usr/local/lib/python2.7/site-packages/yaml/parser.py", line 118, in get_event
    self.current_event = self.state()
  File "/usr/local/lib/python2.7/site-packages/yaml/parser.py", line 193, in parse_document_end
    token = self.peek_token()
  File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 129, in peek_token
    self.fetch_more_tokens()
  File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 223, in fetch_more_tokens
    return self.fetch_value()
  File "/usr/local/lib/python2.7/site-packages/yaml/scanner.py", line 579, in fetch_value
    self.get_mark())
yaml.scanner.ScannerError: mapping values are not allowed here
  in "0.yml", line 2, column 4

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Load multiple documents

If I have one document such as (foo.yaml):

!include ./baz.yaml
!include ./baz.yaml

and baz.yaml looks like:

baz1: x
---
baz2: y

I would like this to be interpreted as:

baz1: x
---
baz2: y
---
baz1: x
---
baz2: y

Is there any way to achieve this? Currently, even if I use load_all() my current setup fails with error.

Python 3.10 unsopported?

Hello, I have been using pyyaml-include with Python3.8 and it worked well, but when I install the package on python3.10, the python source files are empty?

I get this error:
" from yamlinclude import YamlIncludeConstructor
ImportError: cannot import name 'YamlIncludeConstructor' from 'yamlinclude' (/usr/local/lib/python3.10/dist-packages/yamlinclude/init.py)"

The files "constructor.py", "readers.py", "version.py" inside "/usr/local/lib/python3.10/dist-packages/yamlinclude" are empty

Can we use alias with include?

For example:

something like:

vars: &vars !inc test.yaml

x:
<< *vars

Idea: include from remote storages (notably git)

Hi,
I like this project very much, this is exactly what I need for one of mine.
The only thing I would miss is ability to include from remote storage types (especially git).
I thought I would ask whether you would consider this to be a good idea before I'll start programming something by my own or before making a pull request here if you would agree.

I also might appreciate comments support that is present in ruamel. How feasible do you think it would be to port your library to ruamel?

Thanks a lot,
kind regards
Roman

readers_map not getting picked up: RuntimeError: Un-supported file name

I have the following code:

from yamlinclude import YamlIncludeConstructor

readers = {"yaml": "YamlReader", "yml": "YamlReader", "j2": "YamlReader", "yaml.j2": "YamlReader"}

YamlIncludeConstructor(reader_map=readers).add_to_loader_class(loader_class=yaml.FullLoader)

# Read the main yaml file and include the other yaml files.
with open(file_path) as f:
    data = yaml.load(f.read(), Loader=yaml.FullLoader)

I'm attempting to read files named "filename.yaml.j2". With the above code, I'm still getting the:
RuntimeError: Un-supported file name

What am I doing wrong here?