mirkolenz / makejinja Goto Github PK

Add support for reading from standard input and writing to standard output. Both of these features require #57 to be completed first. This would allow something like this:

cat demo.txt.jinja | makejinja > demo.txt

Currently, the tool requires a dictionary to be given as input and output. This may be inconvenient for some use cases. We plan to add support in two phases:

1. Accept files and folders to be provided as input.
2. Handle the special case that there is exactly one input and the output has a file-like pattern as well.

Is your feature request related to a problem? Please describe.

Hi again 👋

I'm looking at using makejinja to replace the Ansible playbook I created to template out a ton of files. I was relieved to find that this project pretty much checks all my boxes as a CLI tool. It's fast and has a ton of features I love like not creating the file if the whole contents of the file are wrapped in an if statement that evaluates to false. I also love how we can pass files and data-vars that get interpreted into variables.

One thing I'm lacking however is some more jinja filters. I'm sure I could use a custom loader for this but I wanted to try and keep my usage only to the CLI.

Some of these filters I'm missing that are included in Ansible are:

• nthhost (src) (docs)
• password_hash (src) (docs)

It would be cool if these were supported but I understand if this maintenance would be burden to you.

Anyways I really love this tool and am trying to make it fit in with my use-cases.

Edit: I'm now seeing the docs on building a custom loader, I'll give this a shot.

Describe the bug

I usually write all my files with a trailing new line, it looks like this isn't preserved when templating out the files with makejinja. This is causing some issues in my linters because they expect a empty newline at the bottom of all files.

To Reproduce*

Add a empty newline a the bottom of the file, then when you template it out makejinja removes that newline.

Expected behavior

This newline to be preserved.

Is your feature request related to a problem? Please describe.

I have some templates that do not render because I am not setting the --keep-empty flag (which is expected), however this can lead to empty folders if a template was never rendered.

Take this file in the path ./input/discord-template-notifier/app/test.yaml.j2 for example

{% if discord_template_notifier.enabled | default(false) %}
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./secret.sops.yaml
  - ./helmrelease.yaml
{% endif %}

When I use makejinja to template this to the output of ./output I see empty folders for discord-template-notifier/app because that if statement evaluate to false

Describe the solution you'd like

Do not create folders if there are templates that won't get rendered to it.

Describe alternatives you've considered

For now I am running find ./output -type d -empty -delete to clean up these empty folders after running makejinja

Currently, the same data is used for every rendered file. It would be nice to additionally support template-specific data that is only used for a given file prefix. Could be useful for overriding values for certain files.

Is your feature request related to a problem? Please describe.

I have request that would reduce some logic needed in files if a var is not true. In this directory all files are wrapped with an if statement of if bootstrap_distribution == 'talos'. If you take a look around the ./bootstrap/templates folder you will see I am doing this quite a bit to exclude folders and files from being templated based on variables.

Describe the solution you'd like

I was thinking it would be cool to support like a .pathignore.j2 file where the contents would hold the logic if that folder, subfolders and files get templated. I could drop this file in ./bootstrap/templates/kubernetes/talos and omit the logic in the files.

{% bootstrap_distribution == 'talos' %}

I am not sure if this can be done in custom Loader but I am all ears if there is already a way to do this but needs some special sauce added on my end.

Describe the bug

Hi 👋🏼

I am trying to override vars in a --data-var arg but I am getting an error, I am probably do it wrong but I have tried a whole bunch of different values for --data-var

command

makejinja --input test.yaml.j2 --output test.yaml --data config.yaml --data-var "test=true" --jinja-suffix ".j2"

output

Traceback (most recent call last):
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/bin/makejinja", line 8, in <module>
    sys.exit(makejinja_cli())
             ^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/typed_settings/cli_click.py", line 247, in new_func
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/cli.py", line 36, in makejinja_cli
    makejinja(config)
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/app.py", line 29, in makejinja
    data = load_data(config)
           ^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/app.py", line 283, in load_data
    for key, value in config.data_vars:
        ^^^^^^^^^^
ValueError: too many values to unpack (expected 2)

Describe the bug

It looks like some people are reporting this issue with using makejinja with Python 3.10. I'm pretty sure you only support 3.11+ so I'm kinda confused how they are even able to install the latest version on 3.10.

I'm not sure if there's something you can do on your end but I wanted to report it here in case others have this issue.

I already have a validation function to check which Python version they have install.

https://github.com/onedr0p/flux-cluster-template/blob/eec9fcf7f0327ee3471185d995965c3ce8cce163/bootstrap/scripts/validation.py#L70L73

To Reproduce

Install and run makejinja --force on Python 3.10

Expected behavior

Output directory to not be removed.

Additional context

My config file: https://github.com/onedr0p/flux-cluster-template/blob/main/makejinja.toml

Here's the issue that has some more information

onedr0p/cluster-template#1174

Is your feature request related to a problem? Please describe.

I found myself needing to put the makejinja config file in a separate location, I noticed in the CLI help it says

Instead of passing CLI options, you can also write them to a file called makejinja.toml in your working directory.

Describe the solution you'd like

I would love to put this file somewhere else and reference it in the makejinja CLI (e.g. makejinja --config ./.github/makejinja.toml)

This could be pretty sweet if you want to pass different config files for templating the same input but have slightly different data-vars or data config options.

makejinja --config ./test1/makejinja.toml
makejinja --config ./test1/makejinja-slightly-different-config-but-same-input.toml

I realize I could have makejinja config files in different directories and run it from there but that could mess up the pathing in the config file.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): lock file maintenance

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update determinatesystems/magic-nix-cache-action action to v4
• chore(deps): update determinatesystems/nix-installer-action action to v10

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Is your feature request related to a problem? Please describe.

I was wondering if it currently possible to validate data items before the templating happens. I was poking around the docs and I see Tests but not sure if that is what I should be looking at.

Describe the solution you'd like

I would love to be able run validation on user supplied variables from the data or data-vars flags.

For example, I was wondering if something like this pseudo-code was possible:

def addressing(value: str) -> bool:
    try:
        netaddr.IPNetwork(value)
        return True
    except netaddr.core.AddrFormatError:
        return False

class Loader:
    def filters(self):
        return [nthhost, encrypt]

    def tests(self):
        return [
            addressing(self.data.bootstrap_cluster_cidr),
            addressing(self.data.bootstrap_service_cidr)
        ]

Add parameters like --exec-pre and --exec-post that allow users to specify arbitrary shell commands to run before/after makejinja (e.g., for preparing files or running clean-up scripts). These should be allowed to be passed multiple times, thus having type list[str].

Describe the bug

Adding --force does not work due to directory already existing from a previous makejinja --force execution

To Reproduce

command

makejinja --force

config

[makejinja]
inputs = ["./bootstrap/templates"]
output = "./"
data = ["./bootstrap/vars/addons.yaml", "./bootstrap/vars/config.yaml"]
import_paths = ["./bootstrap/scripts"]
loaders = ["loader:Loader"]
jinja_suffix = ".j2"

# Block delimiters are changed to avoid conflicts with Renovate
# https://github.com/renovatebot/renovate/discussions/18470
[makejinja.delimiter]
block_start = "#%"
block_end = "%#"
comment_start = "{#"
comment_end = "#}"
variable_start = "{%"
variable_end = "%}"

error

Load data '/Users/devin/Code/repos/onedr0p/flux-template-cluster/bootstrap/vars/addons.yaml'
Load data '/Users/devin/Code/repos/onedr0p/flux-template-cluster/bootstrap/vars/config.yaml'
Render file '/Users/devin/Code/repos/onedr0p/flux-template-cluster/bootstrap/templates/.sops.yaml.j2' -> '/Users/devin/Code/repos/onedr0p/flux-template-cluster/.sops.yaml'
Create folder '/Users/devin/Code/repos/onedr0p/flux-template-cluster/bootstrap/templates/ansible' -> '/Users/devin/Code/repos/onedr0p/flux-template-cluster/ansible'
Traceback (most recent call last):
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/bin/makejinja", line 8, in <module>
    sys.exit(makejinja_cli())
             ^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/rich_click/rich_command.py", line 126, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/typed_settings/cli_click.py", line 247, in new_func
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/cli.py", line 37, in makejinja_cli
    makejinja(config)
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/app.py", line 59, in makejinja
    handle_input_folder(
  File "/Users/devin/Code/repos/onedr0p/flux-template-cluster/.venv/lib/python3.11/site-packages/makejinja/app.py", line 155, in handle_input_folder
    output_path.mkdir()
  File "/opt/homebrew/Cellar/[email protected]/3.11.7/Frameworks/Python.framework/Versions/3.11/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/Users/devin/Code/repos/onedr0p/flux-template-cluster/ansible'
task: Failed to run task "configure": exit status 1

Expected behavior

That adding --force will not fail if the directory already exists.