hifis-net / ansible-collection-toolkit

This Ansible collection provides production-ready Ansible roles used for providing services used in research and by research software engineers, but not exclusively.

Ansible Collection - hifis.toolkit

This collection provides production-ready Ansible roles used for providing services used in research and by research software engineers, but not exclusively. The following use cases are supported:

• DevOps platform:
  • GitLab
  • deploy GitLab-Runner with a focus, but not limited, on Openstack autoscaling
  • Redis
• Help desk:
  • Zammad
• High Availability (HA) / Load Balancing:
  • HAProxy
  • Keepalived
• OS-related:
  • unattended-upgrades
  • netplan
  • distribute authorized SSH keys to users

Looking for the unattended_upgrades role?

You can now find it under roles/unattended_upgrades.

We moved our existing Ansible roles into a single collection to deduplicate code and have a common test suite for all roles. We decided to reuse the unattended_upgrades repository as a collection repo as it is our most popular role.

Minimum required Ansible-version

• Ansible >= 2.15

Installation

Install the collection via ansible-galaxy:

ansible-galaxy collection install hifis.toolkit

Contributing

See CONTRIBUTING.md.

License

Apache-2.0

Author

This collection is maintained by HIFIS Software Services.

ansible-collection-toolkit's Issues

Support for Wifi interfaces

I needed wifi support for some of my devices but it wasn't supported if I'm not mistaken. I implemented it in a fork, would you be interested in a PR?

v4.0.0 release summary

UPGRADE NOTES:

The hifis.unattended_upgrades role has been migrated to the hifis.toolkit collection. Please ensure that you install the new collection as follows:

ansible-galaxy collection install hifis.toolkit

To continue using the unattended_upgrades role as before, please include it as described:

- hosts: all
  roles:
     - role: hifis.toolkit.unattended_upgrades

Detach fork

Allow to configure HTTP/3 in HAProxy

This requires to build HAProxy with Quic-Support. This does not work with Openssl as it does not implement Quic.

Instead, the only option is to use the https://github.com/quictls/openssl fork.

Verify, if this is something we can and should support via this role.

v2.0.0 release summary

UPGRADE NOTES AND BREAKING CHANGES:

If you have used this role before version 2.0.0, the files 20auto-upgrades and 50unattended-upgrades will differ from the system defaults (instead of the configuration being placed in a separate file, as we do now).
These can be left as-is as they will be overridden.
During OS upgrades, when asked if these files should be overwritten by the maintainer's package, say yes.
They will then be reset to their default states, and you won't be asked these questions again.

Add contribution guide

Rename default branch to `main`

Follow up from #9 (comment)

Version 3.2.1 doesn't seem to be available on ansible galaxy

I can find 3.2.1 on GitHub: https://github.com/hifis-net/ansible-role-unattended-upgrades/releases/tag/v3.2.1, but not on ansible galaxy: https://galaxy.ansible.com/ui/standalone/roles/hifis/unattended_upgrades/

Fix badges with Ansible Galaxy NG

Add support for Debian Bookworm

If simple use gitlab_external_url: "https" then don`t get SSL

Hello! Thanks for ansible-role-gitlab.
I try use it on ubuntu-2004-lts.

If i use HTTPS in vars:

  vars:
    gitlab_external_url: "https://gitlab.51.250.38.92.sslip.io"

I can error

Final gitlab nginx config without ssl:

cat /var/opt/gitlab/nginx/conf/gitlab-http.conf
server {
  listen *:80;
  server_name gitlab.51.250.38.92.sslip.io;
}

But if i run

sudo EXTERNAL_URL="https://gitlab.51.250.38.92.sslip.io" apt-get install gitlab-ce

I get letsencypt certificate.

Check for batched background migrations to be finished

In GitLab 13, background migrations were not batched. In GitLab 14 and later, this type of migration was superseded by batched background migrations. We should consider adjusting the task for waiting for background migrations to finish.

Add second test instance to molecule unattended_upgrades scenario

Follow-up #121 (review)

v3.0.0 release summary

UPGRADE NOTES AND BREAKING CHANGES:

As of this release, all Apt options for unattended-upgrades made in the default OS configuration files /etc/apt/apt.conf.d/20auto-upgrades and /etc/apt/apt.conf.d/50unattended-upgrades are now completely overwritten instead of being merged. This means that from now on only the options set by this role are active.

"Bullseye-Workaround" needs to be applied to bookworm and later as well

Debian bookworm needs the same additional 'origin=Debian,codename=${distro_codename}-security,label=Debian-Security' as bullseye (as will probably all future releases for the foreseeable future), but tasks/unattended-upgrades.yml currently only applies the extra origin line on bullseye.

Version 3.3.0 doesn't seem to be available on ansible galaxy

Same problem (and workaround) as in #169? 😢

Allow keepalived to be run as a non root user

keepalived-non-root.service allows keepalived to be run as a non root user, but with specific added capabilities to allow all the functionality that keepalived needs.

Created from https://gitlab.com/hifis/ansible/keepalived-role/-/issues/38

Enable role argument validation based on an argument specification

Once we place the argument specification into the meta/argument_specs.yml file, Ansible will automatically use it to validate variables before executing an Ansible role.

See https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html#role-argument-validation

Remove official support for EOL Ubuntu 18.04

Unattended-Upgrade::Origins-Pattern from 50unattended-upgrades apparently can't be "overruled"

It seems that the way that apt and unattendedupgrades handle the config file parsing, Unattended-Upgrade::Origins-Pattern will be evaluated cumulatively and cannot be overwritten in a "higher-ranking" config file (neither on buster, bullseye or bookworm).

While this won't pose a problem for "additional" origins, it is currently impossible to limit the allowed origins to only those given in the playbook (or the role's defaults). I don't have an idea for a solution just yet, but I think this limitation is at least something that should be mentioned in the Readme until it's solved

How to test: note the allowed origins in /etc/apt/apt.conf.d/50unattended-upgrades and those in 90-ansible-unattended-upgrades, then run unattended-upgrades --dry-run -v and note the list of allowed origins containing all entries of both files

Test role via Molecule

Upgrade from jnv/ansible-role-unattended-upgrades - stray 50unattended-upgrades?

Hi, I am about to migrate from https://github.com/jnv/ansible-role-unattended-upgrade to this role. Just replacing the role seems to cause a stray 50unattended-upgrades file to be left in /etc/apt/apt.conf.d (the new file is called 90-ansible-unattended-upgrades).
I can manually remove 50unattended-upgrades, but I didn't find this in the docs.
Can someone confirm that this is indeed something that has to be done?

[Feature Request] Use force_apt_get

See jnv/ansible-role-unattended-upgrades#64

I get this on ubuntu 18:

[WARNING]: Could not find aptitude. Using apt-get instead

Please use force_apt_get: yes to avoid this.

Make collection REUSE-compliant

See https://reuse.software/tutorial/

Remove support for OS that reached EOL

Allow roles to run with INJECT_FACTS_AS_VARS set to false

Currently, the role failed to run with INJECT_FACTS_AS_VARS set to false as the required ansible_* variables (I believe they are ansible_distribution and ansible_distribution_release in this role) are not defined.

The configuration INJECT_FACTS_AS_VARS and the Ansible fact namespace ansible_facts.* has been added in Ansible 2.5. In the porting guide of that version it stated that:

A new configuration variable, inject_facts_as_vars, has been added to ansible.cfg. Its default setting, ‘True’, keeps the 2.4 behavior of facts variables being set in the old ansible_* locations (while also writing them to the new namespace). This variable is expected to be set to ‘False’ in a future release. When inject_facts_as_vars is set to False, you must refer to ansible_facts through the new ansible_facts.* namespace.

It was also confirmed by Ansible developer on reddit that INJECT_FACTS_AS_VARS was planned to be deprecated in the future.

Therefore, it would be great if the role supported INJECT_FACTS_AS_VARS=false as well.

To fix this just change all ansible_* variables to ansible_facts.*. This should work with ansible versions >= 2.5. I can do a PR when I have time.

Some configurations options are missing

Hello :)

I'm looking at the source code of unnattend-upgrades ( https://github.com/mvo5/unattended-upgrades ) and they are new options that are not configurable via this role.

Theses new options could be added to this role

Add Debain support

Follow-up hifis-net/ansible-role-haproxy#125

Remove support for ansible-core 2.12

Ansible-core 2.12 is now unmaintained - support ended in May 2023.

`unattended_dl_limit` doesn't work

I have passed unattended_dl_limit: 40000 to the role:

- name: Set Unattended upgrades
  ansible.builtin.import_role:
    name: hifis.unattended_upgrades
  vars:
    unattended_dl_limit: 40000
  when: ansible_facts['os_family'] == 'Debian'

But after running the playbook successfully, the change is not made in the 50unattended-upgrades file on the destination:

// Use apt bandwidth limit feature, this example limits the download
// speed to 70kb/sec
//Acquire::http::Dl-Limit "70";

Looking at the defaults for the role, I noticed that unattended_dl_limit is commented out (see below), so perhaps you are already aware of this, but I can't find any mention of it in the repo's README, so I figured I'd share anyways to be sure :)

# Use apt bandwidth limit feature, this example limits the download speed to 70kb/sec
#unattended_dl_limit: 70

I would love if this feature were implemented again!

Ansible skips almost the whole process?

Hi there,
I'm not an expert in Ansible, and I never used unattended-upgrades as a sysadmin before, but it looks strange to me that so many steps were skipped on my Hetzner ubuntu-22.04 server. Is that right? Is it configured as it used to be?

I would have expected Test apt-daily timer expression and Deploy apt-daily timer to be ok or at least status changed, but nope, skipped. Is that right? Is my configuration enough?

My Ansible playbook entry:

- hosts: all
  roles:
    - role: hifis.unattended_upgrades
      unattended_origins_patterns:
        - "origin=Ubuntu,archive=${distro_codename}-security"
        - "o=Ubuntu,a=${distro_codename}"
        - "o=Ubuntu,a=${distro_codename}-updates"
        - "o=Ubuntu,a=${distro_codename}-proposed-updates"

My Ansible version:

ansible-playbook [core 2.15.3]
  python version = 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/opt/homebrew/Cellar/ansible/8.3.0/libexec/bin/python)
  jinja version = 3.1.2
  libyaml = True

Added custom of apt-daily timers apt-daily-upgrade

Hello

It seems to me that an important feature is missing.

• The feature of being able to decide when to DOWNLOAD updates ( apt-daily.timer)
• The feature of being able to decide when to INSTALL updates (apt-daily-upgrade.timer)

This happens in the following systemd timers

mabed@mabed ~ → systemctl list-timers | grep apt 
Fri 2023-01-20 12:34:48 CET 7min left      Fri 2023-01-20 06:08:39 CET 6h ago       apt-daily.timer              apt-daily.service
Fri 2023-01-20 13:08:27 CET 40min left     Fri 2023-01-20 07:09:38 CET 5h 18min ago apt-daily-upgrade.timer      apt-daily-upgrade.service

I don't know their default value because I modify them on my servers

There is no role but an ansible collection that we can add to this role.

*ansible.builtin.systemd

What do you think?

I will start working on the PR

Mathieu

Override configuration in a separate apt.conf.d file

See jnv/ansible-role-unattended-upgrades#32 for details.

Please merge jnv/ansible-role-unattended-upgrades#66, which resolves this by moving the configuration into a new file that doesn't overwrite default settings.

Test Keepalived failovers with Molecule

It is a crucial part to test that the failover procedure works as expected after changes have been made. For that reason we need to test this within Molecule's verify stage. In order to do so we will set up two keepalived / haproxy instances and a back-end server which is a dummy nginx web-server. If one keepalived / haproxy instance is stopped the other need to take over and the back-end still needs to be available.

Created from https://gitlab.com/hifis/ansible/keepalived-role/-/issues/8

Change caused by indentation

When you supply unattended_origins_patterns with the exact same contents as the provided defaults, the rollout will report a change, because of different indentation in the template.

 Unattended-Upgrade::Origins-Pattern {
-     "origin=Debian,codename=${distro_codename},label=Debian-Security";
-     "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
+      "origin=Debian,codename=${distro_codename},label=Debian-Security";
+      "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
   };

Include hifis.gitlab_runner role into collection

https://github.com/hifis-net/ansible-role-gitlab-runner

Migrate role to role in a collection

It feels like support for conventional roles is becoming more and more limited (especially in the Ansible Galaxy). Collections have been the main distribution format for Ansible content for some time now. They can include playbooks, roles, modules, and plug-ins around specific topics.

With the next major release (v4.0.0) we are planning to migrate the hifis.unattended_upgrades role to an Ansible collection. Step by step we will then add all our Ansible roles to this collection. This simplifies maintenance and updating of all our roles and thus enables the long-term development and more regular releases of this role.

We try to respond to your wishes in order to make the migration process as smooth as possible for the user community. We are happy to get your feedback!

SSH public key comment not passed into flatcar linux config on Debian10

If using a predefined SSH key pair on Debian 10, the comment of the public key will be removed (probably by the community.crypto.openssh_keypair module) when added to the flatcar-linux-config.yml.j2 template.

See failed CI job: https://gitlab.com/hifis/ansible/gitlab-ci-openstack/-/jobs/1436902366#L1153

(Workaround: Removed the comment from test public key: https://gitlab.com/hifis/ansible/gitlab-ci-openstack/-/merge_requests/28/diffs?commit_id=dbf941e30cffe7781da2bc094e71e63d2b48a5aa)

Recreated from https://gitlab.com/hifis/ansible/gitlab-ci-openstack/-/issues/26

apt options are not overridden but merged

Hello,
thank you for this wonderful Ansible role, especially for the approach of overriding the unattended-upgrades config file instead of replacing it.
Unfortunately, this doesn't work as intended because config options for apt are not overridden but merged. For details see the solution to this old issue.

Therefore options should be cleared before the override is applied, e.g.
#clear Unattended-Upgrade::Origins-Pattern;
should precede the line
Unattended-Upgrade::Origins-Pattern {

Kind regards

Include hifis.keepalived role into collection

https://github.com/hifis-net/ansible-role-keepalived

Add changelog

Molecule folder not linted by molecule

When providing . as the folder path of the lintables to ansible-lint it does not lint the folder molecule/default containing converge.yml, prepare.yml, verify.yml, hence folder molecule/ need to be added to the ansible-lint call in the molecule lint commands.

Manage file permissions of ignition configuration via Ansible

Created from https://gitlab.com/hifis/ansible/gitlab-ci-openstack/-/issues/25

Include hifis.ssh-keys role into collection

https://github.com/hifis-net/ansible-role-ssh-keys

Documentation role name mismatch

The README uses a dash instead of an underscore.

ERROR! this task "ansible.buildin.import_tasks" has extra params, which...

ERROR! this task 'ansible.buildin.import_tasks' has extra params, which is only allowed in the following modules: include_tasks, raw, command, import_tasks, add_host, include, group_by, meta, include_role, win_shell, win_command, script, shell, import_role, set_fact, include_vars

The error appears to be in '/ansible/AG_Linux/roles/hifis.unattended_upgrades/tasks/main.yml': line 2, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

---

• name: "Import tasks from the unattended-upgrades playbook"
  ^ here

---

I got the issue above.

Ansible version 2.9.6
Python version 3.8.10

Fix installation of powermgmt-base package

See https://github.com/hifis-net/ansible-role-unattended-upgrades/blob/v1.12.2/tasks/unattended-upgrades.yml#L15

ansible.builtin.import_tasks problem

I write a role which depend on this role and I get:
ERROR! this task 'ansible.builtin.import_tasks' has extra params, which is only allowed in the following modules: win_shell, include_tasks, include_role, group_by, import_tasks, add_host, raw, import_role, script, set_fact, include_vars, shell, meta, include, win_command, command

The error appears to be in '/home/fhackenberger/Dokumente/ansible/rexx.apt/tests/roles/hifis.unattended_upgrades/tasks/main.yml': line 2, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

I Use ansible 2.9.6
also ansible.builtin.import_task is only available in ansible >= 2.4, but your meta says:
min_ansible_version: "1.4"

Determine the current master instance dynamically and re-configure the services

In case the fail-over took place and the master is now one of the relicas the role need to determine the new master instance dynamically and re-configure the services accordingly, otherwise the original master will bet set as master again.

Recreated from https://gitlab.com/hifis/ansible/redis-role/-/issues/18

ValueError: not enough values to unpack (expected 2, got 1) on Ubuntu Jammy

With Ubuntu Jammy I get the following error after running the Ansible role:

$ unattended-upgrades --debug --dry-run                                                                                                                                                                                 
Starting unattended upgrades script                                                                                                                                                                                                         
Allowed origins are: Ubuntu:jammy, Ubuntu:jammy-security, Ubuntu:jammy-updates, Ubuntu:jammy-proposed, UbuntuESMApps:jammy-apps-security, UbuntuESM:jammy-infra-security, Docker:jammy
Initial blacklist: 
Initial whitelist (not strict):        
An error occurred: not enough values to unpack (expected 2, got 1)
Traceback (most recent call last):                 
  File "/usr/bin/unattended-upgrades", line 1993, in main
    res = run(options, rootdir, mem_log, logfile_dpkg,
  File "/usr/bin/unattended-upgrades", line 2134, in run                                                              
    cache = UnattendedUpgradesCache(rootdir=rootdir)                                                                  
  File "/usr/bin/unattended-upgrades", line 171, in __init__
    apt.Cache.__init__(self, rootdir=rootdir)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 152, in __init__
    self.open(progress)                                                                                               
  File "/usr/bin/unattended-upgrades", line 333, in open
    self.apply_pinning(self.pinning_from_config())
  File "/usr/bin/unattended-upgrades", line 279, in pinning_from_config
    if not is_allowed_origin(pkg_file, self.allowed_origins):
  File "/usr/bin/unattended-upgrades", line 981, in is_allowed_origin  
    if match_whitelist_string(allowed, origin):                                                                       
  File "/usr/bin/unattended-upgrades", line 820, in match_whitelist_string   
    (what, value) = [s.strip().replace("%2C", ",")  
ValueError: not enough values to unpack (expected 2, got 1)    
Extracting content from /var/log/unattended-upgrades/unattended-upgrades-dpkg.log since 2022-11-02 13:36:04
Traceback (most recent call last):                
  File "/usr/bin/unattended-upgrades", line 2522, in <module>
    sys.exit(main(options))         
  File "/usr/bin/unattended-upgrades", line 1993, in main
    res = run(options, rootdir, mem_log, logfile_dpkg,
  File "/usr/bin/unattended-upgrades", line 2134, in run
    cache = UnattendedUpgradesCache(rootdir=rootdir)
  File "/usr/bin/unattended-upgrades", line 171, in __init__
    apt.Cache.__init__(self, rootdir=rootdir)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 152, in __init__
    self.open(progress)
  File "/usr/bin/unattended-upgrades", line 333, in open
    self.apply_pinning(self.pinning_from_config())
  File "/usr/bin/unattended-upgrades", line 279, in pinning_from_config
    if not is_allowed_origin(pkg_file, self.allowed_origins):
  File "/usr/bin/unattended-upgrades", line 981, in is_allowed_origin
    if match_whitelist_string(allowed, origin):
  File "/usr/bin/unattended-upgrades", line 820, in match_whitelist_string
    (what, value) = [s.strip().replace("%2C", ",")
ValueError: not enough values to unpack (expected 2, got 1) 

With the default unattended-upgrades configuration it works fine.
I compared the original and the newly generated configuration and it seems that Unattended-Upgrade::Origins-Pattern changed to Unattended-Upgrade::Allowed-Origins with Jammy. The Ubuntu documentation also was updated https://help.ubuntu.com/community/AutomaticSecurityUpdates

When I change Origins-Pattern to Allowed-Origins it works again. So I guess there should be a check for the distro codename.