GithubHelp home page GithubHelp logo

koji-ansible's Introduction

koji-ansible

https://img.shields.io/badge/dynamic/json?style=flat&label=galaxy&prefix=v&url=https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/index/ktdreyer/koji_ansible/&query=highest_version.version

Ansible modules to manage Koji resources.

This is not about installing Koji. Instead, it is a way to declaratively define things within Koji, where you might normally use the koji CLI.

koji_tag

The koji_tag module can create, update, and delete tags within Koji. It can also manage the tag inheritance, packages list and groups list for a tag.

- name: Create a koji tag for the ceph product
  koji_tag:
    name: ceph-3.1-rhel-7
    arches: x86_64
    state: present
    packages:
      kdreyer:
        - ansible
        - ceph
        - ceph-ansible
    groups:
      srpm-build:
        - rpm-build
        - fedpkg

koji_target

The koji_target module can create, update, and delete targets within Koji.

- name: Create a koji build target for Fedora 34
  koji_target:
    name: f34-candidate
    build_tag: f34-build
    dest_tag: f34-updates-candidate
    state: present

koji_external_repo

The koji_external_repo module can create, update, and delete external repositories within Koji.

- name: Create an external repo for CentOS "CR"
  koji_external_repo:
    name: centos7-cr
    url: http://mirror.centos.org/centos/7/cr/$arch/
    state: present

You can then configure these repositories (and their priorities) on each of your Koji tags with the external_repos parameter to the koji_tag module.

koji_cg

The koji_cg module can grant or revoke access to a content generator for a user account.

This user account must already exist in Koji's database. For example, you may run an authenticated koji hello command to create the account database entry.

- name: Grant access to the rcm/debbuild account
  koji_cg:
    koji: mykoji
    name: debian
    user: rcm/debbuild
    state: present

Your Koji Hub must be version 1.19 or newer in order to use the new listCGs RPC.

koji_btype

The koji_btype module can add new build types. These are typically in support of content generators.

(Koji only supports adding new build types, not deleting them.)

- name: Add debian build type to Koji
  koji_btype:
    koji: mykoji
    name: debian
    state: present

koji_archivetype

The koji_archivetype module can add new archive types. This allows Koji to recognize new build archive files, for example .dsc files. These are typically in support of content generators.

(Koji only supports adding new archive types, not deleting them.)

Your Koji Hub must be version 1.20 or newer in order to use the new addArchiveType RPC.

- name: Add dsc archive type
  koji_archivetype:
    name: dsc
    description: Debian source control file
    extensions: dsc
    state: present

koji_host

The koji_host module can add new hosts and manage existing hosts.

Koji only supports adding new hosts, not deleting them. Once they're defined, you can enable or disable the hosts with state: enabled or state: disabled.

- name: Add new builder1 host
  koji_host:
    name: builder1.example.com
    arches: [x86_64]
    state: enabled
    channels:
      - default
      - createrepo

If you specify channels that do not yet exist, Ansible will create them. For example, if you are setting up a new builder host for OSBS, you can specify container in the list of channels, and Ansible will automatically create that new "container" channel when it configures the host.

koji_user

The koji_user module can add new users and manage existing users and permissions.

Koji only supports adding new users, not deleting them. Once they're defined, you can enable or disable the users with state: enabled or state: disabled.

- name: Add new kdreyer user
  koji_user:
    name: kdreyer
    state: enabled
    permissions: [admin]

koji_tag_inheritance

The koji_tag module (above) is all-or-nothing when it comes to managing tag inheritance. When you set inheritance with koji_tag, the module will delete any inheritance relationships that are not defined there.

In some cases you may want to declare some inheritance relationships within Ansible without clobbering other existing inheritance relationships. For example, MBS will dynamically manage some inheritance relationships of tags, and you do not want Ansible to fight MBS.

To declare inheritance relationships with finer granularity, you may use the koji_tag_inheritance module.

- name: set devtoolset-7 as a parent of ceph nautilus
  koji_tag_inheritance:
    parent_tag: sclo7-devtoolset-7-rh-release
    child_tag: storage7-ceph-nautilus-el7-build
    priority: 25

This will only mange that single parent-child relationship between the two tags, and it will not delete any other inheritance relationships.

Another approach is to have MBS operate on a dedicated "-modules" tag and then inherit from that, so that you do not have to use this Ansible module.

koji_tag_packages

The koji_tag module (above) is all-or-nothing when it comes to managing packages. When you set packages with koji_tag, the module will delete any packages that are not defined there.

In some cases you may want to declare some packages within Ansible without clobbering existing packages. For example, if you have a separate tool that might add or remove packages from tags dynamically, you do not want Ansible to fight that other tooling.

To declare packages with finer granularity, you may use the koji_tag_packages module.

- name: ensure ceph packages are present and ownership set
  koji_tag_packages:
    tag: ceph-3.1-rhel-7
    packages:
      kdreyer:
        - ceph
      aschoen:
        - ansible
    state: present

- name: ensure koji packages are absent
  koji_tag_packages:
    tag: ceph-3.1-rhel-7
    packages:
      kdreyer:
        - koji
    state: absent

This will only mange the packages defined and will not change any other packages on the tag.

koji_call

The koji_call module allows you to send raw RPCs to the Koji hub. This exposes the entire Koji API to you directly.

Why would you use this module instead of the higher level modules like koji_tag, koji_target, etc? This koji_call module has two main uses-cases:

  1. You may want to do something that the higher level modules do not yet support. It can be easier to use this module to quickly prototype out your ideas for what actions you need, and then write the Python code to do it in a better way later. If you find that you need to use koji_call to achieve functionality that is not yet present in the other koji-ansible modules, please file a Feature Request issue in GitHub with your use case.
  2. You want to write some tests that verify Koji's data at a very low level. For example, you may want to write an integration test to verify that you've set up your Koji configuration in the way you expect.

Note that this module will always report "changed: true" every time, because it simply sends the RPC to the Koji Hub on every ansible run. This module cannot understand if your chosen RPC actually "changes" anything.

- name: make a raw API call:
  koji_call:
    name: getTag
    args: [f34-build]
  register: call_result

- debug:
    var: call_result.data

This will print the tag information for the Fedora 34 -build tag. It is similar to running koji taginfo f34-build on the command-line.

Koji profiles

You must tell koji-ansible which Koji client profile to use.

Here is an example of setting a profile explicitly on the task:

- name: Create a koji tag for the ceph product
  koji_tag:
    koji: kojidev
    name: ceph-3.1-rhel-7
    arches: x86_64
    state: present

The koji: kojidev setting means Ansible will search ~/.koji/config.d/*.conf and /etc/koji.conf.d/*.conf for the [kojidev] config section and perform the tag management on that Koji hub listed there.

To avoid specifying this koji: argument on every task, you can set the KOJI_PROFILE environment variable when running ansible-playbook. koji-ansible will fall back to using KOJI_PROFILE for the tasks that have no explicit koji: argument:

KOJI_PROFILE=kojidev ansible-playbook -v my-koji-playbook.yaml

Installing from Ansible Galaxy

We distribute koji-ansible through the Ansible Galaxy.

If you are using Ansible 2.9 or greater, you can install koji-ansible like so:

ansible-galaxy collection install ktdreyer.koji_ansible

This will install the latest Git snapshot automatically. Use --force upgrade your installed version to the latest version.

Using this Ansible Galaxy Collection inside a role

Here is an example of a simple playbook and role that uses this collection. playbook.yml calls one role named my-koji-project:

top
├── playbook.yml
└── roles
    └── my-koji-project
        ├── collections
        │   └── requirements.yml
        ├── meta
        │   └── main.yml
        └── tasks
            └── main.yml

The playbook.yml file is a small playbook that simply loads our role:

- name: Test a role that uses koji-ansible
  hosts: localhost
  gather_facts: false
  roles:
   - my-koji-project

The roles/my-koji-project/collections/requirements.yml file should require this collection (and a specific version, as described above):

collections:
- name: ktdreyer.koji_ansible

The roles/my-koji-project/meta/main.yml file tells Ansible to load any custom modules in this role from the ktdreyer.koji_ansible collection namespace:

collections:
- ktdreyer.koji_ansible

Lastly you can add your role's tasks as usual to roles/my-koji-project/tasks/main.yml:

- name: create the "my-product-1.0" tag
  koji_tag:
    name: my-product-1.0

Role and collection dependencies must be installed separately. This is true even if the dependencies are defined in the same requirements.yml.

For role dependencies:

ansible-galaxy install -r requirements.yml

For collection dependencies:

ansible-galaxy collection install -r requirements.yml

Running from a Git clone

Instead of using the Ansible Collection tarball, you can use this project directly from a Git clone. This is useful when hacking on the code.

These modules import common_koji from the module_utils directory.

One easy way to arrange your Ansible files is to symlink the library and module_utils directories into the directory with your playbook.

For example, if you have a koji.yml playbook that you run with ansible-playbook, it should live alongside these library and module_utils directories:

top
├── koji.yml
├── module_utils
└── library

and you should run the playbook like so:

ansible-playbook koji.yml

Investigating changes that happened outside Ansible

Koji tracks a history of everything in its database. You can view this history with the koji list-history and koji list-tag-history sub-commands.

For example, let's say that you wake up one morning to find that your Ansible playbook for your tags no longer matches up with what is configured live in Koji. Did someone else on your team make a change with the CLI without editing the playbook or notifying you? Who did it, and when? Use koji list-history --tag=my-tag to see the entire list of changes for your tag in the database. After a friendly chat with the person who made the change, you can work together to record the change within your Ansible playbook so your sources of truth remain consistent.

Generating a playbook from a live Koji instance

Do you have a Koji hub that has many tags, targets, and other settings that were crafted by hand over the years? You can use the ./utils/generate-playbook script to query your Koji hub and write an Ansible playbook that describes some or all of the tags. You can then store this YAML in Git. Other things beyond tags and targets (like content generators or users) are not yet supported.

This generate-playbook utility's output may not be the most elegant way to manage your Koji tags. There will be lots of repetition, because it will not use any Ansible variables, etc. The purpose of this utility is simply to help you get up and running quickly with koji-ansible.

TODO

  • Unit tests

koji-ansible's People

Contributors

ahills avatar andrewschoen avatar crungehottman avatar dependabot[bot] avatar hongweiliu17 avatar ian-panzica avatar ignatenkobrain avatar ktdreyer avatar mizdebsk avatar rhartman93 avatar stickler-ci avatar tojaj avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

koji-ansible's Issues

Extend koji_host to be able to configure channels

It would be nice if koji_host module was able to configure to which channels given host belongs to, for example:

koji_host:
  name: builder1.mydomain.com
  arches: [x86_64]
  channels:
    - default
    - createrepo

add external repo management

Fedora and CentOS use Koji's external repos feature.

For example, in Fedora's make-koji-release-tags script:

"${KOJICLI}" add-external-repo -t "module-f${release}-build" "f${release}-build" "https://kojipkgs.fedoraproject.org/repos/f${release}-build/latest/\$arch/"

I'm thinking we'd implement this with a new koji_external_repo module to create external repos, and then add a new external_repos parameter to the koji_tag module so we can associate the repos with each tag.

Relevant RPCs for CRUD'ing external repos:

  • createExternalRepo
  • getExternalRepo
  • editExternalRepo
  • deleteExternalRepo

and the RPCs for associating with tags:

  • getTagExternalRepos
  • addExternalRepoToTag
  • removeExternalRepoFromTag

RFE: Support for "merge mode" in external repos

Baiscally it is not possible to set up el8 targets which would like to use plain el8 repos...

❯ koji -p gdc list-external-repos --tag tools-el8-build
Pri External repo name        Mode       URL
--- ------------------------- ---------- ----------------------------------------
5   centos8                   bare       https://pulp1.na.intgdc.com/pulp/repos/repos/centos/81/testing/$arch/baseos/
7   centos8-appstream         bare       https://pulp1.na.intgdc.com/pulp/repos/repos/centos/81/testing/$arch/appstream/
8   centos8-powertools        bare       https://pulp1.na.intgdc.com/pulp/repos/repos/centos/81/testing/$arch/powertools/
10  epel8                     bare       https://pulp1.na.intgdc.com/pulp/repos/centos/epel/testing/8/$arch/

this is, for instance, what we have in our koji instance. Setting bare mode is MUST.

Could you add an option to override it? Thanks!

define state "choices" in modules' argument_spec

We have this pattern in all the modules:

    if state == 'present':
        # ensure the resource is present
    elif state == 'absent':
        # ensure the resource is absent
    else:
        module.fail_json(msg="State must be 'present' or 'absent'.",
                         changed=False, rc=1)

We can drop the final "else" clause and let AnsibleModule do that validation and error-handling for us with the choices setting to the argument_spec instead:

state=dict(type='str', choices=['present', 'absent'], required=False, default='present'),

Traceback when priority is not specified for inheritance

fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/home/brain/.ansible/tmp/ansible-tmp-1585328274.585013-221614686299361/AnsiballZ_koji_tag.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/brain/.ansible/tmp/ansible-tmp-1585328274.585013-221614686299361/AnsiballZ_koji_tag.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/brain/.ansible/tmp/ansible-tmp-1585328274.585013-221614686299361/AnsiballZ_koji_tag.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.ktdreyer.koji_ansible.plugins.modules.koji_tag', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.8/runpy.py\", line 206, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_koji_tag_payload_6dgu30_v/ansible_koji_tag_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_tag.py\", line 592, in <module>\n  File \"/tmp/ansible_koji_tag_payload_6dgu30_v/ansible_koji_tag_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_tag.py\", line 588, in main\n  File \"/tmp/ansible_koji_tag_payload_6dgu30_v/ansible_koji_tag_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_tag.py\", line 566, in run_module\n  File \"/tmp/ansible_koji_tag_payload_6dgu30_v/ansible_koji_tag_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_tag.py\", line 484, in ensure_tag\n  File \"/tmp/ansible_koji_tag_payload_6dgu30_v/ansible_koji_tag_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_tag.py\", line 222, in ensure_inheritance\n  File \"/tmp/ansible_koji_tag_payload_6dgu30_v/ansible_koji_tag_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_tag.py\", line 202, in normalize_inheritance\nKeyError: 'priority'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Would be better if I would get some human-readable error :)

"required" should not be used with "default" in argument_spec

In several modules, we are mixing "required" with "default" in the argument_spec, like so:

    module_args = dict(
        koji=dict(type='str', required=False),
        name=dict(type='str', required=True),
        permissions=dict(type='list', required=True),
        krb_principal=dict(type='str', required=False, default=None),
        state=dict(type='str', required=False, default='enabled'),
        ...

From https://docs.ansible.com/ansible/latest/dev_guide/developing_program_flow_modules.html,
required "should not be used in combination with default."

auto-discover all integration test files

When I'm working on a high-priority bug fix and I'm writing a new integration test, it's really easy to forget to add an integration test .yml file to that integration test's main.yml list. It's also dangerous because the integration tests will still be "green", even if I add a newly-failing test.

We should be able to use fileglob on the {{ playbook_dir }} to automatically discover all test .yml in files in the tree and include all of them.

add package group management

We need to add support to koji_tag to manage package groups.

Typically rel-eng does this with the CLI the command-line like so:

koji add-tag foo-1-build-base
...
koji add-group foo-1-build-base build
koji add-group-pkg foo-1-build-base build bash bzip2 coreutils cpio diffutils ...

So foo-1-build-base is a tag, and build is the group name. There are other groups in this tag, like appliance-build, srpm-build, etc.

koji_tag_inheritance cannot replace a parent with the same priority

I have three tags, my-child-tag, and parent-tag-a, and parent-tag-b. The child tag currently inherits from parent-tag-a:

$ kojidev taginfo my-child-tag
Tag: my-child-tag [22]
Arches: None
Groups: 
Tag options:
Inheritance:
  0    .... parent-tag-a [23]

If I try to switch that to parent-tag-b, the hub denies the operation:

  - name: Set partial inheritance
    koji_tag_inheritance:
      child_tag: my-child-tag
      parent_tag: parent-tag-b
      priority: 0
      state: present

The error from the hub is:

    __koji__kojidev.GenericError: Inheritance priorities must be unique (pri 0: [24, 23] )

To fix this bug, we need to delete (delete link) any existing relationship at the given priority when we set our new relationship.

"ImportError: No module named utils.display" on Ansible 2.7.10

On Fedora 29,

$ ansible-playbook --version
ansible-playbook 2.7.10
  config file = /home/kdreyer/.ansible.cfg
  configured module search path = ['/home/kdreyer/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.7.3 (default, Mar 27 2019, 13:41:07) [GCC 8.3.1 20190223 (Red Hat 8.3.1-2)]

I'm getting this error:

File "/tmp/ansible_koji_tag_payload_mu7d0l/__main__.py", line 9, in <module>
ImportError: No module named utils.display

error when swapping priority of two external repos

I have a tag that uses several external repositories, like so:

  - name: Add ktdreyer-el7-build tag
    koji_tag:
      name: ktdreyer-el7-build
      arches: x86_64
      external_repos:
      - repo: el7
        priority: 0
      - repo: el7-updates
        priority: 5
      - repo: el7-extras
        priority: 10
      - repo: epel7
        priority: 15

I ran into an interesting problem here. CentOS Extras ("el7-extras") happens to include ancient versions of some packages that are newer in EPEL 7 (eg. ansible 2.4 is in el7-extras, ansible 2.8 is in EPEL 7). I guess the CentOS administrators do not remove packages from CentOS 7 Extras that are gone in RHEL and subsequently added to EPEL 7.

Since Koji uses "priority" instead of NVR comparisons when it chooses which builds to index, I get the ancient version of ansible in my buildroot.

To fix this, I need to change the priorities to make epel7 higher than el7-extras. I swapped the priorities like so:

      - repo: epel7
        priority: 10
      - repo: el7-extras
        priority: 15

This raises an error:

    __koji__kojidev.GenericError: tag ktdreyer-el7-build already associated with an external repo at priority 10

It seems that we cannot handle this type of priority re-ordering correctly.

Add support for editing krb_principal and having multiple of them

krb_principal:
description:
- Set a non-default krb principal for this user. If unset, Koji will
use the standard krb principal scheme for user accounts.
- Warning, Koji only allows you to set this one time, at the point at
which you create the new account. You cannot edit the krb_principal
for an existing account.

This is not true anymore (since 1.19 or 1.20)...

❯ koji edit-user --help
Usage: koji edit-user <username> [options]
(Specify the --help global option for a list of other help options)

Options:
  -h, --help          show this help message and exit
  --rename=RENAME     Rename the user
  --edit-krb=OLD=NEW  Change kerberos principal of the user
  --add-krb=KRB       Add kerberos principal of the user
  --remove-krb=KRB    Remove kerberos principal of the user

add integration tests for ansible collection

The ./build-collection script generates our Ansible Collection tarball. It's important to test this script, because users will come to rely on the Ansible Collection over time.

We should build an integration test that tries to install this with ansible-galaxy collection install _build/ktdreyer-koji_ansible-0.0.0-git.230+9d7258a6.tar.gz and run some playbook with it.

Architecture for a buildroot tag should be compared as a set

Order of architectures in a buildroot tag doesn't matter. These two are equivalent from Koji point of view:

  • x86_64 ppc64le
  • ppc64le x86_64

E.g.:

TASK [rhceph3 : 3.1 build tag] ***************************************************************************************************************************************************************
changed: [rcm-compose-nightly-02] => {"changed": true, "stdout_lines": ["ceph-3.1-rhel-7-build: changed arches from \"ppc64le x86_64\" to \"x86_64 ppc64le\""]}

cache session information somehow

Currently every Ansible module authenticates its own unique Koji client session. This means we always send the login RPC to Koij Hub every time we want to make a change.

This is not slower than invoking the koji CLI many times, but we may be able to do better.

When we do a client login to the Koji Hub at the /ssllogin endpoint, we get back a XML-RPC response with a session-id (int) and a session-key (string). Example:

cbs -d --debug-xmlrpc hello

The Koji client then uses these two values to authenticate all the requests going forward. Subsequent HTTP requests for XML-RPC look like this:

/kojihub?session-id=123456&session-key=1234-asdf&callnum=0

where callnum is an integer that increases for every authenticated call.

In koji-ansible, we could return session-id, session-key, from a koji_login module, and have all the other modules use these variables if they are defined for the profile.

Note, we would also need to cache callnum in a way that every other koji-ansible module can increment it.

new koji_tag_inheritance module

Currently the koji_tag module is all-or-nothing when it comes to tag inheritance management. Either you can manage a tag's inheritance completely with Ansible, or else you cannot manage it at all with Ansible.

MBS will dynamically manage some of a tag's inheritance, so this conflicts with what koji-ansible tries to do.

To provide finer-grained tag inheritance management and avoid clobbering MBS, we can introduce a new koji_tag_inheritance Ansible module. This will work like:

koji_tag_inheritance:
  parent: my-parent-tag
  child: my-child-tag
  priority: 10

It will be preferable to set inheritance using the koji_tag module directly, but this advanced module will assist users writing playbooks that must work with MBS.

support user account management

This will help managing user permissions (eg. adding a new admin user).

Relevant koji sub-commands:

add-user                  Add a user
disable-user              Disable logins by a user
enable-user               Enable logins by a user

grant-permission          Grant a permission to a user
revoke-permission         Revoke a permission from a user
list-permissions          List user permissions

Account RPCs:

  • getUser
  • createUser
  • enableUser
  • disableUser

Permissions RPCs:

  • getUserPerms
  • grantPermission
  • revokePermission

use multicall for performance

As we add more features to koji_tag, we're making more and more XML-RPC calls, and this is slowing down the playbook operation.

We should be able to get all the information we need about the tag in one single "read" multicall, and then we can make all the changes we need in another single "write" multicall.

koji_btype should query the exact btype name in listBTypes

Instead of asking the hub for all configured btypes and filtering that list client-side, we can query the hub for the exact btype name that we're looking for, like so:

btypes = session.listBTypes(query={'name': 'image'})

Questions:

  1. When did this query feature land in a Koji release? Is it pretty safe to assume most hubs have this?
  2. What happens when there are no matches - does the hub always return an empty list?

various inline docs are not valid YAML

ansible-doc crashes when we install the Ansible Collection and then try to read the inline docs for koji_tag.

Steps to reproduce:

$ ansible-galaxy collection install ktdreyer-koji_ansible-0.0.0-git.230+9d7258a6.tar.gz
$ ansible-doc -t module ktdreyer.koji_ansible.koji_tag
ERROR! module ktdreyer.koji_ansible.koji_tag missing documentation (or could not parse documentation): while scanning a simple key
  in "<unicode string>", line 50, column 10
could not find expected ':'
  in "<unicode string>", line 51, column 10

This is just a yaml.scanner.ScannerError backtrace from PyYAML, not the full backtrace from ansible-doc. The crash is in ansible/parsing/plugin_docs.py , specifically the read_docstring() method.

validate all nested arguments instead of using type=raw

Ansible 2.6+ allows nested options in the argument specification we pass into the constructor with AnsibleModule(argument_spec=...). This would replace our use of raw types in argument_spec, and it would allow us to validate user input even earlier. It might even do these string -> None or string -> int conversions for us that we are doing within koji_tag.py's normalize_inheritance().

This feature is documented at https://docs.ansible.com/ansible/latest/dev_guide/developing_program_flow_modules.html#argument-spec

koji_tag_inheritance "state: absent" should not require "priority"

When I use state: absent in order to delete an inheritance relationship, I get an error:

- koji_tag_inheritance:
    parent_tag: my-parent-tag
    child_tag: my-child-tag
    state: absent
msg: 'missing required arguments: priority'

We should be able to delete this inheritance relationship without having to specify the priority integer.

generate-playbook: gracefully handle koji lib < 1.18

If we are using a Koji library older than 1.18, session.multicall is not callable. We get an error like so:

$ ./utils/generate-playbook  --profile foobar --regex myproduct-17.0
Traceback (most recent call last):
  File "./utils/generate-playbook", line 226, in <module>
    main()
  File "./utils/generate-playbook", line 221, in main
    generate_tags_playbook(session, args.regex)
  File "./utils/generate-playbook", line 199, in generate_tags_playbook
    tags = search(session, 'tag', 'getTag', regex)
  File "./utils/generate-playbook", line 187, in search
    with session.multicall(strict=True, batch=100) as m:
TypeError: 'bool' object is not callable

We should detect this early and bail if not callable(session.multicall)).

support blocking packages

Here is how to define a package set for a tag today:

  - name: Create a main product koji tag
    koji_tag:
      name: ceph-3.2-foo
      state: present
      packages:
        kdreyer:
          - ansible
          - ceph
          - ceph-ansible

This means I can only "whitelist" (koji add-pkg) builds. I need to be able to "blacklist" them (koji block-pkg).

Need to play around with whether it makes sense to build this into the packages parameter. Something like:

      packages:
        kdreyer:
          - ansible
          - ceph:
              blocked: true
          - ceph-ansible

(This would also make room for the extra_arches feature of packageListAdd.)

script to generate plays from live koji

To help with adoption, it would be really cool to have something that could "reverse" a live Koji tag into an Ansible play. This would let users see what the Ansible code would look like if they started using this project.

Removing build target crashes

The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2683, in _callMethod
    return self._sendCall(handler, headers, request)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2601, in _sendCall
    return self._sendOneCall(handler, headers, request)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2647, in _sendOneCall
    ret = self._read_xmlrpc_response(r)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2659, in _read_xmlrpc_response
    result = u.close()
  File "/usr/lib64/python3.8/xmlrpc/client.py", line 655, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1000: "invalid type for id lookup: <type 'dict'>">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/brain/.ansible/tmp/ansible-tmp-1585333758.351233-55142899833345/AnsiballZ_koji_target.py", line 102, in <module>
    _ansiballz_main()
  File "/home/brain/.ansible/tmp/ansible-tmp-1585333758.351233-55142899833345/AnsiballZ_koji_target.py", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/brain/.ansible/tmp/ansible-tmp-1585333758.351233-55142899833345/AnsiballZ_koji_target.py", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.ktdreyer.koji_ansible.plugins.modules.koji_target', init_globals=None, run_name='__main__', alter_sys=True)
  File "/usr/lib64/python3.8/runpy.py", line 206, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib64/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py", line 151, in <module>
  File "/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py", line 147, in main
  File "/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py", line 138, in run_module
  File "/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py", line 103, in delete_target
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2187, in __call__
    return self.__func(self.__name, args, opts)
  File "/usr/lib/python3.8/site-packages/koji/__init__.py", line 2702, in _callMethod
    raise err
__koji__gdc.GenericError: invalid type for id lookup: <type 'dict'>
fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/usr/lib/python3.8/site-packages/koji/__init__.py\", line 2683, in _callMethod\n    return self._sendCall(handler, headers, request)\n  File \"/usr/lib/python3.8/site-packages/koji/__init__.py\", line 2601, in _sendCall\n    return self._sendOneCall(handler, headers, request)\n  File \"/usr/lib/python3.8/site-packages/koji/__init__.py\", line 2647, in _sendOneCall\n    ret = self._read_xmlrpc_response(r)\n  File \"/usr/lib/python3.8/site-packages/koji/__init__.py\", line 2659, in _read_xmlrpc_response\n    result = u.close()\n  File \"/usr/lib64/python3.8/xmlrpc/client.py\", line 655, in close\n    raise Fault(**self._stack[0])\nxmlrpc.client.Fault: <Fault 1000: \"invalid type for id lookup: <type 'dict'>\">\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/brain/.ansible/tmp/ansible-tmp-1585333758.351233-55142899833345/AnsiballZ_koji_target.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/brain/.ansible/tmp/ansible-tmp-1585333758.351233-55142899833345/AnsiballZ_koji_target.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/brain/.ansible/tmp/ansible-tmp-1585333758.351233-55142899833345/AnsiballZ_koji_target.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.ktdreyer.koji_ansible.plugins.modules.koji_target', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.8/runpy.py\", line 206, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.8/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.8/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py\", line 151, in <module>\n  File \"/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py\", line 147, in main\n  File \"/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py\", line 138, in run_module\n  File \"/tmp/ansible_koji_target_payload_1k5h1khy/ansible_koji_target_payload.zip/ansible_collections/ktdreyer/koji_ansible/plugins/modules/koji_target.py\", line 103, in delete_target\n  File \"/usr/lib/python3.8/site-packages/koji/__init__.py\", line 2187, in __call__\n    return self.__func(self.__name, args, opts)\n  File \"/usr/lib/python3.8/site-packages/koji/__init__.py\", line 2702, in _callMethod\n    raise err\n__koji__gdc.GenericError: invalid type for id lookup: <type 'dict'>\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

I am not sure what's wrong here :/

I have it defined as:

- koji_target:
    koji: gdc
    name: openstack-rocky-el7
    build_tag: openstack-rocky-el7-build
    dest_tag: openstack-rocky-el7-testing
    state: absent

document default_client_keytab_name MIT Kerberos feature

When the Koji client makes an authenticated request with requests-gssapi, and the client has no Kerberos TGT in the cache, MIT Kerberos will attempt to authenticate with a keytab at
/var/kerberos/krb5/user/$EUID/client.keytab.

A blog post that explains more about this feature of Kerberos: https://adam.younglogic.com/2015/05/auto-kerberos-authn/

This means that if the remote host has a keytab at /var/kerberos/krb5/user/$EUID/client.keytab, then koji-ansible will use this keytab for authentication. Users do not need to run kinit in a playbook or set a keytab in a Koji profile. This really simplifies the use of koji-ansible with Kerberos.

We should document this (here and Koji upstream) to make it easier to automate authentication.

koji_tag: support "merge_mode" option for external repos

Koji 1.17 adds support for a new "merge_mode" setting on external repos.

From the release notes:

External repos now have a merge_mode option. Valid values are either koji (the old way) or simple (a new alternative). This option can be set with the --mode option to the add-external-repo or edit-external-repo commands.

implement check mode

We're advertising that we support check mode, but we actually make the changes. This is going to be surprising and disappointing to users when check-mode ends up changing live data.

RFE: options for a tag

koji edit-tag has

  -x key=value, --extra=key=value
                        Set tag extra option
  -r key, --remove-extra=key
                        Remove tag extra option

It would be great if ansible modules would support it. It is especially useful if you want to set things like mock.new_chroot=0 or rpm.macro.dist : '.el8+gdc'

support tagging builds

We need an idempotent way to tag builds during a playbook run. Something like:

  - name: Tag a build into the ceph-3.1-rhel-7-override tag
    koji_tag_build:
      koji: kojidev
      name: redhat-release-server-1.0.el7
      tag: ceph-3.1-rhel-7-override
      state: present

tests: centralize FakeKojiSession definition

Every unit test has its own FakeKojiSession class, and its own "session" pytest fixture method. Each unit test file's FakeKojiSession class implements just enough of the Koji Hub API to pass the tests, but there is quite a bit of copy-and-paste here.

I think we've reached the point where we could centralize this into tests/conftest.py. We could define a single FakeKojiSession class and implement all the Koji Hub logic that we need into that class. A single "session" pytest fixture method could just return an instance of this class, and then all the unit test files can use this central fixture.

This would make our FakeKojiSession more complex, but we're at the point where there is considerable complexity already when we have to re-implement so much of the Hub's logic anyway. I think it would make it easier to get to full test coverage if we had a single large fixture. (I also wonder if we could extract this fixture into a completely standalone pytest plugin, for other Python projects that might want to fake calls to Koji).

add support for regenerating repositories

When setting up an entirely new build tag and target, it's a good idea to generate the repositories for the first time with the koji regen-repo command.

I'm thinking of a new koji_repo module, like:

  handlers:
    - name: regenerate ktdreyer-el7-build
      koji_repo:
        name: ktdreyer-el7-build
        state: regenerated

We could make this module return the values from getRepo(name) and the task_id as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.