[BUG?] ERROR: work type did not expect a signature when running health check in AWX with work-kubernetes in Receptor about awx

[kurokobo]

@diademiemi
Hi,

Our goal here is to run Receptor in a Kubernetes cluster so we can host execution and/or hop nodes in Kubernetes.

The current AWX implementation assumes that the execution nodes are running as the hosts where Ansible Runner is running locally and Podman is installed.
So in the first place it's hard to run execution nodes in Kubernetes cluster since if we select execution nodes for some job templates AWX sends request to ansible runner on the execution nodes to run execition environment by creating container on the Podman, instead of Kubernetes.

Alternatively, I recommend you this to achieve similar goals; we can define Container Group with credentials for the remote Kubernetes cluster. This allows us to run EE on remote Kubernetes cluster: https://ansible.readthedocs.io/projects/awx/en/latest/administration/containers_instance_groups.html#create-a-container-group

Running hop node on Kubernetes cluster is not so hard, since hop node never be used to invoke any commands. No podman nor ansible runner are required. In addition, the feature "in-cluster hop node" called AWXMeshIngress will be implemented in the next release: #14640

Here are my answer for your questions for your technical interest:

• worktype is just a name. The documentation uses kubeit not because it is required for Kubernetes work, but simply as one example, given a simple name.
• AWX sends ansible-runner worktype to run health check. This will invoke ansible-runner worker --worker-info on the execution nodes.
• Running jobs on Instance Groups means that AWX requests to remote Ansible Runner to run playbooks with process isolation by podman. Ansible Runner has an ability to run ansible-playbook in isolated environment (means running it in Podman container), so in this case EE container is created by Ansible Runner.
• Running jobs on Container Groups* means that AWX requests to remote Ansible Runner to run playbook locally. Receptor has an ability to create Pod with custom specification on Kubernetes cluster, so in this case EE container that run Ansible Runner is created by Receptor (kubernetes-runtime-auth worktype or kubernetes-incluster-auth worktype).

If you have further insterest, my blog article may helps you (sorry it is in Japanese, so please use some translator): https://blog.kurokobo.com/archives/4847
Or ask further questions on the forum: https://forum.ansible.com/

[kurokobo]

It would be appropriate to improve the error message, perhaps in an enhancement request on the Receptor side.

[fosterseth]

as @kurokobo mentioned, container groups are designed to achieve running jobs on remote k8s clusters

AWX expects execution node to have a work-command called ansible-runner for health checks

but when running jobs, AWX also uses this same work command. So even if you have a proper kubeit work-kubernetes setup in the config, AWX is not going to utilize it sadly. That would require a bit of changes in AWX to get that working.

Is there a use case for this that container groups doesn't cover?

[diademiemi]

Thank you for the detailed response! I understand a lot better now what this is doing under the hood

I'll be checking out the AWXMeshIngress and Container Groups feature today and tomorrow and I'll get back to you for if this covers our usecase.