Job stdout blank after reload about awx

[fosterseth]

awx-manage run_callback_receiver

you don't need to run that (in fact better not to)

are you running playbooks with a different ansible strategy other than the default (i.e. are you using "free")

as for now, for really large stdout, you may just need to download the stdout and examine that way

[Tfinn92]

Hey fosterseth, no, nothing out of the ordinary in the plays. We use one playbook that calls on about 6 different roles, but that's been the same for about a year. Not using free or anything to that nature.

Good to know about the awx-manage run_callback_receiver, I had seen it on a rather old post and figured I'd give it a whirl.

Downloading the output definitely works however. To put it in perspective, those files are roughly 500 KB, so not huge by any means, just a lot of lines.

[Tfinn92]

Just to add a little more to this. I spun up a brand new cluster with Rancher rke2 running version v1.26.13 and added a kubelet arg container-log-max-size=2560Mi just to let the logs stretch their legs. Spun up AWX using the latest Operator 2.12.1 and Helm. These are my values:

AWX:
  enabled: true
  name: awx
  namespace: awx
  spec:
    ## User creation
    admin_user: admin
    admin_email: email@email

    ## Replica Settings
    web_replicas: 3
    task_replicas: 3

    ## Networking and Ingress
    ingress_type: ingress
    ingress_tls_secret: awx-secret-tls
    hostname: awx-dev.domain
    ingress_path: /
    ingress_path_type: Prefix
    ipv6_disabled: true
    bundle_cacert_secret: cacert-custom-certs

    ## Upgrade Strategy
    auto_upgrade: true

    ## Resource Limitation
    web_resource_requirements:
      requests:
        cpu: 500m
        memory: 4Gi
      limits:
        cpu: 2000m
        memory: 8Gi
    task_resource_requirements:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 4000m
        memory: 4Gi
    ee_resource_requirements:
      requests:
        cpu: 500m
        memory: 200Mi
      limits:
        cpu: 1000m
        memory: 4Gi

    ### Environment Variable configuration
    task_extra_env: |
      - name: LDAPTLS_REQCERT
        value: never

    web_extra_env: |
      - name: LDAPTLS_REQCERT
        value: never

    ee_extra_env: |
      - name: LDAPTLS_REQCERT
        value: never
      - name: RECEPTOR_KUBE_SUPPORT_RECONNECT
        value: disabled

    ## Project Persistence
    projects_persistence: true
    projects_storage_class: longhorn
    projects_storage_size: 30Gi

Run the same job against the same inventory (about 40 hosts) and I am seeing the exact same behavior
Going through the job in the API I see the following:

GET /api/v2/jobs/6/job_events/
HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept
X-API-Node: awx-web-677496f994-smb2m
X-API-Product-Name: AWX
X-API-Product-Version: 23.8.1
X-API-Time: 0.080s
X-UI-Max-Events: 4000

{
    "count": 26183,

It definitely is a large job with over 26,000 job events. Downloading the output from the gui works no problem, so that work around definitely works for now. I figured maybe adding this context might help! Please let me know if there is anything else I can provide.

Edit: One other key note is GET /api/v2/jobs/6/stdout/ loads the stdout as expected in the API. So it seems like that transition to the gui isn't happening. Thanks again

[Tfinn92]

Performed a clean install of 23.9.0 using the operator and helm, still seeing the same behavior.