Missing mas-install pipeline with latest CLI v10.0.5 about cli

[rene-oromtz]

I just retried on the cluster were I faced this issue (I first had to clean all MAS references) and was unable to recreate this issue, I even retried with CLI v10.0.5:

2024-07-05 23:43:29,594   DEBUG    pipeline.tekton.dev/mas-db2 created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-install created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-ivt created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-rollback created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-uninstall created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-update created
2024-07-05 23:43:29,595   DEBUG    pipeline.tekton.dev/mas-upgrade created
2024-07-05 23:43:29,818   DEBUG    ---

Seems weird that it failed on the first time but agree if you want to keep this issue open for tracking enhancement to error handling.
Thanks for taking a look.

[durera]

Can you check in the mas.log file, I would assume there is an error in there that was generated when the pipelines were being installed, that we should have handled better and failed the "install tekton definitions" step here, I've not been able to reproduce this; everytime I run the install all the pipeline definitions are successfully installed.

[durera]

The mas.log should contain something like this:

2024-07-05 21:06:24,489   DEBUG    secret/pipeline-sls-entitlement created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-appconnect created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-arcgis created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cert-manager created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-common-services created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cos-deprovision created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cos created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cp4d-service-update created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cp4d-service created
2024-07-05 21:07:42,158   DEBUG    task.tekton.dev/mas-devops-cp4d created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-db2 created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-eck created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-gencfg-workspace created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-grafana created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ibm-catalogs created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-kafka-deprovision created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-kafka created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-launchivt created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-mongodb created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-must-gather created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-nvidia-gpu created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocp-deprovision created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocp-verify-all created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocp-verify created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-ocs created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/sls-registry-update created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-sls created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-config created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-install created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-rollback created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-uninstall created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-upgrade created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-app-verify created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-certs created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-config created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-db2-setup-for-manage created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-dns created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-install created
2024-07-05 21:07:42,159   DEBUG    task.tekton.dev/mas-devops-suite-rollback created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-suite-uninstall created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-suite-upgrade created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-suite-verify created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-turbonomic created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-uds created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-update-configmap created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-wait-for-configmap created
2024-07-05 21:07:42,160   DEBUG    task.tekton.dev/mas-devops-wait-for-tekton created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-db2 created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-install created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-ivt created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-rollback created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-uninstall created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-update created
2024-07-05 21:07:42,160   DEBUG    pipeline.tekton.dev/mas-upgrade created

And I would hope that there would be a reason logged in there why only some of the definitions were installed.

[rene-oromtz]

Unfortunately I think I terminated the pod which had containing this mas.log.

I tried to recreate the issue on a different environment but without success, in this new environment, the pipelines where installed succesfully:

...
2024-07-05 21:55:50,660   DEBUG    pipeline.tekton.dev/mas-db2 created
2024-07-05 21:55:50,660   DEBUG    pipeline.tekton.dev/mas-install created
2024-07-05 21:55:50,660   DEBUG    pipeline.tekton.dev/mas-ivt created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-rollback created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-uninstall created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-update created
2024-07-05 21:55:50,661   DEBUG    pipeline.tekton.dev/mas-upgrade created
2024-07-05 21:55:50,896   DEBUG    ---
apiVersion: tekton.dev/v1beta1

Only difference between one and the other is that the successful was a VPC on IBM Cloud and the failed cluster was behind a firewall.
I'll try to see if I can relaunch on the other environment and validate if this is reproducible

[durera]

Thanks, you should get this error message logged:

    except KubectlCommandError as e:
        logger.warning(f"Error: Unable to install/update Tekton definitions: {e}")
        return False

but I can see that the error handling here is poor, so we will use this issue to track improving that ...

            with Halo(text=f'Installing latest Tekton definitions (v{self.version})', spinner=self.spinner) as h:
                updateTektonDefinitions(pipelinesNamespace, self.tektonDefsPath)
                h.stop_and_persist(symbol=self.successIcon, text=f"Latest Tekton definitions are installed (v{self.version})")

We're not event checking whether the Tekton definitions are successfully applied, I think it's because of a disconnect between the expectation that this function would throw an exception, but instead it catches the exception itself and just returns False.

We'll get that fixed next week so that, in case where this fails for whatever reason, it will fail earlier and prompt the user to check mas.log for the failure reason.

[durera]

This should be addressed by: ibm-mas/python-devops@7294f17 and #1105