Observability only working with infra nodes about operations

[computate]

@larsks I have opened a RH Support Case for ACM 2.5.5 and Observability: https://access.redhat.com/support/cases/#/case/03391625

[computate]

It looks like in the CoreOS Logging Slack Channel, the next release of Red Hat OpenShift Logging with the Retention feature ETA is Jan 10th to release right after OpenShift 4.12.

[larsks]

@computate that section of the documentation seems to be about installing the observability stack (the example they give is for installing it on "infrastructure nodes"), and doesn't seem to have anything to do with the nodes from which it will collect data.

[computate]

Thanks for clarifying @larsks . I am reading further into the docs which state:

Note: When observability is enabled or disabled on OpenShift Container Platform managed clusters that are managed by Red Hat Advanced Cluster Management, the observability endpoint operator updates the cluster-monitoring-config ConfigMap by adding additional alertmanager configuration that restarts the local Prometheus automatically.

It looks like we are manually generating the cluster-monitoring-config for the prod cluster, which may be why I can't see observability data for the prod cluster.

configMapGenerator:
  - name: cluster-monitoring-config
    namespace: openshift-monitoring
    files:
      - config.yaml=configmaps/cluster-monitoring-config.yaml

@larsks Do you know why we are manually creating the cluster-monitoring-config ConfigMap?

[naved001]

@computate looks like I did that for persistent storage for prometheus and alertmanager.

https://github.com/OCP-on-NERC/nerc-ocp-config/pull/158/files

If there's some other way of configuring that storage, then we could remove that configmap.

[computate]

I have been working on removing Observability by disabling autosync on prod and infra clusters, deleting the MultiClusterObservability component and then recreating it. The multicluster-observability-operator is not recreating the deployments and pods for observability, so something else seems to be going wrong in ACM. I checked all the pod logs and events and see something failing in the multicluster-operators-standalone-subscription pod of :

E1212 15:28:42.757806 1 helmrepoutils.go:534] return code: 404 unable to retrieve chart - url: http://multiclusterhub-repo.open-cluster-management.svc.cluster.local:3000/charts/policyreport-2.5.3.tgz
E1212 15:28:42.757827 1 helmreleasemgr.go:99] failed to download chart from helm repo. - url: http://multiclusterhub-repo.open-cluster-management.svc.cluster.local:3000/charts/policyreport-2.5.3.tgz error: return code: 404 unable to retrieve chart - Failed to download the chart
E1212 15:28:42.757836 1 helmreleasemgr.go:42] failed to download chart from helm repo. - url: http://multiclusterhub-repo.open-cluster-management.svc.cluster.local:3000/charts/policyreport-2.5.3.tgz error: return code: 404 unable to retrieve chart - Failed to download the chart
E1212 15:28:42.757787 1 helmreleasemgr.go:42] failed to download chart from helm repo. - url: http://multiclusterhub-repo.open-cluster-management.svc.cluster.local:3000/charts/management-ingress-2.5.3.tgz error: return code: 404 unable to retrieve chart - Failed to download the chart

[computate]

ACM is not upgrading automatically. @larsks Looking at the ACM subscription it failed with the following error:

error validating existing CRs against new CRD's schema for "multiclusterobservabilities.observability.open-cluster-management.io": error listing resources in GroupVersionResource schema.GroupVersionResource{Group:"observability.open-cluster-management.io", Version:"v1beta1", Resource:"multiclusterobservabilities"}: conversion webhook for observability.open-cluster-management.io/v1beta2, Kind=MultiClusterObservability failed: Post "https://multicluster-observability-webhook-service.open-cluster-management.svc:443/convert?timeout=30s": no endpoints available for service "multicluster-observability-webhook-service"

[larsks]

@computate it sounds like this has been resolved. Can we close the issue?

[computate]

@larsks I could just use a review on this PR to upgrade the ACM Operator and also the LokiOperator for the latest support for logging and metrics. OCP-on-NERC/nerc-ocp-config#176