Cannot use Managed Identity about azure-resourcemanager-exporter

[mblaschke]

23.6.0 released with workload identity support

[mblaschke]

working on a new release

[mblaschke]

please try webdevops/azure-resourcemanager-exporter:main

[mickeder]

Thanks a lot @mblaschke, it works like a charm. Patiently waiting for the official release.

One thing worth noting is that with Managed Identity on AKS cluster, the first scrape could fail (all subsequent scrapes are fine) with below error log (IMDS token request timed out):

2022-09-28T10:46:59.165161333Z {"file":"/go/src/github.com/webdevops/azure-resourcemanager-exporter/main.go:65","func":"main.main","level":"info","msg":"starting http server on 0.0.0.0:8080"}

2022-09-28T10:46:59.165165633Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:172","func":"collector.(*Collector).collectionStart","level":"info","msg":"starting metrics collection"}
2022-09-28T10:47:00.156569240Z {"collector":"GraphApps","file":"/go/src/github.com/webdevops/azure-resourcemanager-exporter/metrics_graph_apps.go:69","func":"main.(*MetricsCollectorGraphApps).Collect","level":"panic","msg":"DefaultAzureCredential: failed to acquire a token.\nAttempted credentials:\n\tEnvironmentCredential: missing environment variable AZURE_TENANT_ID\n\tManagedIdentityCredential: IMDS token request timed out\n\tAzureCLICredential: fork/exec /bin/sh: no such file or directory"}
2022-09-28T10:47:00.156619941Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:130","func":"collector.(*Collector).collect.func1.1","level":"error","msg":"panic occurred (panic threshold 1 of 5): DefaultAzureCredential: failed to acquire a token.\nAttempted credentials:\n\tEnvironmentCredential: missing environment variable AZURE_TENANT_ID\n\tManagedIdentityCredential: IMDS token request timed out\n\tAzureCLICredential: fork/exec /bin/sh: no such file or directory"}
2022-09-28T10:47:00.156635241Z {"collector":"GraphApps","duration":1.000991064,"file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:182","func":"collector.(*Collector).collectionFinish","level":"info","msg":"finished metrics collection"}

2022-09-28T10:49:00.157199560Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:172","func":"collector.(*Collector).collectionStart","level":"info","msg":"starting metrics collection"}
2022-09-28T10:49:00.511749569Z {"collector":"GraphApps","duration":0.35457381,"file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:182","func":"collector.(*Collector).collectionFinish","level":"info","msg":"finished metrics collection"}

2022-09-28T10:51:00.512476397Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:172","func":"collector.(*Collector).collectionStart","level":"info","msg":"starting metrics collection"}
2022-09-28T10:51:00.656298987Z {"collector":"GraphApps","duration":0.144378199,"file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:182","func":"collector.(*Collector).collectionFinish","level":"info","msg":"finished metrics collection"}

It can be avoided by adding to the pod an initContainer with a delay (e.g. 5 seconds).
(see Azure/aad-pod-identity#992)

2022-09-28T10:52:40.455113835Z {"file":"/go/src/github.com/webdevops/azure-resourcemanager-exporter/main.go:65","func":"main.main","level":"info","msg":"starting http server on 0.0.0.0:8080"}

2022-09-28T10:52:40.455285945Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:172","func":"collector.(*Collector).collectionStart","level":"info","msg":"starting metrics collection"}
2022-09-28T10:52:40.669964387Z {"collector":"GraphApps","duration":0.214674741,"file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:182","func":"collector.(*Collector).collectionFinish","level":"info","msg":"finished metrics collection"}

2022-09-28T10:54:40.670674096Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:172","func":"collector.(*Collector).collectionStart","level":"info","msg":"starting metrics collection"}
2022-09-28T10:54:40.802757687Z {"collector":"GraphApps","duration":0.13204079,"file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:182","func":"collector.(*Collector).collectionFinish","level":"info","msg":"finished metrics collection"}

2022-09-28T10:56:40.803182200Z {"collector":"GraphApps","file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:172","func":"collector.(*Collector).collectionStart","level":"info","msg":"starting metrics collection"}
2022-09-28T10:56:40.936933704Z {"collector":"GraphApps","duration":0.134056515,"file":"/go/pkg/mod/github.com/webdevops/[email protected]/prometheus/collector/collector.go:182","func":"collector.(*Collector).collectionFinish","level":"info","msg":"finished metrics collection"}

[mblaschke]

are you using aad-pod-identity or azure workload identity?

[mickeder]

aad-pod-identity 1.8.5 deployed using helm chart 4.1.6

[megakid]

What configuration did you use here? I've successfully configured azure workload identity for other pods but I cannot get this one working.

EDIT: got it working.

For the sake of others, here's my values file for the helm chart that made this work (you have to have setup your workload identity etc first) - this is for use in by terraform templatefile function.

image:
  # See https://github.com/webdevops/azure-resourcemanager-exporter/issues/18
  tag: main

secrets:
  AZURE_SUBSCRIPTION_ID: "${join(" ", subscription_ids)}"

serviceAccount:
  name: "${service_account_name}"
  labels:
    azure.workload.identity/use: "true"
  annotations:
    azure.workload.identity/client-id: "${client_id}"
    azure.workload.identity/tenant-id: "${tenant_id}"
    azure.workload.identity/service-account-token-expiration: "86400"

podLabels:
  azure.workload.identity/use: "true"

[mblaschke]

@megakid
tag :23.3.0-beta1 should give you workload identity support but keep in mind that this part of the azure-sdk-for-go is still in beta and they won't release it until AKS workload identity support is GA.