CrashLoopBackOff in Teleport cluster after upgrade to Helm chart v15.3.6 on GCP about teleport

[zmb3]

Duplicate of #41954

(I know this was opened first, but that one has more details)

[cizara]

no, I made a rollback. Will create a new issue as there is no response here

[cizara]

Hi @zmb3,. sorry to bump this closed issue, but I just upgrade Teleport from 15.3.5 to 15.3.7 and now I'm getting:

Defaulted container "teleport" out of: teleport, wait-auth-update (init)
{"caller":"automaticupgrades/channel.go:65","component":null,"level":"debug","message":"'default' automatic update channel not specified, teleport will serve its version by default.","timestamp":"2024-05-30T09:03:23Z"}
{"timestamp":"2024-05-30T09:03:23Z","level":"info","caller":"common/teleport.go:661","message":"Starting Teleport with a config file","version":"15.3.7","config_file":"/etc/teleport/teleport.yaml"}
{"caller":"lite/lite.go:259","component":"sqlite","level":"debug","message":"Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000\u0026_sync=FULL\u0026_txlock=immediate, poll stream period: 1s","timestamp":"2024-05-30T09:03:23Z"}
{"caller":"lite/lite.go:310","component":"sqlite","level":"debug","message":"journal_mode=delete, synchronous=2, busy_timeout=10000","timestamp":"2024-05-30T09:03:23Z"}
SDK 2024/05/30 09:03:23 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 404, request to EC2 IMDS failed

ERROR REPORT:
Original Error: *url.Error Get "https://compute.googleapis.com/compute/v1/projects/my-project/zones/us-east4-a/instances/gke-cluster-primary-7706f694-cdg9": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:
	https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to

`
Stack Trace:
	github.com/gravitational/teleport/lib/cloud/gcp/vm.go:332 github.com/gravitational/teleport/lib/cloud/gcp.(*instancesClient).GetInstance
	github.com/gravitational/teleport/lib/cloud/imds/gcp/imds.go:117 github.com/gravitational/teleport/lib/cloud/imds/gcp.(*InstanceMetadataClient).GetTags
	github.com/gravitational/teleport/lib/cloud/imds/gcp/imds.go:152 github.com/gravitational/teleport/lib/cloud/imds/gcp.(*InstanceMetadataClient).GetHostname
	github.com/gravitational/teleport/lib/service/service.go:940 github.com/gravitational/teleport/lib/service.NewTeleport
	github.com/gravitational/teleport/lib/service/service.go:703 github.com/gravitational/teleport/lib/service.newTeleportProcess
	github.com/gravitational/teleport/lib/service/service.go:719 github.com/gravitational/teleport/lib/service.Run
	github.com/gravitational/teleport/tool/teleport/common/teleport.go:663 github.com/gravitational/teleport/tool/teleport/common.OnStart
	github.com/gravitational/teleport/tool/teleport/common/teleport.go:560 github.com/gravitational/teleport/tool/teleport/common.Run
	github.com/gravitational/teleport/tool/teleport/main.go:33 main.main
	runtime/proc.go:267 runtime.main
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: initialization failed
	Get "https://compute.googleapis.com/compute/v1/projects/my-project/zones/us-east4-a/instances/gke-cluster-primary-7706f694-cdg9": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).
This error could be caused by a missing IAM policy binding on the target IAM service account.
For more information, refer to the Workload Identity documentation:
	https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to

Everything works ok on 15.3.5. Let me know if you want me to create a new bug for this

[mlachevre-pass]

Same issue here. @cizara have you find a workaround ?

[cizara]

created bug #42312