GKE - MySQL CloudSQL Conectivity Issue: ACCESS_TOKEN_SCOPE_INSUFFICIENT about cloud-sql-proxy

[enocom]

Looks good to me. I'd suggest using a non-default service account. Nonetheless, for the sake of completeness, you can use the default service account and then under Node pools -> Security (for standard clusters), enable the Cloud SQL Admin API access:

[anz000]

I was using gcloud. Adding the scope --scopes=https://www.googleapis.com/auth/cloud-platform when creating the cluster fixes the issue.

I appreciate your help.

[enocom]

Two things:

1. You're using the Cloud SQL Java Connector and the Cloud SQL Proxy. In fact, only one is necessary. Either use the Proxy with a standard database driver, or drop the Proxy and use the Cloud SQL Java Connector.
2. Instead of using a credentials file, I recommend using Workload Identity. This will simplify the connection path. For using a credentials file, though, see the Java Connector's docs. You'll also need to ensure your VMs have the Cloud SQL Admin API OAuth2 scope, but only if your GKE VMs are using the default Compute Service Account.

[anz000]

Yeah, I was trying both approach and failed with both. I feel like I'm missing one or two things with either approach. if using Workload Identity is preferred, I can try to use it.

When I build the cluster, if I assign a service account (from AppEngine - that works with access to Cloud SQL on AppEngine instances), then it works. When I don't assign any service account, it falls back to the default compute engine service account, and I run into the issue. Both service account have the same roles, but the compute-engine SA has some weird IAM issues.

Update application.yml to use 127.0.0.1 for the database:

spring:
  datasource:
    url: jdbc:mysql://127.0.0.1:3306/db?serverTimezone=UTC
    username: my-db-username
    password: my-db-password

..

Here's the updated deployment.yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-gcloud-services
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-gcloud-services
  template:
    metadata:
      labels:
        app: my-gcloud-services
    spec:
      serviceAccountName: ksa-cloud-sql #my-gcloud-gke-service-account
      containers:
      - name: my-gcloud-services
        image: gcr.io/my-gcloud-project/my-gcloud-services:gke1
        ports:
        - containerPort: 8080  # Your application port
        env:
        - name: PORT
          value: "8080"
        - name: INSTANCE_CONNECTION_NAME
          value: my-gcloud-project:us-central1:my-gcloud-project-v1
        - name: DB_HOST
          value: "127.0.0.1"
        - name: DB_PORT
          value: "3306"
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: sql-credentials
              key: username
        - name: DB_PASS
          valueFrom:
            secretKeyRef:
              name: sql-credentials
              key: password
        - name: DB_NAME
          valueFrom:
            secretKeyRef:
              name: sql-credentials
              key: database
      - name: cloud-sql-proxy
        image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.8.0
        args:
          - "--structured-logs"
          - "--port=3306"
          - "my-gcloud-project:us-central1:my-gcloud-project-v1"
        securityContext:
          runAsNonRoot: true

[enocom]

Glad to hear it.