kops v1.26 upgrade fails due to serviceAccountIssuer changes about kops

[hakman]

@elliotdobson You definitely need to specify serviceAccountIssuer and serviceAccountJWKSURI to the old value.
I would add that you may want to set additionalSANs (or additionalSans) to master.cluster.domain.
My suggestion would be to try this on a new small test cluster.

[elliotdobson]

Thanks for your response @hakman.

We decided to move forward with the change to serviceAccountIssuer & serviceAccountJWKSURI rather than fight it by specifying our existing value.

Based on this comment we were able to come up with the following procedure that allowed us to migrate the Service Account Issuer (SAI) non-disruptively on two clusters:

1. Add additionalSans with masterInternalName value to the kops cluster spec
2. Remove masterInternalName from the kops cluster spec
3. Add the modify-kube-api-manifest (existing SAI as primary) hook to the master instancegroups

  hooks:
  - name: modify-kube-api-manifest
    before:
      - kubelet.service
    manifest: |
      User=root
      Type=oneshot
      ExecStart=/bin/bash -c "until [ -f /etc/kubernetes/manifests/kube-apiserver.manifest ];do sleep 5;done;sed -i '/- --service-account-issuer=https:\/\/api.internal.cluster.domain/i\ \ \ \ - --service-account-issuer=https:\/\/master.cluster.domain' /etc/kubernetes/manifests/kube-apiserver.manifest"

4. Apply the changes to the cluster
5. Roll the masters
6. Manually create the master.cluster.domain DNS record pointing at the masters IP addresses
7. (If the kubernetes-services-endpoint configMap exists (& using calico CNI) then) Update the KUBERNETES_SERVICE_HOST value to api.internal.cluster.domain in the kubernetes-services-endpoint configMap (kube-system namespace) & Roll the calico daemonset
8. Update the modify-kube-api-manifest (switch the primary/secondary SAI) hook on the master instancegroups

  hooks:
  - name: modify-kube-api-manifest
    before:
      - kubelet.service
    manifest: |
      User=root
      Type=oneshot
      ExecStart=/bin/bash -c "until [ -f /etc/kubernetes/manifests/kube-apiserver.manifest ];do sleep 5;done;sed -i '/- --service-account-issuer=https:\/\/api.internal.cluster.domain/a\ \ \ \ - --service-account-issuer=https:\/\/master.cluster.domain' /etc/kubernetes/manifests/kube-apiserver.manifest"

9. Apply the changes to the cluster
10. Roll the masters (remember to update the master.cluster.domain DNS record with new IP addresses of the masters)
11. Roll all the nodes in the cluster
12. Wait 24 hours until the dynamic SA tokens have refreshed
13. Remove the modify-kube-api-manifest hook on the master instancegroups
14. Remove additionalSans from the kops cluster spec
15. Apply the changes to the cluster
16. Roll the masters
17. Delete the master.cluster.domain DNS record

Care needs to be taken around the master.cluster.domain DNS record as without it unexpected things can occur (kubelet unable to contact API, etc) until the migration is complete.

I think the same procedure could be used to enable IRSA non-disruptively (which we will try in the future).

Do you think it is worth adding this to the kops documentation and/or updating kops cluster spec to allow multiple serviceAccountIssuer to be specified?

[hakman]

Thank you for the detailed guide @elliotdobson. This would be useful to be added to the kOps docs.
Regarding multiple service account issuers, see #16497. Would that be what you were looking for?

[elliotdobson]

Where would a procedure like that live in the kOps docs? Under Operations perhaps?

Yes #16497 is the addition to the cluster spec that would've been required for this migration. Will that feature will be back-ported to previous kOps releases?

[hakman]

Where would a procedure like that live in the kOps docs? Under Operations perhaps?

Yes, under Operations.

Yes #16497 is the addition to the cluster spec that would've been required for this migration. Will that feature will be back-ported to previous kOps releases?

Seems pretty simple to back-port. Probably it will be in kOps 1.28+.

[elliotdobson]

Closing this issue as we've worked around the issue and documented the workaround.