Support credentials to load model from GCS/S3 storage about kserve

[ellistarn]

How do you feel about annotations?

[yuzisun]

@ellis-bigelow any further thoughts on this? By checking this page, looks a bunch of envs are needed
https://github.com/kubeflow/examples/tree/master/mnist#using-s3, not sure how to use annotation.

[ellistarn]

Here's the flow I had in mind:

1. User must have pre-created secret
2. User provides an annotation referencing the secret: "serving.kubeflow.org/kfservice-s3-model-secret: secretRef"
3. In our controller, if the user provides that annotation, we mount those to the kfconfigurations

I only worry a little bit that our default + canary model is a little weird. What happens if a user wants to canary a new secret -- they'll take an outage. Perhaps we can qualify the annotations?

We may want to not specify s3 and just mount the secret to all relevant locations for gcs/s3/blob etc

Alternatively, we could use a k8s service account and mount them in the right place in the pod for s3/gcs/blob. What do you think?

[yuzisun]

With our model once secretRef annotation is changed both default and canary get updated, so user will take an outage for a bad secret rollout. Maybe we can support envs on default/canary spec and pull secrets as envs which works for s3 but not sure for other storage.

We may want to not specify s3 and just mount the secret to all relevant locations for gcs/s3/blob etc
Alternatively, we could use a k8s service account and mount them in the right place in the pod for s3/gcs/blob. What do you think?

Can you elaborate these more not sure if I understand this.

[yuzisun]

/assign @yuzisun

[yuzisun]

Discussed with @ellis-bigelow as a first pass we can add service account to the spec to which user adds the secrets for gcs/s3 or other identities.