szlabs / harbor-automation-4k8s Goto Github PK

a webhook:

image pulling path of the deploying workloads in the specified namespace can be rewritten to the harbor project linked with the k8s namespace

for example:

k8s namespace: my-space
annotations: goharbor.io/project: mypro
goharbor.io/secret-issuer: myharbor

The current reconciliation logic is confusing and might cause confusion. Among the 4 possible combinations of robot and project annotation existence, which are:

1. robot not exist, project not exist
2. robot not exist, project exist
3. robot exist, project not exist
4. robot exist, project exist

Case 2, 3 are invalid. For case 1, we need to automatically create harbor project and robot account, and for case 4, we need check whether these information are valid in harbor. It's overly complicated for PSB as it should be solely focusing on secret binding.

Proposing to move the annotation validation logic to namespace controller and making robot and project required field for PSB object. Namespace controller should validate or create corresponding resources, and create PSB with validated configurations. For PSB, it should always assume that the project and robot in its spec exists in harbor.

For a robot account, from a security standpoint, we should set an expiration for that account or at least rotate its credentials. As currently, we can't rotate robots' credentials/tokens in harbor itself, we should set an expiration date in the operator when creating the robot account as a future improvement. And by the nature of controller reconciliation logic, a new robot account should be created after the old one expires.

e.g: HarborServerConfiguration

Context

  Warning  FailedCreate      3s (x12 over 15s)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "mimg.kb.io": Post "https://harbor-automation-4k8s-webhook-service.harbor-day2-op.svc:443/mutate-image-path?timeout=30s": dial tcp 100.68.145.242:443: connect: connection refused

This is because once the mutationWebhookConfiguration is deployed, our api server will check all pods creation, including our controller pod. API server will call our webhook server. But since our controller pod contains the webhooks server, so it needs to come up first before apiserver could trigger webhook. This will becomes a deadlock if we deploy our controller after the mutationWebhookConfiguration.

Related issue kubernetes-sigs/kustomize#821

Reproduce steps

make deploy

Possible solution

1. Add labels our namespace, so webhook controller could skip namespaces with those labels
   Drawback: k8s related or other system level namespaces needs labels as well, there could be potential issue
2. Webhook controller only checks namespaces with specific labels
   Drawback: we will need to modify namespaces labels, except the annotations on them.

If customer use self-signed certificate, the container runtime of K8s cluster needs to trust the certifcate.
In the future, k8s will be using containerd as primary runtime.
But currently containerd doesn't support hosts-dir folder as docker to inject cert in runtime.
Once this PR is in containerd/containerd#4978 new containerd and used by K8s, we should be able to inject cert without restarting containerd

• auto-create
• binding existing harbor project (robot account is required)

leverage namespace annotations

implement a reconcile to watched the annotated namespace

The harbor server configuration CRD should be extended to support a global default use-case, that if present, applies annotations to namespaces missing the harbor operator annotations.

I think the most straightforward implementation would be to add an additional field on the CRD, default bool. When a harbor server configuration CRD default is set to true, the operator will treat existing and new namespaces lacking the operator annotations as if they were annotated to use the default CRD. Submissions of multiple CRDs with the default: true in the spec will be rejected by the operator, to ensure only 1 CRD exists with default: true.

Migrate work delivered here: https://github.com/indeedeng-alpha/harbor-container-webhook to this project

bind the harbor robot account (k8s secret format) to the specified service account under the k8s namespace.

• CR needs to be defined
• a reconcile ctrl should be implemented