Private docker registries support for polling about keel

[rusenask]

New image with alpha tag is available.

Kubernetes provider - creating a secret

kubectl create secret docker-registry myregistrykey --docker-server=https://index.docker.io/v1/ --docker-username=user --docker-password=pass --docker-email=email

and then using it in a k8s documented way such as

imagePullSecrets:
    - name: myregistrykey

Is enough, Keel will use the same secret your pods are using to access docker registry, so this integration is transparent and doesn't require user to know anything.

Helm provider:

Here user is required to do two things:

1. First is quite obvious, define in your chart secret (repository.imagePullSecret) for your deployment template
2. Provide that secret path for Keel in keels config (imagePullSecret: image.imagePullSecret)

replicaCount: 1
image:
  repository: karolisr/webhook-demo
  tag: "0.0.8"
  pullPolicy: IfNotPresent
  imagePullSecret: myregistrykey
service:
  name: webhookdemo
  type: ClusterIP
  externalPort: 8090
  internalPort: 8090

keel:
  # keel policy (all/major/minor/patch/force)
  policy: all
  # trigger type, defaults to events such as pubsub, webhooks
  trigger: poll
  # polling schedule
  pollSchedule: "@every 3m"
  # images to track and update
  images:
    - repository: image.repository
      tag: image.tag
      imagePullSecret: image.imagePullSecret

Does this make sense?

[bonifaido]

I withdraw my last comment, imagePullSecrets is becoming the part of the Pod descriptor even when using the ServiceAccount way.

[bonifaido]

I think what could be working here is to scan the kubernetes.io/dockercfg type Secrets in the namespace of the actual deployment, and search for the one containing the auth part for the repository of the image. Pass this in trigger/poll/manager.go:scan().
I'm happy to submit a pull-request with a draft implementation.

[rimusz]

interesting approach

[rusenask]

Hi, I had similar plan according to https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/. It appears that secret should be defined in deployment:

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
    - name: private-reg-container
      image: <your-private-image>
  imagePullSecrets:
    - name: regsecret

And then I could fetch secret by name regsecret from that namespace, it should also be of type kubernetes.io/dockercfg. Since plumbing for passing around image metadata like credentials is not in place yet - I will need to do it myself.

What would be really helpful is to get some actual use cases on how you are using private registries.

[bonifaido]

Actually in my case, I don't specify the imagePullSecret in all cases, because this can be bypassed via serviceaccounts, which is a more convenient use I think: https://kubernetes.io/docs/concepts/containers/images/#referring-to-an-imagepullsecrets-on-a-pod

This needs to be done for each pod that is using a private registry.

However, setting of this field can be automated by setting the imagePullSecrets in a serviceAccount resource.

[rusenask]

setting of this field can be automated by setting the imagePullSecrets in a serviceAccount resource.

I understand that k8s automatically sets this field for deployments so when Keel accesses deployment through API would it be set?

I will create transport for secrets from providers to triggers using imagePullSecrets, once it's there - it will be easier to add more options for authenticating.

[bonifaido]

Unfortunately I haven't found them so this is a runtime thing I guess, I checked the pods and the deployments as well.

[rusenask]

awesome!

[bonifaido]

I'm curious why do one needs to provide the imagePullSecret in the values file, if it is already part of the Pod descriptor on the API? Right now I don't have to define imagePullSecret in the Helm configuration, and now I have to define it two times.

[rusenask]

I will have a look again at it, but it doesn't look like Healm releases have pod descriptors inside them.

[rimusz]

@bonifaido
Helm charts/k8s deployments by default do not have imagePullSecret in pods descriptions by default, at least you put them in by yourself.

[bonifaido]

If keel checks the deployment descriptor this is right, but on the API this information is available without specifying it by hand.

[rusenask]

Are you talking about Helm API or Kubernetes API and which provider?

[bonifaido]

Polling provider and the Kubernetes API.

[rusenask]

Ok, I think you mean polling trigger and Helm provider, right?
https://keel.sh/user-guide/

The reason why secret reference needs to be defined in values.yml when using Helm provider is that Keel doesn't contact Kubernetes API directly, it communicates with Helms Tiller and uses Tillers API to start release updates, therefore you get revisions 1,2,3, etc..
I was quite happy getting rid of k8s client library from Helm provider but it's necessary anyway later when we retrieve secrets.

I will have another look at how we can identify deployments from helm releases :)

[rusenask]

Slightly upgraded secret retrieving mechanism, it's no longer needed to define secrets specifically for Keel anywhere. It will figure it out from where to get them. Triggering alpha image build.

[rimusz]

nice one :)

[bonifaido]

Great! Just checking out the new version.

[bonifaido]

Somehow I get this error:

time="2017-08-01T08:07:53Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://registry.ng.bluemix.net/v2/: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":null}]}\\n\")" image="registry.ng.bluemix.net/bonifaido/http-counter:0.0.7" 
time="2017-08-01T08:07:53Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://registry.ng.bluemix.net/v2/: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":null}]}\\n\")" image_name="registry.ng.bluemix.net/bonifaido/http-counter:0.0.7" registry_username= 
time="2017-08-01T08:07:53Z" level=error msg="trigger.poll.manager: failed to start watching repository" error="Get https://registry.ng.bluemix.net/v2/: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":null}]}\\n\")" image="registry.ng.bluemix.net/bonifaido/http-counter:0.0.7" schedule="@every 1m"

I have installed the keel right now with:

helm upgrade --install keel keel --set helmProvider.enabled="true" --set image.tag=alpha --set image.pullPolicy=Always

This is how my Pod looks like on the API:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"http-counter-2295768256","uid":"5ef19a71-7690-11e7-ac73-562b543d49b9","apiVersion":"extensions","resourceVersion":"2120322"}}
    prometheus.io/port: "8000"
    prometheus.io/scrape: "true"
  creationTimestamp: 2017-08-01T08:06:49Z
  generateName: http-counter-2295768256-
  labels:
    app: http-counter
    pod-template-hash: "2295768256"
  name: http-counter-2295768256-xqhtt
  namespace: default
  ownerReferences:
  - apiVersion: extensions/v1beta1
    controller: true
    kind: ReplicaSet
    name: http-counter-2295768256
    uid: 5ef19a71-7690-11e7-ac73-562b543d49b9
  resourceVersion: "2120364"
  selfLink: /api/v1/namespaces/default/pods/http-counter-2295768256-xqhtt
  uid: 5ef280c8-7690-11e7-ac73-562b543d49b9
spec:
  containers:
  - env:
    - name: REDIS_HOST
      value: http-counter-redis
    - name: REDIS_PASSWORD
      valueFrom:
        secretKeyRef:
          key: redis-password
          name: http-counter-redis
    image: registry.ng.bluemix.net/bonifaido/http-counter:0.0.7
    imagePullPolicy: Always
    name: http-counter
    ports:
    - containerPort: 8000
      name: prometheus
      protocol: TCP
    - containerPort: 8080
      name: http-counter
      protocol: TCP
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 250m
        memory: 128Mi
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-8qx69
      readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
  - name: bluemix-default-secret
  nodeName: x.x.x.x
  restartPolicy: Always
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - name: default-token-8qx69
    secret:
      defaultMode: 420
      secretName: default-token-8qx69
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2017-08-01T08:06:50Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-08-01T08:07:09Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-08-01T08:06:49Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://d7a44226e511f170caf8824866c5a25dae34c95a21f56858226e1f0df35f791e
    image: registry.ng.bluemix.net/bonifaido/http-counter:0.0.7
    imageID: docker-pullable://registry.ng.bluemix.net/bonifaido/http-counter@sha256:f3d03500c524a753134a50b8c652b6223cd9ec40652fd510a70ddedbcd39a213
    lastState: {}
    name: http-counter
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-08-01T08:07:05Z
  hostIP: x.x.x.x
  phase: Running
  podIP: 172.30.178.214
  startTime: 2017-08-01T08:06:50Z

[rusenask]

do you see a message similar to
INFO[0000] secrets.defaultGetter: secret looked up successfully image=index.docker.io/karolisr/webhook-demo namespace=default provider=helm registry=index.docker.i
just before those errors?

[bonifaido]

No this message is not in the logs, this is the last message before the error: time="2017-08-01T08:06:58Z" level=info msg="registry client: getting digest" registry="https://registry.ng.bluemix.net" repository=bonifaido/http-counter tag=0.0.7

[rusenask]

added additional log message to show when no secrets for pod found (will remove it later):
time="2017-08-01T10:45:31Z" level=info msg="secrets.defaultGetter: no secrets for image found" image=index.docker.io/karolisr/keel namespace=default provider=helm registry=index.docker.io
Could you try alpha again?

[bonifaido]

Yes, this error message appears now: time="2017-08-01T10:57:13Z" level=info msg="secrets.defaultGetter: no secrets for image found" image=registry.ng.bluemix.net/bonifaido/http-counter namespace=default provider=helm registry=registry.ng.bluemix.net

The secrets in the default namespace:

$ kubectl get secrets
NAME                     TYPE                                  DATA      AGE
bluemix-default-secret   kubernetes.io/dockercfg               1         27d

[rusenask]

Pushed yet another alpha image to display more debugging info, in that previous log as values it should also show selector, how it's looking for pods.

It looks like it could be labeling/selector issue.
Keel identifies which pod belongs to Helm by app name and release name, example webhook-demo chart labels:

  labels:
    app: {{ template "name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }

https://github.com/webhookrelay/webhook-demo/blob/master/webhookdemo/templates/deployment.yaml#L5-L8

While your pod labels:

  labels:
    app: http-counter
    pod-template-hash: "2295768256"

I expected this to be default things on Helm. Could you add release label to your deployment?

[bonifaido]

This is how my Pod looks like now:

$ kubectl get pods -o yaml http-counter-20233072-6j8m1
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"http-counter-20233072","uid":"adcc485f-76b0-11e7-ac73-562b543d49b9","apiVersion":"extensions","resourceVersion":"2125556"}}
    prometheus.io/port: "8000"
    prometheus.io/scrape: "true"
  creationTimestamp: 2017-08-01T11:58:06Z
  generateName: http-counter-20233072-
  labels:
    app: http-counter
    chart: http-counter-0.0.7
    heritage: Tiller
    pod-template-hash: "20233072"
    release: http-counter
...

But still the same error message, I can't see the new log messages about the selectors :(

[rusenask]

Try pulling again alpha, previously haven't correctly tagged image, sorry :(. You should see something like this:

time="2017-08-01T12:59:09Z" level=info msg="secrets.defaultGetter.lookupSecrets: no secrets for image found" image=index.docker.io/karolisr/keel namespace=default pod_selector="app=keel,release=keel" pods_checked=0 provider=helm registry=index.docker.io

and then use that selector to see if correct pod is retrieved (in my case it's keel):

kubectl --namespace=kube-system get pods -l "app=keel,release=keel"
NAME                        READY     STATUS    RESTARTS   AGE
keel-keel-618517890-7h1d1   1/1       Running   0          1m

[bonifaido]

Okay, I have upgraded keel (please post the image digest, so I can make 100% sure I'm using the exasct same version), and now the error message saying no secrets for image found has disappeared, so I don't see the pod_selector it is using, but the UNAUTHORIZED error still exists.

time="2017-08-01T13:35:05Z" level=info msg="secrets.defaultGetter.lookupSecrets: no secrets for image found" image=index.docker.io/karolisr/keel namespace=default pod_selector="app=keel,release=keel" pods_checked=0 provider=helm registry=index.docker.io 
time="2017-08-01T13:36:00Z" level=info msg="registry client: getting digest" registry="https://registry.ng.bluemix.net" repository=bonifaido/http-counter tag=0.0.7 
time="2017-08-01T13:36:00Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://registry.ng.bluemix.net/v2/: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"authentication required\\\",\\\"detail\\\":null}]}\\n\")" image="registry.ng.bluemix.net/bonifaido/http-counter:0.0.7" 

Keel version:

image: karolisr/keel:alpha
imageID: docker-pullable://karolisr/keel@sha256:bd1e96bd8ddf04c1e4fda5d1fa3e853949d9ce2f63f3b65b3b4a2d38e4786ad1

[rusenask]

Digest is correct.

How did you create your secret? It seems that it finds the secret but then doesn't decode it.
I used:

kubectl create secret docker-registry myregistrykey --docker-server=https://index.docker.io/v1/ --docker-username=<user> --docker-password=<pass> --docker-email=<email>

Pushed new image that should show looked up secret names. Digest 90515df8f4b5bc95042f1bbeee05221ae6f27b8f2ded55286063d2a8e4b901c1

Edit: Also, some more info would be helpful - Kubernetes and Helm versions.

[bonifaido]

I think we are now on track:

time="2017-08-01T14:41:31Z" level=info msg="secrets.defaultGetter.lookupSecrets: pod secrets found" image=registry.ng.bluemix.net/bonifaido/http-counter namespace=default pod_selector="app=http-counter,release=http-counter" provider=helm registry=registry.ng.bluemix.net secrets="[bluemix-default-secret]" 
time="2017-08-01T14:41:31Z" level=warning msg="secrets.defaultGetter.lookupSecrets: docker credentials were not found among secrets" image=registry.ng.bluemix.net/bonifaido/http-counter namespace=default provider=helm registry=registry.ng.bluemix.net secrets="[bluemix-default-secret]" 
time="2017-08-01T14:41:31Z" level=info msg="registry client: getting digest" registry="https://registry.ng.bluemix.net" repository=bonifaido/http-counter tag=0.0.7

The issue is that my secret bluemix-default-secret.data..dockercfg has the format of

{"registry.ng.bluemix.net":{"auth": "the auth token of registry"}}

I guess the code looks for usernames and password right now. You have to base64 --decode "auth", and split it by : username will be part[0], and password is part[1].

[rusenask]

yeah, that's not something I can probably add today as I have plans this evening. Sorry about that, token support will definitely be added in the next few days. If you could add credentials to your secrets and see whether everything else works as expected - it would be great.

[bonifaido]

Absolutely no issues, and thanks for the support and blazing fast responses!

[stickycode]

I'm trying to do use Artifactory as docker store with polling using the Kubernetes provider and it seems the user is not coming through.

Its picking up the secret OK and shows the correct username in the log, but somehow does not seem to be sending it on correctly. I've tried using the api key and user/pass that are validated on the Artifactory side.

Using 90515df8f4b5bc95042f1bbeee05221ae6f27b8f2ded55286063d2a8e4b901c1
Kube 1.5.7

time="2017-08-02T02:23:47Z" level=debug msg="performing scan" 
time="2017-08-02T02:23:47Z" level=info msg="secrets.defaultGetter: secret looked up successfully" image=company-name-docker.jfrog.io/app-report-services namespace=delivery provider=kubernetes registry=company-name-docker.jfrog.io 
time="2017-08-02T02:23:47Z" level=info msg="registry client: getting digest" registry="https://company-name-docker.jfrog.io" repository=app-report-services tag=10.29 
time="2017-08-02T02:23:47Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://company-name-docker.jfrog.io/v2/: http: non-successful response (status=403 body=\"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 403,\\n    \\\"message\\\" : \\\"Forbidden\\\"\\n  } ]\\n}\")" image="company-name-docker.jfrog.io/app-report-services:10.29" 
time="2017-08-02T02:23:47Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://company-name-docker.jfrog.io/v2/: http: non-successful response (status=403 body=\"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 403,\\n    \\\"message\\\" : \\\"Forbidden\\\"\\n  } ]\\n}\")" image_name="company-name-docker.jfrog.io/app-report-services:10.29" registry_username=namebuild 
time="2017-08-02T02:23:47Z" level=error msg="trigger.poll.manager: failed to start watching repository" error="Get https://company-name-docker.jfrog.io/v2/: http: non-successful response (status=403 body=\"{\\n  \\\"errors\\\" : [ {\\n    \\\"status\\\" : 403,\\n    \\\"message\\\" : \\\"Forbidden\\\"\\n  } ]\\n}\")" image="company-name-docker.jfrog.io/app-report-services:10.29" schedule="@every 1m"

[rusenask]

@bonifaido - added support for those base64 auth strings in #74. Created new alpha image: alpha: digest: sha256:06655c0045a2ae63f7b5a092042233f210929184ae659c75c55799236562cd81 size: 950.

@stickycode - I assume there can be issues with artifactory as I have seen problems in other libraries authenticating to it, especially that 403 error. Could you please open a separate issue specifically for Artifactory related problems? I will need to get a dev environment with it, saw that they have a 30 days trial, should be good enough.

[rusenask]

Implemented in #74, available from release 0.4.3. If there any issues then please create a new ticket, this one became a bit too long :) Thanks for help everyone!

[maehld]

Hi. We're using node configuration to avoid imagePullSecrets in our deployments (as described here https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry under "Configuring Nodes to Authenticate to a Private Repository". Is there a way to get keel to work with such a setup?

[rusenask]

Hi @maehld, please create a new issue for this. Keel can't access local docker config, we will have to find another way. It would be good to know more, whether you are running your own private registry and what kind of authentication it expects (basic, token).