phenixblue / imageswap-webhook

Image Swap Mutating Admission Webhook for Kubernetes

Hi there 👋

imageswap-webhook's Issues

Fix code scanning alert - Regular expression injection

Tracking issue for:

• https://github.com/phenixblue/imageswap-webhook/security/code-scanning/22

MutatingWebhookConfiguration update issues

What would you like to be added:

1. Currently, the MutatingWebhookConfiguration is created by init-containers, which means if we want to update existing MutatingWebhookConfiguration, we have to do

kubectl delete -f install.yaml && kubectl create -f install.yaml

to let init-containers works again and check if existing MutatingWebhookConfiguration needs updating. If there a better way to do this? By Kubernetes operators?

2. The compare_mwc_fields function in imageswap-init.py only checks keys in new MutatingWebhookConfiguration now.

for key in sorted(new):

Which means if newer MutatingWebhookConfiguration has less key then the existing one, init-containers will assert them to be same, which is actually not.

Why is this needed:

To avoid MutatingWebhookConfiguration updating failures

Bump `docker/build-push-action` to v2

What would you like to be added:

Need to bump the docker/build-push-action to v2 in image build workflow

Why is this needed:

v1 has now been deprecated

Record original image in annotation

What would you like to be added:

The ImageSwap webhook should record the original image reference within an annotation if a swap occurs.

Why is this needed:

This will give a historic record of image swaps per resource for troubleshooting purposes.

This will need to account for multiple containers/init-containers within a pod

Maybe a json based list within an annotation value.

Example:

annotations:
  imageswap-webhook-swaps: ["image1", "image2", "image3"]

Self-signed cert error with testing/test-deployment.yaml

Hi, I have successfully set up imageswap-webhook on GKE but when trying the very same setup on-prem (offline env) I keep getting this error when running the test deployment from your repository:

ubuntu@juju:~/nymcard/devops/registry/imageswap-webhook$ kubectl apply -f ./testing/test-deploy.yaml -n test1
Error from server (InternalError): error when creating "./testing/test-deploy.yaml":
Internal error occurred: failed calling admission webhook "imageswap-webhook.k8s.twr.io":
Post https://imageswap-webhook-svc.kube-system.svc:443/?timeout=30s: x509: certificate signed by unknown authority

What is missing here? I have tried to manually copy the cert generated by your script to the cluster nodes (and updated the CA certs there) but no luck...

How can I debug it further? Any idea?

Extend the same concept for imagePullSecrets as well

HI,

Can we have this extended for imagePullSecrets as well, because in order to access an private registry , secret is also required.. If the same can be added through webhooks, that would be great

ImageSwap 1.4.2 inherits vulnerabilities from image python:3-alpine thats was latest at build time

What would you like to be added:

A fixed tag (not a moving tag, such as latest or python:3-alpine) base image for Dockerfiles, in order for the base image to be easily inspectables for vulnerabilities.

Also, final imageswap and imageswap-init images should be free of vulnerabilities, at least those marked CRITICAL and HIGH.

A vulnerability-free image on python:3-latest timeline is python:3.8.12-alpine3.15 (if the projects must stick to python 3.8, specified in python_version of requires section in Pipfiles), or python:3.11.0a3-alpine3.15 (if imageswap can bump to Python 3.11).

Why is this needed:

Current image (v.1.4.2) has various vulnerabilities that can be inspected via:

• trivy image thewebroot/imageswap-init:v1.4.2 trivy-report-imageswap-init.txt
• trivy image thewebroot/imageswap:v1.4.2 trivy-report-imageswap.txt

Most of those vulnerabilities are inherited by the ptyhon:3-alpine image, being the latest when v1.4.2 of imageswap was built.

Note that fixing vulnerabilities in base image might not be enough, since some vulnerabilities derive from python packages:

• in imageswap-init, kubernetes 11.0.0 [CVE-2020-1747] should be upgraded to 12.0.1 (at least)
• in imageswap, werkzeug 2.0.1 [CVE pyup.io-42050] could be upgraded to 2.0.2 (albeit this "vulnerability" affects just SameSite cookie, not important in this context)

However, those could be addressed in a different issue.

Add exclusion label to skip image swap

What would you like to be added:

Need to add logic to look for a specific label and change behavior of the webhook based on the label's value.

Example:

imageswap-webhook=disable

Why is this needed:

This will allow for manual override of the imageswap if needed.

Cert setup hangs webhook

I think maybe deploy/webhook-ssl-cert-gen.sh has some problem as it keeps finishing with exit 1 when waiting for the cert to be created, and thus the secret cannot be set up correctly, it seems:

$ ./deploy/webhook-ssl-cert-gen.sh \
>   --service imageswap-webhook-svc \
>   --secret imageswap-webhook-certs \
>   --namespace default
creating certs in tmpdir /tmp/tmp.aAGzCUMfUa
Generating RSA private key, 2048 bit long modulus
.......+++
.......................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io "imageswap-webhook-svc.default" deleted
certificatesigningrequest.certificates.k8s.io/imageswap-webhook-svc.default created
NAME                            AGE       REQUESTOR   CONDITION
imageswap-webhook-svc.default   0s        admin       Pending
certificatesigningrequest.certificates.k8s.io/imageswap-webhook-svc.default approved
ERROR: After approving csr imageswap-webhook-svc.default, the signed certificate did not appear on the resource. Giving up after 10 attempts.

But it is there, it is just the jsonpath that is not getting the right stuff?

$ kubectl  get csr imageswap-webhook-svc.default
NAME                            AGE       REQUESTOR   CONDITION
imageswap-webhook-svc.default   1m        admin       Approved

$ kubectl  get csr imageswap-webhook-svc.default  -o jsonpath='{.status.certifi
cate}'
[empty response]
$

[...]
status:
  conditions:
  - lastUpdateTime: 2019-02-19T15:15:31Z
    message: This CSR was approved by kubectl certificate approve.
    reason: KubectlApprove
    type: Approved

If I create the secret anyways, by commenting out the exit 1 condition, the webhook pod hangs with this command if I exec into it, so I presume it hangs for other calls as well with some cert issue being the culprit?

# curl -vX GET https://localhost:5000/
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying ::1...
* TCP_NODELAY set
* connect to ::1 port 5000 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
[after some time it exits the container automatically]

The end result of all that is that imageswap-webhook is not working, not even processing calls to it and not printing any logs inside the pod 😢

Any idea?

"library image" support for "docker.io" "index.docker.io"

What would you like to be added:

This is how imageswap determine whether a image is a library image now:

if image_split[1] == "" and image_split[2] == "":

It can deal with cases like "nginx", but can not detect "docker.io/nginx" and "index.docker.io/nginx" as library images.
Because:

>>> "nginx".partition("/")
('nginx', '', '')
>>> "docker.io/nginx".partition("/")
('docker.io', '/', 'nginx')
>>> "index.docker.io/nginx".partition("/")
('index.docker.io', '/', 'nginx')

Why is this needed:
To cover more cases when input images are from first level repositories

Add logic for init-containers

Need to add logic to handle patching of init-containers.

Istio sidecar injection is an example use case to test functionality against

Ability to remove org/project during swap - MAPS

What would you like to be added:

The ability to do what the deprecated legacy mode can do but within the MAPS method.
Currently, I cannot see a way to reproduce what the legacy method can do, which is to transform say docker.io/a/b/c/d:v1 to quay.io/x/y/z/d:v1

Maybe we could use a new reserved keyword ([replace] in the example below) and wildcard, e.g.:

default::
[replace]quay.io*::docker.io/x/y/z

Expectation: quay.io/a/b/c/d:v1 -> docker.io/x/y/z/d:v1

The following would be equivalent, but now would not replace quay.io/a/d:v1.

default::
[replace]quay.io/a/b/*::docker.io/x/y/z

Why is this needed:

Sometimes the whole repo path needs amending, this ability appears to only be possible using legacy mode or exact matching - which is not ideal for when dealing with a large number of images.

Push Images to multiple registries

What would you like to be added:

Push imageswap container images to multiple container image registries beyond DockerHub (ie. GCR, quay.io, etc.)

Why is this needed:

Docker images for the imageswap project should be pushed to multiple registries for availability/diversity concerns.

Imageswap swaps out complete registry and repository path

What happened:

Deploy an image with definition somepublic.io/repo/subrepo/image:4.0.
Using IMAGE_PREFIX=someprivate.io to swap the registry, the resulting image was someprivate.io/image:4.0 (note the missing paths).

What you expected to happen:

I would have expected the resulting image to be someprivate.io/repo/subrepo/image:4.0.

How to reproduce it (as minimally and precisely as possible):

Take any kubernetes resource with an image with more than one slash in the path and look at the replacement.

Anything else we need to know?:

If this is indeed intended behaviour, I think it would be valuable to control it somehwo to allow partial replacement to keep repository structure. This is interesting when having multiple internal registries (e.g. dev + prod).

Environment:

• Kubernetes version (use kubectl version): 1.18.17
• Cloud provider or hardware configuration: AKS (Azure)
• Others: tried with latest release + master of imageswap-webhook

Retrieve API Version from request

What happened:

Due to an mistake I created PODs with apiVersion "v1beta1" instead of "v1".
In my Kubernetes cluster everything worked well, but the Webhook had no effect.
In the logfile I could see, that it was active and working, but the created but was unmodified.

What you expected to happen:

The POD should have been created with the imageswap modifications.

How to reproduce it (as minimally and precisely as possible):

Create a POD with apiKind "v1beta1" and an image, which is mapped by Image-Swap.

apiVersion: v1beta1
kind: Pod
metadata:
  name: demopod
spec:
  containers:
    - name: demopod
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP

In the Image-Swap debuglog of the "Admission Review" you see that the field "apiVersion" of the response is set to "v1".
Kubernetes seems to ignore this modification and the POD is created without the Image-Swap modifications.

• Kubernetes version (use kubectl version):

Kubernetes v1.21

• Cloud provider or hardware configuration:

AWS EKS

License

Hi there,

Which license is this repo under? Would like to know before I start using it.

Many thanks!

Swap isn't keeping repository name

Hello!

First of all i would like to thank you for your great work!

I'm trying to redirect the image pull to my private registry but whenever i set the IMAGEPREFIX imageswap cut the "repository" portion off.

Example:

• Original request:
  openshift/origin-haproxy-router:latest

• Modified request:
  registry.localdomain.com/origin-haproxy-router:latest

Is it supposed to work like this?

Also, a suggestion if you allow me: it would be nice to rewrite the request based on the FQDN (let's suppose docker.io) to something else. I know that there's a project (https://github.com/indeedeng-alpha/harbor-container-webhook/blob/main/README.md) that supposedly does that but it only works with kubernetes 1.16+.

Rewriting unqualified images is something we can achieve by so many ways.

Thanks!

Cleanup map files/tests when support for ":" as a separator is removed

What would you like to be added:

Testing map configurations will have a mix of : and :: separator syntax while we transition away from the older : syntax.

Once we remove support for the : syntax we will need to cleanup the existing map configs and unit tests.

Why is this needed:

General project hygiene

Add structured logging

Need to add a proper logging library and move to structured logging

Helm Chart considerations

What would you like to be added:

Hi. I suggest an official Helm Chart for this project.

Why is this needed:

It would be easier to adjust the static resources within the deploy folder to infrastructural needs.

Examples:

• Changing the policy for the MutationWebhookConfiguration with kustomize is only possible (to my knowledge) by fully replacing the configmap value. Which is the whole webhook config.

• Ressource that are not necessary for the own infrastructure could be disabled. Like the HorizontalPodAutoscaler for smaller clusters.

support custom key names for bring your own certificate

Hi,
It looks like the init sidecar is hard-coded to use cert.pem and tls.pem. This prevents one from using cert-manager to handle webhook certificates, for those of us who can't use the built-in k8s certificate generation (e.g. canonical charmed kubernetes default installation).

Would you consider allowing for an environment variable that replaces the strings 'cert.pem' and 'tls.pem' with a user-provided value?

Cert-manager uses tls.key, tls.crt, and ca.crt by default.

Support replacement

Replace public registry to private one by custom configuration

Nice work ;-)

Hello,
I stumbled across this when looking for a python example of a webhook to learn how it works. Found here exactly what I was looking for.
Just one suggestion: The creation of the required cert secret can easily be done by another job/container. The people from cert-manager created one that injects such a cert by annotations:

https://github.com/jetstack/cert-manager/tree/master/cmd/cainjector
https://docs.cert-manager.io/en/latest/reference/cainjector.html

Best regards,
Michael.

Environment variable to set listening port

What would you like to be added:

I'd like to be able to set which port the application listens on, instead of being fixed to 5000.

Why is this needed:

It would add flexibility in allowing me to re-use already open ports as opposed to having to open a new one.

Mapping docker.io to localhost incorrectly prepends new host twice

What happened:
I tried mapping docker.io to my local Harbor pull cache project, which resulted in the mapped host appearing twice in the pod container image.

example/image -> localhost:30003/dockerhub/localhost:30003/dockerhub/example/image

What you expected to happen:
The substitution should only happen once.

example/image -> localhost:30003/dockerhub/example/image

How to reproduce it (as minimally and precisely as possible):
I don't believe the port is relevant here, but still using the locally hacked version to get around that until the next release with the :: change.

default:
docker.io:localhost:30003/dockerhub

Anything else we need to know?:
I figured maybe adding an empty entry for localhost: would have been enough to override the second pass which was apparently triggered according to the logs, but that didn't work. Looks like this will happen for any target host that doesn't have a . in it, just not sure that will ever be an issue for anything other than localhost. I was able to get a hack working to bypass the issue for now.

imageswap-webhook/app/imageswap/imageswap.py

Line 262 in d7d8d65

if "." in image_split[0] and image_split[1] != "" and image_split[2] != "":

if "." in image_split[0] and image_split[1] != "" and image_split[2] != "":
    image_registry = image_split[0]
elif image_split[0].lower().startswith("localhost"):
    image_registry = image_split[0]
else:
    # Set docker.io if no registry is detected
    image_registry = "docker.io"
    no_registry = True

Environment:

• Kubernetes version (use kubectl version): 1.20
• Cloud provider or hardware configuration: AWS/EKS
• Others:

pod in crashloop back

I tried installing imageswap on one of the k8s cluster im in charge with and im getting the following error message (see logs below)...
I can see the certificat request status is approved.

NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
imageswap.imageswap-system.cert-request 4m12s kubernetes.io/kubelet-serving system:serviceaccount:imageswap-system:imageswap-sa Approved

[2022-11-11 19:38:49,691] INFO: ImageSwap Init
[2022-11-11 19:38:49,691] INFO: Starting TLS init process
[2022-11-11 19:38:49,715] INFO: Did not find secret "imageswap-tls" in the "imageswap-system" namespace
[2022-11-11 19:38:49,715] INFO: Generating new cert/key pair for TLS
[2022-11-11 19:38:49,764] INFO: Building K8s CSR
[2022-11-11 19:38:49,767] INFO: Looking for existing CSR
[2022-11-11 19:38:49,788] INFO: Deleting k8s csr
[2022-11-11 19:38:49,808] INFO: Existing certificate request deleted
[2022-11-11 19:38:49,809] INFO: Create k8s CSR
[2022-11-11 19:38:49,831] INFO: Certificate signing request "imageswap.imageswap-system.cert-request" has been created
[2022-11-11 19:38:49,846] INFO: Patch k8s CSR: imageswap.imageswap-system.cert-request
[2022-11-11 19:38:49,864] INFO: Certificate signing request "imageswap.imageswap-system.cert-request" is approved
[2022-11-11 19:38:49,879] INFO: Waiting for certificate approval
[2022-11-11 19:38:49,887] INFO: Waiting for certificate approval
......
[2022-11-11 19:38:54,832] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,839] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,849] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,858] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,868] INFO: Waiting for certificate approval
[2022-11-11 19:38:54,868] INFO: Timed out reading certificate request "imageswap.imageswap-system.cert-request"
Traceback (most recent call last):
File "/app/imageswap-init.py", line 1201, in
main()
File "/app/imageswap-init.py", line 1190, in main
init_tls_pair(imageswap_namespace_name)
File "/app/imageswap-init.py", line 677, in init_tls_pair
tls_pair = build_tls_pair(
File "/app/imageswap-init.py", line 347, in build_tls_pair
tls_cert_pem = get_tls_cert_from_request(
File "/app/imageswap-init.py", line 319, in get_tls_cert_from_request
tls_cert = base64.b64decode(k8s_csr.status.certificate)
File "/usr/local/lib/python3.10/base64.py", line 80, in b64decode
s = _bytes_from_decode_data(s)
File "/usr/local/lib/python3.10/base64.py", line 45, in _bytes_from_decode_data
raise TypeError("argument should be a bytes-like object or ASCII "
TypeError: argument should be a bytes-like object or ASCII string, not 'NoneType'

Adding option to [PREFIX] images

What would you like to be added:

Is there any chance that you could support the action [PREFIX], so when is used with default mapping, it swaps all the images with the requested prefix?

something like
[PREFIX]default::internal.registry.com

Why is this needed:

In order to simplify the operations and also keep track of the original source of the images, we tend to clone public images into our internal repository.

The repository url is like: internal.registry.com

Given above registry url, public images will end up in our internal registry with following url:

• internal.registry.com/docker.io/nginx
• internal.registry.com/gcr.io/mint-linux

If we could use this [PREFIX] action, it would be great.

Otherwise, if the current mapping function supports this scenario, I would appreciate if you can give a sample map config.

ImageSwap does not honour IMAGESWAP_DISABLE_AUTO_MWC

What happened:
I provided the env var IMAGESWAP_DISABLE_AUTO_MWC (introduced in v1.5.1) to stop ImageSwap from creating/reconciling the MutatingWebhookConfiguration. I wanted to manage it externally using Pulumi. However, ImageSwap did not honour the env var; it finally errored because I had no ConfigMap volume mounted, but it should have skipped the code that logs INFO: Did not find existing MWC "imageswap-webhook".

❯ kubectl -n imageswap-7dee7409 logs -c imageswap-init -f pod/imageswap-fc645b17-76df75cc8b-d6m5x
[2022-11-11 16:02:28,909] INFO: ImageSwap Init
[2022-11-11 16:02:28,910] INFO: Starting TLS init process
[2022-11-11 16:02:28,924] INFO: Existing TLS cert and key found
[2022-11-11 16:02:28,925] INFO: Days until Cert Expiration: 89
[2022-11-11 16:02:28,925] INFO: Generating new cert/key pair for TLS
[2022-11-11 16:02:28,995] INFO: Building K8s CSR
[2022-11-11 16:02:28,997] INFO: Looking for existing CSR
[2022-11-11 16:02:29,009] INFO: Deleting k8s csr
[2022-11-11 16:02:29,018] INFO: Existing certificate request deleted
[2022-11-11 16:02:29,018] INFO: Create k8s CSR
[2022-11-11 16:02:29,026] INFO: Certificate signing request "imageswap.imageswap-7dee7409.cert-request" has been created
[2022-11-11 16:02:29,031] INFO: Patch k8s CSR: imageswap.imageswap-7dee7409.cert-request
[2022-11-11 16:02:29,042] INFO: Certificate signing request "imageswap.imageswap-7dee7409.cert-request" is approved
[2022-11-11 16:02:29,048] INFO: Waiting for certificate approval
[2022-11-11 16:02:29,053] INFO: Waiting for certificate approval
[2022-11-11 16:02:29,059] INFO: Found approved certificate
[2022-11-11 16:02:29,061] INFO: Using existing secret "imageswap-tls" in namespace "imageswap-7dee7409"
[2022-11-11 16:02:29,061] INFO: Waiting for race winning pod to startup
[2022-11-11 16:02:29,061] INFO: Still waiting for race winning pod to startup
[2022-11-11 16:02:29,061] INFO: Writing cert and key locally
[2022-11-11 16:02:29,069] INFO: Did not find existing MWC "imageswap-webhook"
[2022-11-11 16:02:29,069] ERROR: Error opening MWC template file "/mwc/imageswap-mwc.yaml": 
[Errno 2] No such file or directory: '/mwc/imageswap-mwc.yaml'

What you expected to happen:
I expected the init container to skip creating the MutatingWebhookConfiguration.

How to reproduce it (as minimally and precisely as possible):
Run the init container, passing a minimal set of environment variables. When in the container, comment out the 1135: init_tls_pair(imageswap_namespace_name) line since we're not interested in that part.

https://github.com/phenixblue/imageswap-webhook/blob/v1.5.1/app/imageswap-init/imageswap-init.py#L1135

❯ docker run --rm -it --entrypoint sh \
    -e IMAGESWAP_DISABLE_AUTO_MWC=TRUE \
    -e IMAGESWAP_NAMESPACE_NAME=mynamespace \
    -e IMAGESWAP_POD_NAME=mypod \
    thewebroot/imageswap-init:v1.5.1

# comment out line 1135
/app # vi imageswap-init.py

# Run the script. It fails on loading kubeconfig, but it should not get that far with IMAGESWAP_DISABLE_AUTO_MWC=TRUE
/app # ./imageswap-init.py 
[2022-11-11 17:04:41,548] INFO: ImageSwap Init
[2022-11-11 17:04:41,548] INFO: Exception loading incluster configuration: Service host/port is not set.
[2022-11-11 17:04:41,548] INFO: Loading local kubeconfig
[2022-11-11 17:04:41,548] ERROR: Exception loading local kubeconfig: Invalid kube-config file. No configuration found.

# change `lower` to `lower()`
/app # vi imageswap-init.py
/app # grep lower imageswap-init.py 
    if imageswap_disable_auto_mwc.lower() == "true":

# Run the script. It correctly skips MWC manipulation
/app # ./imageswap-init.py 
[2022-11-11 17:06:15,602] INFO: ImageSwap Init
[2022-11-11 17:06:15,602] INFO: ImageSwap is running with auto-mwc disabled. You will need to deploy the MWC on your own
[2022-11-11 17:06:15,602] INFO: Done

/app # python
Python 3.10.6 (main, Aug 10 2022, 03:54:46) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> "HELLO".lower
<built-in method lower of str object at 0xffff9c757170>
>>> "HELLO".lower == "hello"
False
>>> "HELLO".lower()
'hello'
>>> "HELLO".lower() == "hello"
True

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

❯ kubectl version --short
Client Version: v1.23.8
Server Version: v1.23.8-gke.1900

• Cloud provider or hardware configuration: GKE
• Others:

Error installing PDB on newer K8s versions

What happened:

It looks like there are API changes in newer K8s versions for PodDisruptionBudget resources. We get the following error when trying to install on k8s v1.25+

error: resource mapping not found for name: "imageswap-pdb" namespace: "imageswap-system" from "/Users/phenixblue/GIT/github/tmp/imageswap-webhook/deploy/install.yaml": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first

What you expected to happen:

Install cleanly

How to reproduce it (as minimally and precisely as possible):

• Install cluster with kind using v1.25+ node image
• Install ImageSwap

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

Map doesn't parse correctly when a port is specified

What happened:
My use case requires the mapping of a public endpoint to an internal Harbor proxy cache exposed via NodePort. Using the map below, the imageswap deployment reports the following error.

[2021-08-20 16:40:01,187] ERROR in app: Exception on / [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/app/imageswap.py", line 110, in mutate
    needs_patch = swap_image(container_spec)
  File "/app/imageswap.py", line 260, in swap_image
    swap_maps = build_swap_map(imageswap_maps_file)
  File "/app/imageswap.py", line 212, in build_swap_map
    (key, val) = line.split(":")

What you expected to happen:
The map should parse correctly and apply the mapped image to a pod on creation.

How to reproduce it (as minimally and precisely as possible):

default:
registry.foobar.com:localhost:30003/foobar

Anything else we need to know?:
I've worked around this in the meantime by making the following change locally:

imageswap-webhook/app/imageswap/imageswap.py

Line 212 in 4be30ee

(key, val) = line.split(":")

(key, val) = line.split(":", 1)

Environment:

• Kubernetes version (use kubectl version): 1.20
• Cloud provider or hardware configuration: AWS/EKS
• Others:

default imageswap-sa serviceaccount does not has permission to read extension-apiserver-authentication configmap in kube-system namespace

What happened:
after run kubectl apply -f install.yaml, the init container can not get the correct ca cert
What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
kubectl apply -f deploy/install.yaml
kubectl auth can-i get cm --as=system:serviceaccount:imageswap-system:imageswap-sa -n kube-system
no
Anything else we need to know?:

Environment:
bare k3s

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4+k3s1", GitCommit:"3e250fdbab72d88f7e6aae57446023a0567ffc97", GitTreeState:"clean", BuildDate:"2021-08-19T19:09:53Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4+k3s1", GitCommit:"3e250fdbab72d88f7e6aae57446023a0567ffc97", GitTreeState:"clean", BuildDate:"2021-08-19T19:09:53Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• Others:

Support IPV6

What would you like to be added:
Support IPV6 for imageswapper

Why is this needed:
Required for ipv6 based k8s cluster. IPV6 is popular & is necessary because it provides a vast number of unique addresses for the growing number of devices, improves security, & network efficiency.

Bump code-ql action to v2

What would you like to be added:

CodeQL Action v1 will be deprecated on December 7th, 2022. Please upgrade to v2. For more information, see https://github.blog/changelog/2022-04-27-code-scanning-deprecation-of-codeql-action-v1/

Why is this needed:

Keep security scanning up to date

Bump KinD Node images to test against latest K8s Versions

What would you like to be added:

KinD node images currently target k8s 1.16, 1.17, 1.18, and 1.19

These should be updated to support k9s 1.18, 1.19, 1.20, and 1.21

Why is this needed:

To test compatibility against community supported k8s releases.

simple image defs misdetect registry...

What happened:
imageswap-maps was specified as:

  maps: |
    default:harbor-repo.vmware.com/dockerhub-proxy-cache
    docker.io:harbor-repo.vmware.com/dockerhub-proxy-cache

Image was specified as: rabbitmq:3.8.18-management
Image was replaced with: harbor-repo.vmware.com/dockerhub-proxy-cache

Logs:

[2021-07-29 20:27:32,808] INFO in imageswap: Processing container: default/hello-world-server-0
[2021-07-29 20:27:32,811] INFO in imageswap: ImageSwap Webhook running in "MAPS" mode
[2021-07-29 20:27:32,812] DEBUG in imageswap: No Swap map for "rabbitmq" detected, using default map
[2021-07-29 20:27:32,812] DEBUG in imageswap: Swap Map = "default" : "harbor-repo.vmware.com/dockerhub-proxy-cache"
[2021-07-29 20:27:32,813] INFO in imageswap: External image definition detected: rabbitmq:3.8.18-management
[2021-07-29 20:27:32,813] INFO in imageswap: External image updated to Internal image: harbor-repo.vmware.com/dockerhub-proxy-cache
[2021-07-29 20:27:32,813] INFO in imageswap: Processing init-container: default/hello-world-server-0
[2021-07-29 20:27:32,813] INFO in imageswap: ImageSwap Webhook running in "MAPS" mode
[2021-07-29 20:27:32,816] DEBUG in imageswap: No Swap map for "rabbitmq" detected, using default map
[2021-07-29 20:27:32,816] DEBUG in imageswap: Swap Map = "default" : "harbor-repo.vmware.com/dockerhub-proxy-cache"
[2021-07-29 20:27:32,816] INFO in imageswap: External image definition detected: rabbitmq:3.8.18-management
[2021-07-29 20:27:32,817] INFO in imageswap: External image updated to Internal image: harbor-repo.vmware.com/dockerhub-proxy-cache
[2021-07-29 20:27:32,817] DEBUG in imageswap: Needs patch
[2021-07-29 20:27:32,818] INFO in imageswap: Diffing original request to modified request and generating JSONPatch
[2021-07-29 20:27:32,819] DEBUG in imageswap: JSON Patch: [{"op": "replace", "path": "/spec/containers/0/image", "value": "harbor-repo.vmware.com/dockerhub-proxy-cache"}, {"op": "replace", "path": "/spec/initContainers/0/image", "value": "harbor-repo.vmware.com/dockerhub-proxy-cache"}]

What you expected to happen:

Image should have been: harbor-repo.vmware.com/dockerhub-proxy-cache/rabbitmq:3.8.18-management

How to reproduce it (as minimally and precisely as possible):
Clean kind cluster, above map:

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
kubectl apply -f https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/docs/examples/hello-world/rabbitmq.yaml

Anything else we need to know?:
It looks like swap_image() in imageswap.py splits on / and then assumes that if there's a . in the first component, it's a registry. For a single-component (no /), this is incorrect...the . in this case is in the image tag, not part of a hostname.

Environment:
kind cluster:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-07-12T20:40:20Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

Fix unit tests for multi-container patches

What would you like to be added:

Need to identify a fix for unit tests where multiple container/init-containers are performed.

Why is this needed:

The resulting jsonpatch has no guaranteed ordering (ie. init-container before container) and tests are not consistent with expected assertions.

There are currently 2 tests that this is applicable for. Here's one example

The code for the swap/patch needs to be changed and/or the assertions in the test need to be changed.

Example of the inconsistent output:

Example Patch 1 (init-container first)

[{"op": "replace", "path": "/spec/initContainers/0/image", "value": "jmsearcy/hello-kubernetes:1.5"}, {"op": "replace", "path": "/spec/containers/0/image", "value": "jmsearcy/hello-kubernetes:1.5"}]

Example Patch 2 (container first):

[{"op": "replace", "path": "/spec/containers/0/image", "value": "jmsearcy/hello-kubernetes:1.5"}, {"op": "replace", "path": "/spec/initContainers/0/image", "value": "jmsearcy/hello-kubernetes:1.5"}]

Both produce the same results from a patch perspective, but if order can't be guaranteed, then assertions will be flaky.

ECR Style Repo Support

Hello,

Firstly thank you to the maintainers and contributors to this excellent project. I was hoping something like this existed and it's really exactly what I am after.

I am building a network isolated EKS Kubernetes cluster and want to mirror public docker images to my ECR repository.

I set IMAGE_PREFIX to the ECR repo 111111111111.dkr.ecr.eu-west-1.amazonaws.com/mirror-

But when I run a test container, in this case based off the debian image, instead of mirror-debian being the new image, it is this:

[2020-10-20 13:43:31,809] INFO in imageswap: External image definition detected: debian
[2020-10-20 13:43:31,809] INFO in imageswap: External image updated to Internal image: 111111111111.dkr.ecr.eu-west-1.amazonaws.com/mirror-/debian

I was hoping it would be replaced with 111111111111.dkr.ecr.eu-west-1.amazonaws.com/mirror-debian.

It does look pretty wacky, but ECR repository names can include slashes, etc, so I am going to mirror public image paths exactly as they appear in the YAML. Eg, for non-docker hub URLs:

111111111111.dkr.ecr.eu-west-1.amazonaws.com/mirror-quay.io/calico/node
111111111111.dkr.ecr.eu-west-1.amazonaws.com/mirror-k8s.gcr.io/cluster-proportional-autoscaler-amd64

I hope the report makes sense and you can help.

Thanks in advance

how to deal with 1st level repositories?

hi
according to harbor docs, in case a repository is 1st level (like for example nginx, debian, ubuntu, etc), they suggest to add "library" before the wanted image... how to deal with this automaGically using imageswap? In maps config i can add a single docker.io line, to point images to my harbor server and project, so i find no way to add 2 lines there, one with "library" and one without... the only thing i could do is alter all images lines adding there the library/ pre-path... thoughts? Thanks in advance

Setup Makefile

Need to setup a Makefile to simplify normal development/deployment operations

MutatingWebhookConfigurations - context deadline exceeded

What happened:

Invocation of the webhook fails:

Error from server (InternalError): Internal error occurred: failed calling webhook "imageswap.webhook.k8s.twr.io": failed to call webhook: Post "https://imageswap.imageswap-system.svc:443/?timeout=10s": context deadline exceeded

When I port forward (kubectl port-forward service/imageswap 5000:443) manually accessing the service works fine.
As well as when using a container within the cluster:

kubectl run busybox --rm -it --image=yauritux/busybox-curl -- /bin/sh
# curl -k https://imageswap.imageswap-system.svc:443/healthz

{"date_time":"2022-09-20 13:56:45.835758","health":"ok","pod_name":"imageswap-86999dc8c5-c22vm"}

However, whenever the webhook is called during a pod creation, the application doesn't log any sort of request.

What you expected to happen:

The webhook to successfully send a POST request to the application, followed by the pod being created with the image amendments.

How to reproduce it (as minimally and precisely as possible):

Follow the README guide with the addition of the following environment variable:

• IMAGESWAP_CSR_SIGNER_NAME: "beta.eks.amazonaws.com/app-serving"

Environment:

• Kubernetes version (use kubectl version): 1.22
• Cloud provider or hardware configuration: AWS EKS
• Image version: v1.5.1

imageswap-init should use v1 of CertificateSigningRequest instead of v1beta1

What happened: imageswap-init fails to run in Kubernetes v1.22+ due to v1beta1/CertificateSigningRequest having been removed.

What you expected to happen: imageswap-init to start.

How to reproduce it (as minimally and precisely as possible):

1. Create a Kind cluster running k8s v1.22+: kind create cluster --image=kindest/node:v1.22.4
2. Install imageswap: kubectl apply -f https://raw.githubusercontent.com/phenixblue/imageswap-webhook/v1.3.0/deploy/install.yaml
3. See it burn: watch -n 0.5 kubectl get all --namespace=imageswap-system

Anything else we need to know?:

This should just be a s/v1beta1/v1/g inside of imageswap_init.py. I apologize for not having cut the PR myself; I will do so when time allows.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}

• Cloud provider or hardware configuration: Docker (via Kind)

• Others:

Need to set reinvocationPolicy

In k8s 1.15+ the reinvocationPolicy field was added for mutating admission controllers. This allows for mutations to be revisited if further changes are made after the initial invocation. This should be added and the value set to IfNeeded.

This is helpful for situations like the Istio sidecar injector webhook

More info here: https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#reinvocation-policy

Use `kube-root-ca.crt` configMap for K8s Root CA Cert

What would you like to be added:

Currently there's logic to pull the k8s root ca cert from the in-cluster kubeconfig or a ConfigMap in the pks-system namespace for PKS based clusters.

It looks like the transition of PKS -> TKGi has brought a tighter upstream k8s conformance and the kube-root-ca.crt ConfigMap should exist in all namespaces on conformant clusters.

We should be able to target this ConfigMap for all cluster types and drop the split logic for cluster type and reduce the scope of RBAC permissions outside of the image swap-system namespace

Related code is here

Why is this needed:

Simplify the code base and reduce cluster scoped permissions

Support more granular swap disables in MAP file

What would you like to be added:

Currently the syntax to disable swaps only works for the registry level.

quay.io::

It would be good if this would work down to the image level

quay.io/example/test-image::

Why is this needed:

to support a higher degree of flexibility in image swapping.

Allow MutatingWebhookConfiguration to be deployed separately

What would you like to be added:
Currently the webhook configuration is created/reconciled by imageswap-init from the template in the imageswap-mwc ConfigMap. It would be useful if we could deploy this resource separately to allow its lifecycle to be managed by an external tool (in our case, Pulumi).

We have imported all resources from deploy/install.yaml into Pulumi, but when we use Pulumi to undeploy ImageSwap, the MutatingWebhookConfiguration remains, meaning that pods are denied by default since the controller is not there.

It would be great if ImageSwap could detect whether there is an existing webhook configuration present (which it does already), but have some mechanism to say "use it as-is; do not reconcile it".

Why is this needed:
We are onboarding ImageSwap and this is of a particular problem in our CI clusters where we apply Pulumi from a branch and then revert to the main branch afterwards, ready for the next branch build which may not install ImageSwap. It means pods from future runs are not admitted until we manually delete the webhook configuration.

Support for cluster-wide image-swap without setting labels for each ns.

What would you like to be added:

Is there a way to set this up for cluster-wide level ?

After going through the documentation and testing the existing functionality, I have come to understanding that image-swap works on only namespaces which has label k8s.twr.io/imageswap=enabled.
If there is a way to set image-swap for all the namespaces by default without adding that label it would be great.

Why is this needed:

In airgapped systems, we would require to swap images in every ns, setting the label might be an additional step and can be missed at times.

Support EXACT mapping mode

What would you like to be added:

I would like to add an "EXACT" mapping mode, where docker images are mapped only for exact matches in the mapping configuration (full name with optional image-tag)

The mapping mode can be activated by setting the environment variable IMAGESWAP_MODE=EXACT

Here an example mapping file:

# exact-mappin.conf

mysql/mysql-server:5.6::myownrepo.example.com/base/public-image-cache:mysql_mysql-server_5.6
nvcr.io/nvidia:k8s-device-plugin_v0.9.0::myownrepo.example.com/base/private-image-cache:nvcr.io_nvidia_k8s-device-plugin_v0.9.0
...

Because the ":" is used as a separator and image tags also use ":" as seperator, the fix for ISSUE #49 is needed (using "::" as a seperator).

If now a POD with the image mysql/mysql-server:5.6 is deployed it should be swaped with myownrepo.example.com/base/public-image-cache:mysql_mysql-server_5.6

All images, which do not have an exact match will be left unchanged.

Why is this needed:

I want to build up an infrastructure for image caching and automated vulnerability scans.
New public images will be added to the private docker registry and a new entry is added to the mapping file.
So, I have exact mappings, no logic is needed.
If there are multiple writings with/without "docker.io"/"latest", there will be multiple entries in the mapping file.

Add healthcheck endpoint

Need to add a health check endpoint to be used for liveness/readiness probes.

I think /healthz with a simple 200 OK should be sufficient.

Add K8s Version Compatibility Matrix to README

What would you like to be added:

Need to add a compatibility matrix to the README.

Why is this needed:

To show which versions of ImageSwap Webhook align to specific Kubernetes versions and which Kubernetes versions are supported overall.

New release including PR-85

Hi,

What would you like to be added:
We're interested in trying the replace_maps option introduced in #85
However it looks like there hasn't been a new release of Imageswap and associated Docker Hub image.

Would it be possible to create a new release + Docker image with the changes merged since last August?

Why is this needed:

Always nice to be able to rely on an existing/already built Docker images matching release tags!