azure / aks-app-routing-operator

Kubernetes operator that implements AKS Application Routing

AKS Web Application Routing Operator

A Kubernetes operator that manages resources related to AKS Application Routing functionality.

Docs

View the docs folder for more information.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

aks-app-routing-operator's Issues

Enable disabling http protocol on the controller service

The unmanaged ingress controller permits deactivating the HTTP protocol for the controller service by adjusting the helm chart parameter --set controller.service.enableHttp=false.

For the app routing add-on, require a similar configuration parameter to set up HTTPS-only service configurations.

external certs shows the default

local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate

I see the controller accessing the cert from the keyvault, however I only see the default cert on my request via nginx/loadbalancer?

Events:
  Type    Reason                  Age                     From                        Message
  ----    ------                  ----                    ----                        -------
  Normal  SecretRotationComplete  2m45s (x1351 over 45h)  csi-secrets-store-rotation  successfully rotated K8s secret keyvault-aks-helloworld

view the page

here is the logs:

W0403 20:49:55.943237       8 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0403 20:49:55.943356       8 main.go:230] "Creating API client" host="https://10.0.0.1:443"
I0403 20:49:55.969354       8 main.go:274] "Running in Kubernetes cluster" major="1" minor="25" git="v1.25.5" state="clean" commit="34f89fd3fb1a106e1b23d3454b2f2cbf305602a1" platform="linux/amd64"
I0403 20:49:56.117844       8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0403 20:49:56.156436       8 nginx.go:256] "Starting NGINX Ingress controller"
I0403 20:49:56.167175       8 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"app-routing-system", Name:"nginx", UID:"ab23add5-1ee7-4087-9f36-4aa9e6b89231", APIVersion:"v1", ResourceVersion:"888515", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap app-routing-system/nginx
I0403 20:49:57.262751       8 store.go:429] "Found valid IngressClass" ingress="hello-web-app-routing/aks-helloworld" ingressclass="webapprouting.kubernetes.azure.com"
I0403 20:49:57.262953       8 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"hello-web-app-routing", Name:"aks-helloworld", UID:"0728b251-5789-4717-ad35-98c774c24647", APIVersion:"networking.k8s.io/v1", ResourceVersion:"70171", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0403 20:49:57.263085       8 backend_ssl.go:45] Error obtaining X.509 certificate: no object matching key "hello-web-app-routing/aks-helloworld" in local store
I0403 20:49:57.358809       8 nginx.go:299] "Starting NGINX process"
I0403 20:49:57.358865       8 leaderelection.go:248] attempting to acquire leader lease app-routing-system/ingress-controller-leader...
W0403 20:49:57.359297       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
I0403 20:49:57.359362       8 controller.go:167] "Configuration changes detected, backend reload required"
I0403 20:49:57.365888       8 status.go:84] "New leader elected" identity="nginx-5fcff8b64d-kmxtl"
I0403 20:49:57.412721       8 controller.go:184] "Backend successfully reloaded"
I0403 20:49:57.412812       8 controller.go:195] "Initial sync, sleeping for 1 second"
I0403 20:49:57.412888       8 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"app-routing-system", Name:"nginx-5fcff8b64d-q67qb", UID:"831f0f20-5d12-438b-b9f0-a773133c8348", APIVersion:"v1", ResourceVersion:"889106", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0403 20:50:01.166306       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
I0403 20:50:03.178975       8 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"app-routing-system", Name:"nginx", UID:"ab23add5-1ee7-4087-9f36-4aa9e6b89231", APIVersion:"v1", ResourceVersion:"889155", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap app-routing-system/nginx
W0403 20:50:04.500398       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
W0403 20:50:07.833100       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
W0403 20:50:11.166617       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
W0403 20:50:30.529919       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
W0403 20:50:33.864325       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
W0403 20:50:45.910019       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
I0403 20:51:20.797095       8 leaderelection.go:258] successfully acquired lease app-routing-system/ingress-controller-leader
I0403 20:51:20.797185       8 status.go:84] "New leader elected" identity="nginx-5fcff8b64d-q67qb"
W0403 20:52:06.462263       8 controller.go:1334] Error getting SSL certificate "hello-web-app-routing/aks-helloworld": local SSL certificate hello-web-app-routing/aks-helloworld was not found. Using default certificate
I0403 20:52:06.479694       8 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"app-routing-system", Name:"nginx", UID:"ab23add5-1ee7-4087-9f36-4aa9e6b89231", APIVersion:"v1", ResourceVersion:"889835", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap app-routing-system/nginx

I guess I missed something, however there wasn't too much in the docs ?

Affinity Possible

Hello, I had a quick look and I can't see a good way to set the affinity for the Ingress Controller Pods.

Is this something that can be configured in another way, or would it need to be added here?

For now I believe I can use Node Taints for the same purpose, but it's not ideal for our use-case.

Enable nginx annotations

Hello there,

We are still forced to use non-managed NGINX because we use some annotations like client-max-body-size and proxy-body-size. It would be nice to be able to use them with the Azure managed solution.

Thanks,
Leandro

Setting a default certificate in nginx ingress controller gets overridden

We deployed the Web Application Routing Add-on for AKS ingress solution, which enables a Managed Nginx Ingress Controller. While testing, we noticed that when reaching out the Public IP of the managed load balancer (without host headers), a default fake certificate is being created on the Ingress controller.

To avoid this, we instead relied on the Ingress controller’s Default SSL Certificate flag --default-ssl-certificate which contains the default ssl cert for “catch-all” scenario. However, this setting is always getting overridden (may be because it is managed, and there are settings that we cannot override). Wondering if there is a way to persist this setting?

Configurable nginx access logs

What would be the best way to configure application access logs at nginx level? We want to log both the access when passing through the managed nginx and at the application to trace the performance and for debugging purposes.

For now we could edit the nginx configmap to add logging config, for example

log-format-upstream: '{"time": "$time_iso8601", "remote_addr": "$proxy_protocol_addr",
    "x_forwarded_for": "$proxy_add_x_forwarded_for", "request_id": "$req_id", "remote_user":
    "$remote_user", "bytes_sent": $bytes_sent, "request_time": $request_time, "status":
    $status, "vhost": "$host", "request_proto": "$server_protocol", "path": "$uri",
    "request_query": "$args", "request_length": $request_length, "duration": $request_time,"method":
    "$request_method", "http_referrer": "$http_referer", "http_user_agent": "$http_user_agent"
    }'

but it is not supported and will be reconciled/reset following #139

Upgrade security context of controller container to be compatible with "Restricted" official pod security standard

Currently, only the "runAsUser" option is set in the controller's container securityContext.

securityContext:
            runAsUser: 101

Official Azure policies expect "allowPrivilegeEscalation" to be declared false.
Kubernetes clusters should not allow container privilege escalation Azure policy
Gatekeeper template: k8sazurev3noprivilegeescalation

I propose to upgrade the securityContext to match the official "restricted" pod security standard.
Official pod security standards documentation.

Proposed securityContext:

spec:
  template:
    spec:
      containers:
        - name: controller
           securityContext:
             runAsUser: 101
             allowPrivilegeEscalation: false
             seccompProfile:
               type: RuntimeDefault
             capabilities:
               drop:
                 - ALL

[external-dns] Private hosts in public DNZ zone

Hi,
I'm trying to add internal ingress using instruction https://learn.microsoft.com/en-us/azure/aks/create-nginx-ingress-private-controller.
Current controller implementation does not provide configuration options for external-dns controller.

Issue

All hosts from Private DNS Zone private.example.com created in DNS zone example.com also.
This makes impossible to use operator for split-horizon/split-view DNS completely.

Proposal

Add configuration options to NginxIngressController for external-dns controller to filter watched ingresses:

• annotation-filter
• label-filter
• ingress-class (available from v0.13.5)

Enable SSL Passthrough

Hi,

I'd like to be able to add --enable-ssl-passthrough as an argument for starting up NGINX. I currently can't find a way of doing it while using the app-routing plugin, would this be possible?

Thanks,

Ed

DNS zone management when deployment is removed

I have been using the Web Application Routing preview feature for my AKS cluster for a while now. While the lifecycle of the K8s objects are managed by my pipeline in Gitlab, I kind of expected the app-routing-operator to manage the deletion of DNS records that were automatically created when deploying an Ingress resource. This seems not to be the case, so when creating short-lived test environments that are based on Git branch names, the number of DNS records piles up.

It would be a nice addition to the operator if it also performs cleanup created resources.

Is there a plan to add the ability to manually adjust the number of replicas on nginx controllers in app-routing-system?

I modified the "aks-app-routing-operator" deployment to adjust the number of replica, but it was impossible.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: "2024-03-12T08:23:16Z"
  generation: 15
  labels:
    app.kubernetes.io/component: ingress-controller
    app.kubernetes.io/managed-by: aks-app-routing-operator
    app.kubernetes.io/name: nginx
  name: nginx
  namespace: app-routing-system
  ownerReferences:
  - apiVersion: approuting.kubernetes.azure.com/v1alpha1
    controller: true
    kind: NginxIngressController
    name: default
    uid: 6ea3e476-059e-44b3-9f8d-86655ee2a5dc
  resourceVersion: "113876214"
  uid: 37ea9332-baaf-47cf-aef4-f69d0ff2abef
spec:
  progressDeadlineSeconds: 600
  replicas: 2     <=============== I want to revise this

Currently, two nging controllers seem to be the default values.

kubectl get po -n app-routing-system
NAME                     READY   STATUS    RESTARTS   AGE
nginx-5d4cbcf56b-nrt46   1/1     Running   0          40h
nginx-5d4cbcf56b-swq58   1/1     Running   0          40h

Is there a plan to add the ability to manually adjust the number of replica on the nginx controller?

Not able to add CSP Headers with single quotes

Issue
We are not able to add CSP headers that contains single quotes. Example:
nginx.ingress.kubernetes.io/configuration-snippet: >
more_set_headers "Content-Security-Policy: upgrade-insecure-requests;default-src 'self';" ;
...

I did remove ' from annotation-value-word-blocklist in nginx configmap, but it keeps getting reverted to default values after restarting nginx deployment.

I do understand single quote is blocked because of security, but, I couldn't find another way to add CSP header.

Proposal
We should be able to overwrite nginx configmap.

Byte-range support

Hi,
How to enable byte-range support on ingress resource?

On kubernetes/ingress-nginx it can be enabled with annotation, but this does not work with addon controller

annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    proxy_force_ranges on;

Current controller image:

mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller:v1.9.4