Modsecurity log dosent use $request_id from nginx when modsec blocks (403) about ingress-nginx

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug

• Can you please reproduce this on a minimal configuration on a cluster created using kind or minikube
• Please describe why your ingress-nginx-controller service is of type ClusterIP

[longwuyuan]

/triage needs-information

[husa570]

/remove-kind bug

* Can you please reproduce this on a minimal configuration on a cluster created using kind or minikube

I will try to see if I can do that

* Please describe why your ingress-nginx-controller service is of type ClusterIP

We use hostports on controller and have haproxy in front of the cluster

[longwuyuan]

thanks for updating.

• appreciate you will try reproducing in minikube or kind. Please ensure that you use a service --type LoadBalancer or the unique networking of kind configuration as we do it in CI https://github.com/kubernetes/ingress-nginx/blob/main/build/kind.yaml . We don't test this HAProxy in front of ingress-nginx networking in CI so this will help a lot

• At some point of type I hope you will be testing with service type LoadBalancer in front of ingress-nginx as well as that long snippet for modsecurity. Idea is install metallb in minikube if you choose miniube, and configure the minikube ip-address as the starting and ending of the pool for minikube. That way the service type LoadBalancer gets that external-ip

[longwuyuan]

I would test in stages ;

• No modifications except enabling modsec
• Then make one change like req_id annotation but no complete set of rules if possible
• Next one add complete rules
• But I would check the log_format_upstream (because I have not ingested all the info)

[longwuyuan]

/kind support

[longwuyuan]

if you meant to say you can reproduce on minikube, then please do this.

From your minikube cluster, copy/paste the output of commands here in one single post;

• kubectl cluster-info
• helm -n ingress-ingress get values ingress-nginx
• kubectl get all,ing -A -o wide
• kubectl -n ingress-nginx get cm -o wide
• kubectl -n ingress-nginx describe cm ingress-nginx-controller
• kubectl -n ingress-nginx describe po $ingress-nginx-controller-pod-name
• kubectl -n $appnamespace describe ing
• kubectl -n $appnamespace logs $apppodname
• Curl command complete and exactly as used with a -v and its reponse
• kubectl -n ingress-nginx logs $ingress-nginx-controller-podname
• Anh other related info

[longwuyuan]

You can actually reduce he clutter here by deleting less informative posts and posting all that important minikube info in the original issue-description

[longwuyuan]

Also, the controller v1.10.x is using nginx v1.25 (it was v1.21 earlier) so have to check if any upstream nginx changes impacted your log_format or nginx_vars or mosec config etc

[longwuyuan]

Thanks.

I asked for those command outputs so I can reproduce. I suspect that if there is a genuine problem and if it is caused by the controller, then maybe the upgrade of the inernal component nginx (stating that nginx is a component of the controller) from v1.21 to v1.25 has introduced changes that are related.

[husa570]

if you meant to say you can reproduce on minikube, then please do this.

From your minikube cluster, copy/paste the output of commands here in one single post;

* kubectl cluster-info

* helm -n ingress-ingress get values ingress-nginx

* kubectl get all,ing -A -o wide

* kubectl -n ingress-nginx get cm -o wide

* kubectl -n ingress-nginx describe cm ingress-nginx-controller

* kubectl -n ingress-nginx describe po $ingress-nginx-controller-pod-name

* kubectl -n $appnamespace describe ing

* kubectl -n $appnamespace logs $apppodname

* Curl command complete and exactly as used with a -v and its reponse

* kubectl -n ingress-nginx logs $ingress-nginx-controller-podname

* Anh other related info

I will se what I can do, some of this commands extract information that might be sesitive for us, but parts of it I might be able to anonymize

[husa570]

Im stuck at the moment, I cant reproduce it in minkube. One differense between minikube and our kluster is that we use containerd (ver 1.7.10) and not docker, unfortunately I dont seem to have the knowlege to run minikube on containerd.
So at the moment Im stuck with a the fact that the ingress-nginx nginx logs the same req_id twice (happend when we upgraded to 1.10.0) and modsecurity uses it own unique_id.

[husa570]

Deleted most of my "clutter" post and closing this issue unresolved

[longwuyuan]

minikube start --container-runtime --help should show you this

@husa570 we can do a zoom session if you think you are ok with that way to make progress

[husa570]

minikube start --container-runtime --help should show you this

@husa570 we can do a zoom session if you think you are ok with that way to make progress

Thanks but this was another deadend. Minkube worked as expected
Minikube start

minikube start --container-runtime=containerd
😄  minikube v1.33.0 on Ubuntu 20.04 (amd64)
✨  Automatically selected the docker driver. Other choices: none, ssh
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.43 ...
💾  Downloading Kubernetes v1.30.0 preload ...
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
📦  Preparing Kubernetes v1.30.0 on containerd 1.6.31 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The request

curl --resolve waf-demo.localdev.me:8080:127.0.0.1 http://waf-demo.localdev.me:8080/?id=1+union+select+1,2,3/*

And the log, everything works as expected
unique_id=request_id

2024/04/24 13:19:01 [error] 775#775: *7024 [client 127.0.0.1] ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `15' ) [file "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "81"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 15)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.5"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "127.0.0.1"] [uri "/"] [unique_id "41b34f98bb4e4a703d54e6597a5785a1"] [ref ""], client: 127.0.0.1, server: waf-demo.localdev.me, request: "GET /?id=1+union+select+1,2,3/* HTTP/1.1", host: "waf-demo.localdev.me:8080"
{"time": "2024-04-24T13:19:01+00:00", "remote_address": "127.0.0.1", "remote_user": "-", "request": "GET /?id=1+union+select+1,2,3/* HTTP/1.1", "response_code": "403", "referer": "-", "useragent": "curl/7.68.0", "request_length": "115", "request_time": "0.000", "proxy_upstream_uname": "default-demo-80", "proxy_alternative_upstream_name": "", "upstream_addr": "-", "upstream_response_length": "-", "upstream_response_time": "-", "upstream_status": "-", "request_id": "41b34f98bb4e4a703d54e6597a5785a1", "x-forward-for": "127.0.0.1", "uri": "/", "request_query": "id=1+union+select+1,2,3/*", "method": "GET", "http_referrer": "-", "vhost": "waf-demo.localdev.me"}