ingress validate webhook concurrent issue about ingress-nginx

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug
/kind support
/triage needs-information

To a reader, the configuration seems like options to be opted-in or opted-out using the values file, so kindly explain the details and full description of why you think the current behavior is a bug.

Also kindly describe in tiny details, what is the change you are expecting.

[fengxx]

thanks, updated expected result with reasoning

[longwuyuan]

Please wait for others to comment.

Personally I think that if a user chooses not to configure the ingress.spec.ingressClassName field in a ingress resource, its a deviation from recommended practice because the ingress api already changed in K8S v1.24.

So if a user makes a clear choice to not set a ingressClassName, in a ingress resource, then they should ideally make the effort to install with controller with the field controller.watchIngressWithoutClass to true.

[fengxx]

sorry, I double checked the logs, found out the webhook watchIngressWithoutClass is not the root cause.
when spec.ingressClassName is missing, validation webhook will be called twice, first it will ignore payload because ingressClassName is missing, and k8s will mutate the resource to add spec.ingressClassName back because default ingress class is set to nginx, then webhook will validate it again.

The issue is we have two ingress with same hostname, and each alone is valid, but when they merged together, it became a problem.

I guess there is a concurrent issue that the webhook validation tasks were done in two standance nginx instances seperatly, and each returned success result, but it is hard to reproduce

# file1.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test1
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: ingress.test
      http:
        paths:
          - backend:
              service:
                name: web-traffic-svc
                port:
                  name: bar-port
            path: /api/v1/(\d+)/bar
            pathType: ImplementationSpecific

---
# file2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      location /api/v1/invalid {
       return 404;
      }
  name: test2
spec:
  rules:
    - host: ingress.test
      http:
        paths:
          - backend:
              service:
                name: bar-svc
                port:
                  name: bar
            path: /
            pathType: Prefix

Anyway, thanks for the help, I think the issue can be resolved by merging two config together

[longwuyuan]

Thanks for the update @fengxx . Kindly close the issue since not a problem anymore

[fengxx]

thanks, I finally found a stable way to reproduce it, for anyone interested, here is the script

1. prereq

# enable allowSnippetAnnotations
namespaceOverride: nginx-default
controller:
  allowSnippetAnnotations: true
  ingressClass: nginx-default
  ingressClassResource:
    default: true
    enabled: true
    name: nginx-default
    controllerValue: "k8s.io/ingress-nginx-default"
  config:
    annotation-value-word-blocklist: 'load_module,lua_package,_by_lua,root,serviceaccount'
    enable-underscores-in-headers: 'true'

# file1.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test1
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
    - host: ingress.test
      http:
        paths:
          - backend:
              service:
                name: web-traffic-svc
                port:
                  name: bar-port
            path: /api/v1/(\d+)/bar
            pathType: ImplementationSpecific

---
# file2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      location /api/v1/invalid {
       return 404;
      }
  name: test2
spec:
  rules:
    - host: ingress.test
      http:
        paths:
          - backend:
              service:
                name: bar-svc
                port:
                  name: bar
            path: /
            pathType: Prefix

2. reproduce

for x in $(seq 100);do
    kubectl delete ingress test1 test2
    sleep 1
    ( kubectl create -f test1.yaml 2>/dev/null|| echo "failed to create test1" ) &
    ( kubectl create -f test2.yaml 2>/dev/null|| echo "failed to create test2" ) &
    if kubectl get ingress test1 test2 > /dev/null;then
        echo "created two ingress, job done!"
        exit
    fi
done