Annotation whitelist-source-range not using client real IP about ingress-nginx

[barywhyte]

I am currently having this same issue but on Azure. Adding nginx.ingress.kubernetes.io/whitelist-source-range: "74.234.138.x/32" annotation basically make the service internally (from pods that have access within the same namespace) and externally (from 74.234.138.x, over the internet) inaccessible. Removing the annotation restore access back to the service. It's totally strange to me.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug

Please enable proxy-protocol on the NLB as well as in the controller https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#proxy-protocol

/kind support

[rouke-broersma]

Because of this bug report we decided to test this before upgrading. We are not experiencing this problem so this does indeed seem to be a problem related to your setup and not with the upgrade itself.

[Kavuti]

Hi,
I tried to activate the proxy protocol, but I got errors. More in detail, I get logs of broken headers like this:

2024/05/02 09:32:24 [error] 445#445: *4633986 broken header: "84�x�^��۩" while reading PROXY protocol, client: 172.31.15.204, server: 0.0.0.0:443

I did the following operations:

• Activated the proxy protocol v2 on the AWS NLB Target Groups for ports 443 and 80
• Changed the ingress-nginx-controller ConfigMap to set the use-proxy-protocol option to true
• Changed the ingress-nginx-controller Service by editing the value of the following annotation:

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true,proxy_protocol_v2.enabled=true

Anyway, during these days, while I was checking other similar issues, I changed the ingress-nginx-controller Service by adding more annotations. Here's the full list of annotations present on the ingress-nginx-controller Service.

annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: nginx
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /healthz
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: '80'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-type: nlb

None of these annotations actually changed the real behaviour of the controller.

@longwuyuan The docs you linked me talk about the proxy protocol on the AWS ELB, which means the Classic Load Balancer and not the Network Load Balancer. On the Classic version, the linked AWS docs talk just about proxy protocol v1, while on the Network version, just the v2 is available.
Moreover, seems that the broken header is affecting other people too, like in the issue #9643 from the previous year.
If you have any suggestion on how to resolve this, I'm available to test it.

@rouke-broersma Regarding the upgrade, I don't think my problem is strictly related to the ingress-nginx version. I upgraded the controller in another cluster and everything went fine.

[longwuyuan]

There is one issue about the proxy-protocol-v2 and they had the same problem (which they solved AFAIK). Searching for the issue number now

[longwuyuan]

Check if the info here helps in any way #10982

[Kavuti]

While I was checking on this

Check if the info here helps in any way #10982

I made some modification and redeployed the ingress-nginx with externalTrafficPolicy: Cluster and the proxy protocol enabled both on the controller and the Load Balancer. I also changed the healthcheck port as suggested in the issue. Anyway, that wasn't working.

Then I reverted the configuration to the preivous one, which at least served not-whitelisted traffic. Anyway, the situation got worse and the services where the IP was wrong, now weren't serving traffic anymore. The connections to these services were being closed. The Chrome browser showed the ERR_CONNECTION_CLOSED error and there was no trace about that requests in the ingress-nginx logs.

Since this was causing a real downtime on the systems, I opted to completely remove ingress-nginx, which led to the removal of the Network Load Balancer on AWS. After reainstalling ingress-nginx, the new Load Balancer has been created and everything started working again, the whitelist annotation too.

Something I noticed was that the DNS records on Route53 were actually pointing to the NLB but they were an Alias typed to be used on Classic or Application Load Balancers. I corrected that records too, that may have been managed by an old version of external-dns with an old ingress-nginx. Anyway, I don't have any proof that this may have affected the traffic (which worked until the first update, like I mentioned above in the issue).

I suspect that there was something not working with that particular NLB instance. Anyway, if the same problem is happening on Azure, the problem may be in some internal (mis)configuration of ingress-nginx, or something between the ingress-nginx and the Load Balancer?