geoip2 enabling causes crash of controller v1.10 about ingress-nginx

[longwuyuan]

/remove-kind bug

Lets add the bug label after triaging is completed

• You have been succint and cryptic in your issue description. It does not help

• Please answer the questions that are asked in a new issue template exactly as is because that info is input for a reader to understand and reproduce

• For example, the simple question is how did you install so the expectation is you copy paste the exact command executed and the complete values file used. In your case, since you use ArgoCD, then it is expected that at least the complete values as is from the original install be presented as is. Providing the later modifications is great

• At this stage for a problem like this, its not going to be possible to test ArgoCD in the CI. So please do a helm install command using the same values file and update the results. This will know if using ArgoCD introduces the problem or not

/triage needs-information
/kind support

[ducnm0711]

Noted same issue - bump from helm-chart 4.6.1 to 4.10.0.
Update: chart 4.9.1 doesn't have this issue.
My current config:

controller:
  kind: DaemonSet
  maxmindLicenseKey: change-me
  config:
    use-geoip2: "true"
    log-format-escape-json: "true"
    log-format-upstream: '{
      ....
      "geoip_country_code": "$geoip_country_code"}'

It's appear that use-geoip2: "true" is not taken into nginx config, therefore throw error for additional log label geoip_country_code

2024/04/27 18:12:36 [emerg] 376#376: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable

[longwuyuan]

Any chance you can try to reproduce this problem on a minikube cluster or a kind cluster but only with geoip2 enabled and no other customization

[ducnm0711]

Hi @longwuyuan
Thank you for replying.
Above is the minimum values.yaml config to reproduce this issue.

[longwuyuan]

Thanks. Can you ping me on slack. I am trying to figure out if it can also be reproduced only and only with geop2 enabled and no other customization.

[longwuyuan]

/remove-kind support
/kind bug
/triage accepted

[longwuyuan]

@ducnm0711 I don't have a licence to test so can you change the variable name and test

leev/ngx_http_geoip2_module#92 (comment)

We have removed the non geoip2 components but it will be a least effort test to do this. thanks

[longwuyuan]

/assign

[longwuyuan]

/retitle geoip2 enabling causes crash of controller v1.10

[longwuyuan]

i saw that there is a lite database for free so I will attempt to reproduce on minikube. meanwhile if you can also confirm that no variable no daemonset and no other customization, just enable geoip2, crashes the controller. If you have to use variable, then at least I will try to reproduce with var name as geop2_data_country_code , as shown in the docs and examples of geoip2 module

[longwuyuan]

cc @rikatz

[longwuyuan]

• I don't get a crash when I install without the variable you used like this (licensekey redacted)

helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --set controller.maxmindLicenseKey=dslfhdfddOIUJJDFKDF&DSFDlkf --set controller.config.use-geoip2=true

• I see the flag in the pod

k -n ingress-nginx describe po ingress-nginx-controller-7878f4b84-z5bxs | grep "Args" -A 10                     
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-nginx-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --maxmind-license-key=dslfhdfddOIUJJDFKDF&DSFDlkf

• And I can see geoip2 in nginx.conf

% k -n ingress-nginx exec ingress-nginx-controller-7878f4b84-z5bxs -- cat /etc/nginx/nginx.conf | grep -i geoip2
load_module /etc/nginx/modules/ngx_http_geoip2_module.so;
        # https://github.com/leev/ngx_http_geoip2_module#example-usage
        geoip2 /etc/ingress-controller/geoip/GeoLite2-City.mmdb {
                $geoip2_city_country_code source=$remote_addr country iso_code;
                $geoip2_city_country_name source=$remote_addr country names en;
                $geoip2_city_country_geoname_id source=$remote_addr country geoname_id;
                $geoip2_city source=$remote_addr city names en;
                $geoip2_city_geoname_id source=$remote_addr city geoname_id;
                $geoip2_postal_code source=$remote_addr postal code;
                $geoip2_dma_code source=$remote_addr location metro_code;
                $geoip2_latitude source=$remote_addr location latitude;
                $geoip2_longitude source=$remote_addr location longitude;
                $geoip2_time_zone source=$remote_addr location time_zone;
                $geoip2_region_code source=$remote_addr subdivisions 0 iso_code;
                $geoip2_region_name source=$remote_addr subdivisions 0 names en;
                $geoip2_region_geoname_id source=$remote_addr subdivisions 0 geoname_id;
                $geoip2_subregion_code source=$remote_addr subdivisions 1 iso_code;
                $geoip2_subregion_name source=$remote_addr subdivisions 1 names en;
                $geoip2_subregion_geoname_id source=$remote_addr subdivisions 1 geoname_id;
                $geoip2_city_continent_code source=$remote_addr continent code;
                $geoip2_city_continent_name source=$remote_addr continent names en;one
        geoip2 /etc/ingress-controller/geoip/GeoLite2-ASN.mmdb {
                $geoip2_asn source=$remote_addr autonomous_system_number;
                $geoip2_org source=$remote_addr autonomous_system_organization;

• This means that the variable name is the root-cause of the crash as reported by your error-message

• I will try to play with the variable name and report

[longwuyuan]

• I added the variable I see in the example (not the variable name that you used)

• And I did not see a crash. I also see logs after a curl request. It was local so no real-country-code in logs

% cat values.yaml 
controller:
  maxmindLicenseKey: dslfhdfddOIUJJDFKDF&DSFDlkf
  config:
    use-geoip2: "true"
    log-format-escape-json: "true"
    log-format-upstream: '{
      ....
      "geoip_country_code": "$geoip2_city_country_code"}'

%  helm -n ingress-nginx upgrade ingress-nginx ingress-nginx/ingress-nginx -f values.yaml

% k -n ingress-nginx describe cm ingress-nginx-controller                               
Name:         ingress-nginx-controller
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.10.1
              helm.sh/chart=ingress-nginx-4.10.1
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx

Data
====
use-geoip2:
----
true
allow-snippet-annotations:
----
false
log-format-escape-json:
----
true
log-format-upstream:
----
{ .... "geoip_country_code": "$geoip2_city_country_code"}

BinaryData
====

%  k create deployment test --image nginx:alpine
%  k expose deployment test --port 80
%  k create ing test --class nginx --rule test.mydomain.com/"*"=test:80
%  curl test.mydomain.com --resolve test.mydomain.com:80:172.19.0.3
% k -n ingress-nginx logs ingress-nginx-controller-7878f4b84-z5bxs
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.10.1
  Build:         4fb5aac1dd3669daa3a14d9de3e3cdb371b4c518
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.25.3

-------------------------------------------------------------------------------

I0429 01:10:32.218651      13 flags.go:387] "downloading maxmind GeoIP2 databases"
W0429 01:10:40.846551      13 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0429 01:10:40.846722      13 main.go:205] "Creating API client" host="https://10.96.0.1:443"
I0429 01:10:40.851276      13 main.go:248] "Running in Kubernetes cluster" major="1" minor="29" git="v1.29.2" state="clean" commit="4b8e819355d791d96b7e9d9efe4cbafae2311c88" platform="linux/amd64"
I0429 01:10:40.967775      13 main.go:101] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0429 01:10:40.978880      13 ssl.go:535] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0429 01:10:40.985792      13 nginx.go:264] "Starting NGINX Ingress controller"
I0429 01:10:40.989446      13 event.go:364] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"086e8c4c-6857-43b6-b2bf-9f42efabc582", APIVersion:"v1", ResourceVersion:"1964", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0429 01:10:42.188540      13 nginx.go:307] "Starting NGINX process"
I0429 01:10:42.188619      13 leaderelection.go:250] attempting to acquire leader lease ingress-nginx/ingress-nginx-leader...
I0429 01:10:42.188924      13 nginx.go:327] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0429 01:10:42.189106      13 controller.go:190] "Configuration changes detected, backend reload required"
I0429 01:10:42.198572      13 leaderelection.go:260] successfully acquired lease ingress-nginx/ingress-nginx-leader
I0429 01:10:42.198631      13 status.go:84] "New leader elected" identity="ingress-nginx-controller-7878f4b84-z5bxs"
I0429 01:10:42.229832      13 controller.go:210] "Backend successfully reloaded"
I0429 01:10:42.229888      13 controller.go:221] "Initial sync, sleeping for 1 second"
I0429 01:10:42.229941      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-z5bxs", UID:"6b7659ae-8b6b-402c-bfd7-9c03f51d33a7", APIVersion:"v1", ResourceVersion:"2062", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0429 01:37:04.378551      13 event.go:364] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"086e8c4c-6857-43b6-b2bf-9f42efabc582", APIVersion:"v1", ResourceVersion:"4392", FieldPath:""}): type: 'Normal' reason: 'UPDATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0429 01:37:04.381893      13 controller.go:190] "Configuration changes detected, backend reload required"
I0429 01:37:04.423304      13 controller.go:210] "Backend successfully reloaded"
I0429 01:37:04.423585      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-z5bxs", UID:"6b7659ae-8b6b-402c-bfd7-9c03f51d33a7", APIVersion:"v1", ResourceVersion:"2062", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0429 01:42:06.903116      13 main.go:107] "successfully validated configuration, accepting" ingress="default/test"
I0429 01:42:06.909966      13 store.go:440] "Found valid IngressClass" ingress="default/test" ingressclass="nginx"
I0429 01:42:06.910106      13 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test", UID:"6d0542c6-c1e7-4d3f-9b4f-d6090aa80e7c", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4890", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0429 01:42:06.910378      13 controller.go:190] "Configuration changes detected, backend reload required"
I0429 01:42:06.958640      13 controller.go:210] "Backend successfully reloaded"
I0429 01:42:06.958806      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-z5bxs", UID:"6b7659ae-8b6b-402c-bfd7-9c03f51d33a7", APIVersion:"v1", ResourceVersion:"2062", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0429 01:42:42.204168      13 status.go:304] "updating Ingress status" namespace="default" ingress="test" currentValue=null newValue=[{"ip":"172.19.0.3"}]
I0429 01:42:42.209251      13 event.go:364] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"test", UID:"6d0542c6-c1e7-4d3f-9b4f-d6090aa80e7c", APIVersion:"networking.k8s.io/v1", ResourceVersion:"4946", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
{ .... "geoip_country_code": ""}
{ .... "geoip_country_code": ""}
[~] 

[longwuyuan]

• Finally when I use the variable name you used, I can reproduce the crash

• values.yaml

% cat values.yaml 
controller:
  maxmindLicenseKey: dslfhdfddOIUJJDFKDF&DSFDlkf
  config:
    use-geoip2: "true"
    log-format-escape-json: "true"
    log-format-upstream: '{
      ....
      "geoip_country_code": "$geoip_country_code"}'
[~/Documents/ingressnnginx/issues/11320] 
% k -n ingress-nginx describe cm ingress-nginx-controller 
Name:         ingress-nginx-controller
Namespace:    ingress-nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.10.1
              helm.sh/chart=ingress-nginx-4.10.1
Annotations:  meta.helm.sh/release-name: ingress-nginx
              meta.helm.sh/release-namespace: ingress-nginx

Data
====
allow-snippet-annotations:
----
false
log-format-escape-json:
----
true
log-format-upstream:
----
{ .... "geoip_country_code": "$geoip_country_code"}
use-geoip2:
----
true

BinaryData
====

• logs

 > key="ingress-nginx/ingress-nginx-controller-6j598"
I0429 02:05:02.205347      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-hjrth", UID:"3be1bafb-e7cc-4cb3-9c75-684b4556f28c", APIVersion:"v1", ResourceVersion:"6944", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: 
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/29 02:05:02 [emerg] 59#59: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg1742803439 test failed

-------------------------------------------------------------------------------
I0429 02:05:05.503847      13 controller.go:190] "Configuration changes detected, backend reload required"
E0429 02:05:05.533550      13 controller.go:205] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2024/04/29 02:05:05 [emerg] 60#60: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg3109804287 test failed

-------------------------------------------------------------------------------
E0429 02:05:05.533596      13 queue.go:131] "requeuing" err=<

        -------------------------------------------------------------------------------
        Error: exit status 1
        2024/04/29 02:05:05 [emerg] 60#60: unknown "geoip_country_code" variable
        nginx: [emerg] unknown "geoip_country_code" variable
        nginx: configuration file /tmp/nginx/nginx-cfg3109804287 test failed

        -------------------------------------------------------------------------------
 > key="ingress-nginx/ingress-nginx-controller-admission-vbpg5"
I0429 02:05:05.533673      13 event.go:364] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-7878f4b84-hjrth", UID:"3be1bafb-e7cc-4cb3-9c75-684b4556f28c", APIVersion:"v1", ResourceVersion:"6944", FieldPath:""}): type: 'Warning' reason: 'RELOAD' Error reloading NGINX: 
-------------------------------------------------------------------------------
Error: exit status 1
2024/04/29 02:05:05 [emerg] 60#60: unknown "geoip_country_code" variable
nginx: [emerg] unknown "geoip_country_code" variable
nginx: configuration file /tmp/nginx/nginx-cfg3109804287 test failed

So please change the variable name

/remove-kind bug
/remove-triage accepted
/remove-triage needs-information
/kind support

[longwuyuan]

@jlm0x017 Please re-open the issue if you find a problem with the controller. For now I will close the issue as there is no problem found in the controller. Problem is just the variable name is invalid

/close

[k8s-ci-robot]

@longwuyuan: Closing this issue.

In response to this:

@jlm0x017 Please re-open the issue if you find a problem with the controller. For now I will close the issue as there is no problem found in the controller. Problem is just the variable name is invalid

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jlm0x017]

@longwuyuan Thanks for diving into this. You're exactly right, the variable name was being used in 'log-format-upstream:'; it was an artifact sticking around from prior versions. I identified where this was being set and removed it. 4.10.0+ are running just fine.