Unable to use Proxy Protocol in Knative service about serving

[skonto]

Hi @jdiazper I have some cycles this week let me take a look first. I will get back to it here.

[skonto]

Do you enable the proxy protocol?

I will double check and I will report back here.

Do you install istio via istioctl?

No downsteam I used Openshift Service Mesh.

Could you please describe the steps & order of your knative/istio installation/configuration?

Let me first reproduce and then we can port to Istio as I use Openshift SM.

[skonto]

Hi @jdiazper, not sure about the rest of the setup but I suspect the configuration error you are seeing:

Didn't find a registered implementation for 'envoy.filters.listener.proxy_protocol' with type URL: ''
thread=16

is related to the outdated envoy filter syntax. Check a correct Envoy sample here.
Instead of just:

    - name: envoy.filters.listener.proxy_protocol

you need:

    - name: envoy.filters.listener.proxy_protocol
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol

See a similar issue here. Would you like to try that?

[jdiazper]

Hi @skonto , this indeed has fixed the filter error. Thanks!

However, I am still receiving upstream connect error or disconnect/reset before headers. reset reason: connection termination

For what I can see from the logs, the request reaches the istio gateway pod.

kubectl -n istio-system logs -f istio-ingressgateway-8fccfb4b9-rfrmb

{"connection_termination_details":null,"bytes_sent":95,"response_code_details":"upstream_reset_before_response_started{connection_termination}","bytes_received":0,"upstream_transport_failure_reason":null,"route_name":null,"requested_server_name":null,"upstream_host":"10.99.105.178:8081","upstream_service_time":null,"user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36","upstream_cluster":"outbound|80||knative-local-gateway.istio-system.svc.cluster.local","response_code":503,"authority":"backend.test-service.svc.cluster.local","start_time":"2023-12-12T09:53:02.692Z","downstream_local_address":"10.0.49.117:80","x_forwarded_for":"XXX.XXX.XXX.XXX","downstream_remote_address":"XXX.XXX.XXX.XXX:45822","method":"GET","response_flags":"UC","duration":0,"upstream_local_address":"10.99.105.178:50034","request_id":"6651db19-24a5-4921-99a9-2a0beae96fa7","path":"/","protocol":"HTTP/1.1"}

However when I query the knative pod istio-proxy there is nothing from the request above.

kubectl -n test-service logs -c istio-proxy backend-00001-deployment-67f9d98749-7nj44 | grep "XXX.XXX.XXX.XXX"

[skonto]

Hey could it be that you have set: networking.knative.dev/visibility: "cluster-local" ?

[jdiazper]

Hi @skonto ,

I have modified the service file to look like below

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: backend
  namespace: test-service
  labels:
    networking.knative.dev/disable-auto-tls: "true"
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/initial-scale: "1"
        autoscaling.knative.dev/min-scale: "1"
        autoscaling.knative.dev/max-scale: "10"
    spec:
      containers:
        - image: fake-registry.github.com/fakepath/backend:1.0.0
          env:
            - name: ENVIRONMENT
              value: test

But unfortunatelly I am still receiving the same upstream connect error or disconnect/reset before headers. reset reason: connection termination and request are not reaching the service.

As an additional note, the Kubernetes cluster is on AWS EKS 1.28.

[skonto]

Hi @jdiazper, before even adding the envoy filter which will block any request anyway unless it is a proxy based request (unless you set "allow_requests_without_proxy_protocol": true), let's try fix the Knative setup.

Btw I am not sure if the envoy filter makes sense as it it meant only for tcp traffic. What should happen from what I read is that:

The client IP is retrieved from the PROXY protocol by the gateway and set (or appended) in the X-Forwarded-For and X-Envoy-External-Address header.

Not sure what is successful with the httpbin example above.

Let's do the following to test/fix the Knative setup:

a) Let's avoid using the virtual service.

b) set a sample domain:

kubectl patch configmap/config-domain   --namespace knative-serving   --type merge   --patch '{"data":{"example.com":""}}'

c) use curl -H "Host: backend.test-service.example.com" http://a8ace71c642ba49b39f0f0bc25d9f37b-1902910119.eu-central-1.elb.amazonaws.com

You can check a similar guide here, unfortunately I can't test on EKS at the moment so let's try make Knative work first on the cluster.

[jdiazper]

Hi @skonto ,

Thanks for your help!!

Following your advice I can see how the service is successfully called and the headers contains the client ip (see logs below).

Sample logs from backend service

[2023-12-14 16:30:55 +0000] [7] [DEBUG] GET /
[2023-12-14 16:30:55 +0000] [7] [DEBUG] Request Headers Host: backend.test-service.example.com
User-Agent: curl/7.81.0
Accept: */*
Forwarded: for=XXX.XXX.XXX.XXX;proto=http, for=10.99.134.226
K-Proxy-Request: activator
X-B3-Parentspanid: 4197d0dd9c083110
X-B3-Sampled: 0
X-B3-Spanid: 6db83aed2f1e8d88
X-B3-Traceid: 9bf42975efafa1514197d0dd9c083110
X-Envoy-Attempt-Count: 1
X-Envoy-External-Address: XXX.XXX.XXX.XXX
X-Forwarded-For: XXX.XXX.XXX.XXX, 10.99.134.226, 127.0.0.6
X-Forwarded-Proto: http
X-Request-Id: d1def3cf-339f-4a9e-a4d2-537361c3aeee

While, this is great, all my endpoints are currently using Virtual Service so I can direct the request from https://mydomain.com/ based on the path.

e.g.

https://mydomain.com/service1/hello -> call service1
https://mydomain.com/service2/hello -> call service2
etc...

However if I use the config-domain all my requests should be changed to

https://service1.namespace.mydomain.com/
https://service2.namespace.mydomain.com/
etc...

Is there any way I can continue using the Virtual Service? or an alternative to avoid having to change all the existing endpoints?

Thanks again for your help!

[skonto]

I think what you are looking for is a domain mapping.

Here is an example.
First enable: autocreate-cluster-domain-claims: "true" in config-network cm in knative-serving ns.

Then apply the dm with a custom domain hello.home:

apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata:
  name: hello.home
spec:
  ref:
    name: helloworld-go
    kind: Service
    apiVersion: serving.knative.dev/v1

Then I can do (this is minikube so probably different than your setup and so I use a host header):

$ kubectl get ksvc -n test
NAME            URL                                     LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.test.example.com   helloworld-go-00001   helloworld-go-00001   True

$ kubectl get po -n test
NAME                                              READY   STATUS    RESTARTS   AGE
helloworld-go-00001-deployment-76f4bd5c4b-fjm7j   2/3     Running   0          4s

$ curl -H "Host: hello.home" http://192.168.39.125:32385
Hello Go Sample v1!
    
$ curl -H "Host: helloworld-go.test.example.com" http://192.168.39.125:32385
Hello Go Sample v1!

The virtual services you get:

$ kubectl get virtualservices.networking.istio.io   -n test
NAME                    GATEWAYS                                                                              HOSTS                                                                                                                     AGE
hello.home-ingress      ["knative-serving/knative-ingress-gateway"]                                           ["hello.home"]                                                                                                            6m35s
helloworld-go-ingress   ["knative-serving/knative-ingress-gateway","knative-serving/knative-local-gateway"]   ["helloworld-go.test","helloworld-go.test.example.com","helloworld-go.test.svc","helloworld-go.test.svc.cluster.local"]   8m28s
helloworld-go-mesh      ["mesh"]                                                                              ["helloworld-go.test","helloworld-go.test.svc","helloworld-go.test.svc.cluster.local"]                                    8m28s

Domain mapping is essentially using a rewrite host rule and targets the related gateway.
In detail:

$ kubectl describe virtualservices.networking.istio.io hello.home-ingress    -n test 
Name:         hello.home-ingress
Namespace:    test
Labels:       networking.internal.knative.dev/ingress=hello.home
Annotations:  networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
              serving.knative.dev/creator: minikube-user
              serving.knative.dev/lastModifier: minikube-user
API Version:  networking.istio.io/v1beta1
Kind:         VirtualService
Metadata:
  Creation Timestamp:  2023-12-20T11:35:21Z
  Generation:          1
....
Spec:
  Gateways:
    knative-serving/knative-ingress-gateway
  Hosts:
    hello.home
  Http:
    Headers:
      Request:
        Set:
          K - Network - Hash:  e1c8b6eafa3b8d150f7596327b60280db748bd900e97a4c07791299719425f8d
    Match:
      Authority:
        Prefix:  hello.home
      Gateways:
        knative-serving/knative-ingress-gateway
      Headers:
        K - Network - Hash:
          Exact:  override
    Retries:
    Rewrite:
      Authority:  helloworld-go.test.svc.cluster.local
    Route:
      Destination:
        Host:  helloworld-go.test.svc.cluster.local
        Port:
          Number:  80
      Headers:
        Request:
          Set:
            K - Original - Host:  hello.home
      Weight:                     100
    Match:
      Authority:
        Prefix:  hello.home
      Gateways:
        knative-serving/knative-ingress-gateway
    Retries:
    Rewrite:
      Authority:  helloworld-go.test.svc.cluster.local
    Route:
      Destination:
        Host:  helloworld-go.test.svc.cluster.local
        Port:
          Number:  80
      Headers:
        Request:
          Set:
            K - Original - Host:  hello.home
      Weight:                     100
Events:                           <none>

[jdiazper]

Hi @skonto ,

That's interesting...

I have done the suggested changes above and when I use the domain mapping and I curl hello.home I am getting the same upstream connect error or disconnect/reset before headers. reset reason: connection termination error as if I do it using the virtual service.

What I have done is:

I have edited the configmap (kubectl -n knative-serving edit cm config-network) and added autocreate-cluster-domain-claims: "true" after _example.

kubectl -n knative-serving describe cm config-network

Output:

Name:         config-network
Namespace:    knative-serving
Labels:       app.kubernetes.io/component=networking
              app.kubernetes.io/name=knative-serving
              app.kubernetes.io/version=1.12.2
Annotations:  knative.dev/example-checksum: 0573e07d
              manifestival: new

Data
====
_example:
----
################################
#                              #
#    EXAMPLE CONFIGURATION     #
#                              #
################################

# This block is not actually functional configuration,
# but serves to illustrate the available configuration
# options and document them in a way that is accessible
# to users that `kubectl edit` this config map.
#
# These sample configuration options may be copied out of
# this example block and unindented to be in the data block
# to actually change the configuration.

# ingress-class specifies the default ingress class
# to use when not dictated by Route annotation.
#
# If not specified, will use the Istio ingress.
#
# Note that changing the Ingress class of an existing Route
# will result in undefined behavior.  Therefore it is best to only
# update this value during the setup of Knative, to avoid getting
# undefined behavior.
ingress-class: "istio.ingress.networking.knative.dev"

# certificate-class specifies the default Certificate class
# to use when not dictated by Route annotation.
#
# If not specified, will use the Cert-Manager Certificate.
#
# Note that changing the Certificate class of an existing Route
# will result in undefined behavior.  Therefore it is best to only
# update this value during the setup of Knative, to avoid getting
# undefined behavior.
certificate-class: "cert-manager.certificate.networking.knative.dev"

# namespace-wildcard-cert-selector specifies a LabelSelector which
# determines which namespaces should have a wildcard certificate
# provisioned.
#
# Use an empty value to disable the feature (this is the default):
#   namespace-wildcard-cert-selector: ""
#
# Use an empty object to enable for all namespaces
#   namespace-wildcard-cert-selector: {}
#
# Useful labels include the "kubernetes.io/metadata.name" label to
# avoid provisioning a certificate for the "kube-system" namespaces.
# Use the following selector to match pre-1.0 behavior of using
# "networking.knative.dev/disableWildcardCert" to exclude namespaces:
#
# matchExpressions:
# - key: "networking.knative.dev/disableWildcardCert"
#   operator: "NotIn"
#   values: ["true"]
namespace-wildcard-cert-selector: ""

# domain-template specifies the golang text template string to use
# when constructing the Knative service's DNS name. The default
# value is "{{.Name}}.{{.Namespace}}.{{.Domain}}".
#
# Valid variables defined in the template include Name, Namespace, Domain,
# Labels, and Annotations. Name will be the result of the tag-template
# below, if a tag is specified for the route.
#
# Changing this value might be necessary when the extra levels in
# the domain name generated is problematic for wildcard certificates
# that only support a single level of domain name added to the
# certificate's domain. In those cases you might consider using a value
# of "{{.Name}}-{{.Namespace}}.{{.Domain}}", or removing the Namespace
# entirely from the template. When choosing a new value be thoughtful
# of the potential for conflicts - for example, when users choose to use
# characters such as `-` in their service, or namespace, names.
# {{.Annotations}} or {{.Labels}} can be used for any customization in the
# go template if needed.
# We strongly recommend keeping namespace part of the template to avoid
# domain name clashes:
# eg. '{{.Name}}-{{.Namespace}}.{{ index .Annotations "sub"}}.{{.Domain}}'
# and you have an annotation {"sub":"foo"}, then the generated template
# would be {Name}-{Namespace}.foo.{Domain}
domain-template: "{{.Name}}.{{.Namespace}}.{{.Domain}}"

# tag-template specifies the golang text template string to use
# when constructing the DNS name for "tags" within the traffic blocks
# of Routes and Configuration.  This is used in conjunction with the
# domain-template above to determine the full URL for the tag.
tag-template: "{{.Tag}}-{{.Name}}"

# auto-tls is deprecated and replaced by external-domain-tls
auto-tls: "Disabled"

# Controls whether TLS certificates are automatically provisioned and
# installed in the Knative ingress to terminate TLS connections
# for cluster external domains (like: app.example.com)
# - Enabled: enables the TLS certificate provisioning feature for cluster external domains.
# - Disabled: disables the TLS certificate provisioning feature for cluster external domains.
external-domain-tls: "Disabled"

# Controls weather TLS certificates are automatically provisioned and
# installed in the Knative ingress to terminate TLS connections
# for cluster local domains (like: app.namespace.svc.<your-cluster-domain>)
# - Enabled: enables the TLS certificate provisioning feature for cluster cluster-local domains.
# - Disabled: disables the TLS certificate provisioning feature for cluster cluster local domains.
# NOTE: This flag is in an alpha state and is mostly here to enable internal testing
#       for now. Use with caution.
cluster-local-domain-tls: "Disabled"

# internal-encryption is deprecated and replaced by system-internal-tls
internal-encryption: "false"

# system-internal-tls controls weather TLS encryption is used for connections between
# the internal components of Knative:
# - ingress to activator
# - ingress to queue-proxy
# - activator to queue-proxy
#
# Possible values for this flag are:
# - Enabled: enables the TLS certificate provisioning feature for cluster cluster-local domains.
# - Disabled: disables the TLS certificate provisioning feature for cluster cluster local domains.
# NOTE: This flag is in an alpha state and is mostly here to enable internal testing
#       for now. Use with caution.
system-internal-tls: "Disabled"

# Controls the behavior of the HTTP endpoint for the Knative ingress.
# It requires auto-tls to be enabled.
# - Enabled: The Knative ingress will be able to serve HTTP connection.
# - Redirected: The Knative ingress will send a 301 redirect for all
# http connections, asking the clients to use HTTPS.
#
# "Disabled" option is deprecated.
http-protocol: "Enabled"

# rollout-duration contains the minimal duration in seconds over which the
# Configuration traffic targets are rolled out to the newest revision.
rollout-duration: "0"

# autocreate-cluster-domain-claims controls whether ClusterDomainClaims should
# be automatically created (and deleted) as needed when DomainMappings are
# reconciled.
#
# If this is "false" (the default), the cluster administrator is
# responsible for creating ClusterDomainClaims and delegating them to
# namespaces via their spec.Namespace field. This setting should be used in
# multitenant environments which need to control which namespace can use a
# particular domain name in a domain mapping.
#
# If this is "true", users are able to associate arbitrary names with their
# services via the DomainMapping feature.
autocreate-cluster-domain-claims: "false"

# If true, networking plugins can add additional information to deployed
# applications to make their pods directly accessible via their IPs even if mesh is
# enabled and thus direct-addressability is usually not possible.
# Consumers like Knative Serving can use this setting to adjust their behavior
# accordingly, i.e. to drop fallback solutions for non-pod-addressable systems.
#
# NOTE: This flag is in an alpha state and is mostly here to enable internal testing
#       for now. Use with caution.
enable-mesh-pod-addressability: "false"

# mesh-compatibility-mode indicates whether consumers of network plugins
# should directly contact Pod IPs (most efficient), or should use the
# Cluster IP (less efficient, needed when mesh is enabled unless
# `enable-mesh-pod-addressability`, above, is set).
# Permitted values are:
#  - "auto" (default): automatically determine which mesh mode to use by trying Pod IP and falling back to Cluster IP as needed.
#  - "enabled": always use Cluster IP and do not attempt to use Pod IPs.
#  - "disabled": always use Pod IPs and do not fall back to Cluster IP on failure.
mesh-compatibility-mode: "auto"

# Defines the scheme used for external URLs if auto-tls is not enabled.
# This can be used for making Knative report all URLs as "HTTPS" for example, if you're
# fronting Knative with an external loadbalancer that deals with TLS termination and
# Knative doesn't know about that otherwise.
default-external-scheme: "http"

autocreate-cluster-domain-claims:
----
true

BinaryData
====

Events:  <none>

I restarted istio-system pods

kubectl -n istio-system delete pods --all

apply the domain mapping

domain-mapping.yaml

apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata:
  name: hello.home
spec:
  ref:
    name: backend
    kind: Service
    apiVersion: serving.knative.dev/v1

kubectl -n test-service apply -f domain-mapping.yaml

output

domainmapping.serving.knative.dev/hello.home created

Run commands:

kubectl get ksvc -n test-service

output:

NAME      URL                                       LATESTCREATED   LATESTREADY     READY   REASON
backend   http://backend.test-service.example.com   backend-00001   backend-00001   True    

kubectl get po -n test-service

output:

NAME                                                READY     STATUS    RESTARTS   AGE
backend-00001-deployment-84b7f96856-cxc2f   3/3     Running   0         51m

curl -H "Host: hello.home" http://a8ace71c642ba49b39f0f0bc25d9f37b-1902910119.eu-central-1.elb.amazonaws.com

output:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

However if I run:

curl -H "Host: backend.test-service.example.com" http://a8ace71c642ba49b39f0f0bc25d9f37b-1902910119.eu-central-1.elb.amazonaws.com -v

output

*   Trying 3.120.24.218:80...
* Connected to a8ace71c642ba49b39f0f0bc25d9f37b-1902910119.eu-central-1.elb.amazonaws.com (3.120.24.218) port 80 (#0)
> GET / HTTP/1.1
> Host: backend.test-service.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 0
< content-type: text/html; charset=utf-8
< date: Fri, 22 Dec 2023 15:54:14 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 9
< 
* Connection #0 to host a8ace71c642ba49b39f0f0bc25d9f37b-1902910119.eu-central-1.elb.amazonaws.com left intact

The list of virtual service I get are:

kubectl get virtualservices.networking.istio.io -n test-service

output

NAME                          GATEWAYS                                                                              HOSTS                                                                                                                                                             AGE
hello.home-ingress            ["knative-serving/knative-ingress-gateway"]                                           ["hello.home"]                                                                                                                                                    73s
backend-ingress               ["knative-serving/knative-ingress-gateway","knative-serving/knative-local-gateway"]   ["backend.test-service","backend.test-service.example.com","backend.test-service.svc","backend.test-service.svc.cluster.local"]   44m
backend-mesh                  ["mesh"]                                                                              ["backend.test-service","backend.test-service.svc","backend.test-service.svc.cluster.local"]                                      44m

virutal service

kubectl describe virtualservices.networking.istio.io hello.home-ingress -n test-service

output

Name:         hello.home-ingress
Namespace:    test-service
Labels:       networking.internal.knative.dev/ingress=hello.home
Annotations:  networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
              serving.knative.dev/creator: kubernetes-admin
              serving.knative.dev/lastModifier: kubernetes-admin
API Version:  networking.istio.io/v1beta1
Kind:         VirtualService
Metadata:
  Creation Timestamp:  2023-12-22T15:07:22Z
  Generation:          1
  Managed Fields:
    API Version:  networking.istio.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:networking.knative.dev/ingress.class:
          f:serving.knative.dev/creator:
          f:serving.knative.dev/lastModifier:
        f:labels:
          .:
          f:networking.internal.knative.dev/ingress:
        f:ownerReferences:
          .:
          k:{"uid":"d16410fb-0f8c-4156-b089-c36a496cad63"}:
      f:spec:
        .:
        f:gateways:
        f:hosts:
        f:http:
    Manager:    controller
    Operation:  Update
    Time:       2023-12-22T15:07:22Z
  Owner References:
    API Version:           networking.internal.knative.dev/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  hello.home
    UID:                   d16410fb-0f8c-4156-b089-c36a496cad63
  Resource Version:        43307
  UID:                     f86ce515-84e9-45c5-8289-20b570f2430c
Spec:
  Gateways:
    knative-serving/knative-ingress-gateway
  Hosts:
    hello.home
  Http:
    Headers:
      Request:
        Set:
          K - Network - Hash:  5ab79157aea7d6972dc18ac6a3eaae7e303fba9072348a66f6accffb8afc66ce
    Match:
      Authority:
        Prefix:  hello.home
      Gateways:
        knative-serving/knative-ingress-gateway
      Headers:
        K - Network - Hash:
          Exact:  override
    Retries:
    Rewrite:
      Authority:  backend.test-service.svc.cluster.local
    Route:
      Destination:
        Host:  backend.test-service.svc.cluster.local
        Port:
          Number:  80
      Headers:
        Request:
          Set:
            K - Original - Host:  hello.home
      Weight:                     100
    Match:
      Authority:
        Prefix:  hello.home
      Gateways:
        knative-serving/knative-ingress-gateway
    Retries:
    Rewrite:
      Authority:  backend.test-service.svc.cluster.local
    Route:
      Destination:
        Host:  backend.test-service.svc.cluster.local
        Port:
          Number:  80
      Headers:
        Request:
          Set:
            K - Original - Host:  hello.home
      Weight:                     100
Events:                           <none>

What am I doing wrong?!

[jdiazper]

Hi @skonto ,

Sorry to bother you! Did you have any chance to look at this?

Thanks for your help in advance!!

[jdiazper]

Hi @skonto,

After repeating the process multiple times, I wonder if this could be a bug? If so, could you please change the label accordingly?

Thanks again for your help and support.

[skonto]

I tried this on OCP on AWS by exposing the ingress gateway on the AWS lb:

$ curl  -k -H "Host: hello.home"  https://a7b958e51e73240569946d2dce5c436f-1398313270.us-east-1.elb.amazonaws.com
Hello Go Sample v1!

$ curl  -k -H "Host: helloworld-go-kserve-demo.apps.ci-ln-8y4h6vk-76ef8.origin-ci-int-aws.dev.rhcloud.com"  https://a7b958e51e73240569946d2dce5c436f-1398313270.us-east-1.elb.amazonaws.com
Hello Go Sample v1!

I am using https but that should not affect the header stuff:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: knative-local-gateway
  namespace: knative-serving
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 8081
        protocol: HTTP

Could you pls confirm that you have istio enabled at the Serving ns. In my setup I have something similar to this (sidecars are actually injected to activator for example):

NAME                                     READY   STATUS    RESTARTS   AGE
activator-79bb9b57b8-hbwtx               2/2     Running   0          37m
activator-79bb9b57b8-kvzmw               2/2     Running   0          38m
autoscaler-795fcb857b-5g9jm              2/2     Running   0          38m
autoscaler-795fcb857b-tw4h4              2/2     Running   0          38m
autoscaler-hpa-6f8d5766b9-2zx8p          1/1     Running   0          38m
autoscaler-hpa-6f8d5766b9-zmzv9          1/1     Running   0          38m
controller-7c66679589-2kvtj              1/1     Running   0          37m
controller-7c66679589-h6q7g              1/1     Running   0          37m
domain-mapping-74b6bf4bd5-scwqd          1/1     Running   0          38m
domain-mapping-74b6bf4bd5-xrbqw          1/1     Running   0          38m
domainmapping-webhook-6c5b5496b9-425w5   1/1     Running   0          38m
domainmapping-webhook-6c5b5496b9-8fjrv   1/1     Running   0          38m
net-istio-controller-56cf8f5bd9-lz47f    1/1     Running   0          37m
net-istio-controller-56cf8f5bd9-v8nw2    1/1     Running   0          37m
net-istio-webhook-5665bb679f-b6cbw       1/1     Running   0          37m
net-istio-webhook-5665bb679f-c2zt4       1/1     Running   0          37m
webhook-6477c746f9-25b2n                 1/1     Running   0          38m
webhook-6477c746f9-8c2fd                 1/1     Running   0          37m

Could you also print the status of the dm:

NAME         URL                  READY   REASON
hello.home   https://hello.home   True    

Here are some output about resources created, hope that helps to debug:

virtualsvcs.json
ingresses.json
dms.json

[jdiazper]

Hi @skonto ,

Please find below the requested information:

I can see that sidecar is in most of the pods.

NAME                                    READY   STATUS    RESTARTS   AGE
activator-b7df55675-2z4gg               2/2     Running   0          32m
autoscaler-64655fb9c-s8scd              2/2     Running   0          32m
autoscaler-hpa-5cfbbbf988-c54bm         2/2     Running   0          32m
controller-7b854b6dfb-5zxnl             2/2     Running   0          32m
net-istio-controller-586c554d76-bg8lr   1/1     Running   0          32m
net-istio-webhook-6bcff4d984-xj9k9      2/2     Running   0          32m
webhook-6977cb78d8-bw2br                2/2     Running   0          32m

Status of dm:

NAME         URL                 READY   REASON
hello.home   http://hello.home   True  

I've also comapred files and the main difference (other than expected namespaces and service names) is that in your ingress file you have "serving.knative.openshift.io/enablePassthrough": "true", while in mine that doesn't exists (I don't user OCP), so I guess that makes sense?

When you install istio...

• Do you enable the proxy protocol?
• Do you install istio via istioctl?
• Could you please describe the steps & order of your knative/istio installation/configuration?

Thank you!

[skonto]

I played a bit with envoy filter so I ended up using something like:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
spec:
  workloadSelector:
    labels:
      app: helloworld-go
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: proxy_protocol
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol"
            allow_requests_without_proxy_protocol: false
        - name: tls_inspector
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"

This has to be applied with a selector for every sidecar on the path including the ingressgateway, activator.
So you need separate filters for example with some proper selectors eg ingressgateway:

  workloadSelector:
    labels:
      "istio": "ingressgateway"

Note that I could not apply the latest syntax as in here https://istio.io/latest/docs/reference/config/networking/envoy-filter/ (there is an example there) as in my env LISTENER_FILTER is not supported (would have been more convenient I think).

When setting https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/listener/proxy_protocol/v3/proxy_protocol.proto.html#extensions-filters-listener-proxy-protocol-v3-proxyprotocol to true requests did pass:

curl -k -H "Host: helloworld-go....com"  --haproxy-protocol  https://aef14d3fa841843339ba78a333aba001-1473205865...com:443
Hello Go Sample v1!

With the default false value requests don't pass (also activator needs to have this true to be healthy). In any case if reqs only pass when flag is true then it means that the requests sent are not proxy protocol based. However I see that you need to enable proxy protocol at the AWS lb side: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html#proxy-protocol.
Regarding the ingress annotation there is an extra needed: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" (https://istio.io/v1.15/blog/2020/show-source-ip)? The blog post is old so needs verififcation.

So I suspect you need to make sure each hop has proxy protocol enabled. Could you please make sure you have the above verified a) AWS LB setup, b) annotation if Istio supports it as expected and report back here (I am not able to do the above in my env)?
For the latter I would first check if AWS LB + ISTIO + simple app works as expected with proxy protocol enabled on EKS. Could you do that showing that client ip/host is printed with a regular K8s deployment using Istio (no Knative), as it is not clear to me from your description earlier?

[jdiazper]

Hi @skonto,

Answering your request about "I would first check if AWS LB + ISTIO + simple app works as expected with proxy protocol enabled on EKS. Could you do that showing that client ip/host is printed with a regular K8s deployment using Istio (no Knative), as it is not clear to me from your description earlier?"

I have followed this guide to deploy a simple service and enable proxy protocol.

As a summary:

1. I have installed Istio using istioctl

istioctl install

output:

This will install the Istio 1.20.0 "default" profile (with components: Istio core, Istiod, and Ingress gateways) into the cluster. Proceed? (y/N) y
✔ Istio core installed                                                                                                                                                    
✔ Istiod installed                                                                                                                                                        
✔ Ingress gateways installed                                                                                                                                              
✔ Installation complete                                                                                                                                                   Made this installation the default for injection and validation.

2. Then I deployed a workload, httpbin, in namespace foo with sidecar injection enabled and exposed httpbin through an ingress gateway.
3. Then I called the service.

curl http://a681b3f3f054e4cddbc814a9d8477d38-1939553369.eu-central-1.elb.amazonaws.com/headers

output

{
  "headers": {
    "Accept": "*/*", 
    "Host": "a681b3f3f054e4cddbc814a9d8477d38-1939553369.eu-central-1.elb.amazonaws.com", 
    "User-Agent": "curl/7.81.0", 
    "X-B3-Parentspanid": "be4209649da0b1fd", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "b8d396f508cbcdf7", 
    "X-B3-Traceid": "8c18d72ddca54aecbe4209649da0b1fd", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Envoy-Internal": "true", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/foo/sa/httpbin;Hash=b0949ca3855a3d4b02ca6ccae79594d70e4a063b42d453d4bcfaa879f8a42d13;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }
}

As you can see at this stage the source ip is not available.

4. Then, as I use AWS Classic Load Balancer, I have used TCP Proxy configuration.

enable-proxy-protocol.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogEncoding: JSON
    accessLogFile: /dev/stdout
    defaultConfig:
      gatewayTopology:
        proxyProtocol: {}
  components:
    ingressGateways:
    - enabled: true
      name: istio-ingressgateway
      k8s:
        hpaSpec:
          maxReplicas: 10
          minReplicas: 5
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

istioctl install -f enable-proxy-protocol.yaml

output:

This will install the Istio 1.20.0 "default" profile (with components: Istio core, Istiod, and Ingress gateways) into the cluster. Proceed? (y/N) y
✔ Istio core installed                                                                                                                                                    
✔ Istiod installed                                                                                                                                                        
✔ Ingress gateways installed                                                                                                                                              
✔ Installation complete                                                                                                                                                   Made this installation the default for injection and validation.

5. I have restarted the istio-system pods.

kubectl -n istio-system delete pods --all

6. I sent an API request to the same endpoint as before

curl http://a681b3f3f054e4cddbc814a9d8477d38-1939553369.eu-central-1.elb.amazonaws.com/headers

output

{
  "headers": {
    "Accept": "*/*", 
    "Host": "a681b3f3f054e4cddbc814a9d8477d38-1939553369.eu-central-1.elb.amazonaws.com", 
    "User-Agent": "curl/7.81.0", 
    "X-B3-Parentspanid": "3db3aa3e2a1d92ce", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "ec7cdac78ce22af4", 
    "X-B3-Traceid": "54e8dc89c1f4163e3db3aa3e2a1d92ce", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Envoy-External-Address": "XXX.XXX.XXX.XXX", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/foo/sa/httpbin;Hash=0ba9152cd166a2de5856f52bf11aac279457d28fb1462c4f76d5ded16b1330d0;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }
}

The IP address is available in X-Envoy-External-Address header.

Now that we know that Istio is handleing the source ip, I have deployed knative (as shown in previous comments) but when I curl the load balancer I get the upstream connect error or disconnect/reset before headers. reset reason: connection termination.

Could it be that knative-serving/knative-ingress-gateway specified in my Virtual Service is not proxy-protocol ready? If so, how can I enable it?

[skonto]

Could it be that knative-serving/knative-ingress-gateway specified in my Virtual Service is not proxy-protocol ready? If so, how can I enable it?

It should be enabled since you have that set globaly for all gateways. To set it for each gateway you can do (you have to double check with your Istio version):

metadata:
  annotations:
    "proxy.istio.io/config": '{"gatewayTopology" : { "proxyProtocol": {} }}'

I would recommend to use istioctl to check the configuration at the ingress gateway and compare it with the one before installing Knative. Could be that something is overridden.

1. You mentioned "X-Envoy-External-Address" works without Knative. Did you apply the envoy filter?
2. Could you please show how your Knative service behaves without applying the envoy filter and after you installed Knative? Can you access the ksvc if you target the ingress service?

[jdiazper]

Hi @skonto ,

1. You mentioned "X-Envoy-External-Address" works without Knative. Did you apply the envoy filter?

It worked withouth the Envoy Filter.

2. Could you please show how your Knative service behaves without applying the envoy filter and after you installed Knative? Can you access the ksvc if you target the ingress service?

I just ran a simple curl like:

curl -v -HHost:test.me http://a75311345845043c08caa3ec2390a77c-192380941.eu-central-1.elb.amazonaws.com/test

output

*   Trying 3.122.77.145:80...
* Connected to a75311345845043c08caa3ec2390a77c-192380941.eu-central-1.elb.amazonaws.com (3.122.77.145) port 80 (#0)
> GET /test HTTP/1.1
> Host:test.me
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< access-control-allow-credentials: true
< access-control-allow-origin: http://test.me
< content-length: 0
< content-type: text/html; charset=utf-8
< date: Wed, 07 Feb 2024 17:04:00 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 62
< 
* Connection #0 to host a75311345845043c08caa3ec2390a77c-192380941.eu-central-1.elb.amazonaws.com left intact

If I look at the logs I can see the printed headers.

[2024-02-07 17:04:00 +0000] [7] [DEBUG] Request Headers Host: backend.test-service.svc.cluster.local
User-Agent: curl/7.81.0
Accept: */*
Forwarded: for=10.0.14.83;proto=http, for=10.99.41.9, for=127.0.0.6
K-Proxy-Request: activator
X-B3-Parentspanid: 06dc0b522c9404be
X-B3-Sampled: 0
X-B3-Spanid: b68f05c3de8e79c2
X-B3-Traceid: 21706d1e61626fa95d0fd72775d02a88
X-Envoy-Attempt-Count: 1
X-Envoy-External-Address: 10.99.41.9
X-Forwarded-For: 10.0.14.83,10.99.41.9, 127.0.0.6, 127.0.0.6
X-Forwarded-Proto: http
X-Request-Id: 35e5a6f4-140b-4802-8a5a-42db031c56b5


127.0.0.1 - - [07/Feb/2024:17:04:00 +0000] 'GET /test HTTP/1.1' 200 0 '-' 'curl/7.81.0' in 18072µs

The ip address 10.99.41.9 belongs to the istio-ingressgateway pod.

However, when I enable proxy protocol, it doesn't reach the knative service and the logs from ingress gateway shows the 503 error but also that it is detecting the client ip (XXX.XXX.XXX.XXX).

{
  "response_code_details": "upstream_reset_before_response_started{connection_termination}",
  "bytes_sent": 95,
  "method": "GET",
  "upstream_transport_failure_reason": null,
  "path": "/test",
  "connection_termination_details": null,
  "upstream_service_time": null,
  "requested_server_name": null,
  "response_code": 503,
  "upstream_host": "10.99.139.99:8081",
  "duration": 1,
  "downstream_local_address": "10.0.48.78:80",
  "start_time": "2024-02-07T21:48:53.062Z",
  "upstream_local_address": "10.99.139.99:59898",
  "protocol": "HTTP/1.1",
  "x_forwarded_for": "XXX.XXX.XXX.XXX",
  "request_id": "58bb3760-f210-45dd-a9f4-26e47d246b7e",
  "downstream_remote_address": "XXX.XXX.XXX.XXX:65260",
  "route_name": null,
  "response_flags": "UC",
  "authority": "backend.test-service.svc.cluster.local",
  "user_agent": "curl/7.81.0",
  "bytes_received": 0,
  "upstream_cluster": "outbound|80||knative-local-gateway.istio-system.svc.cluster.local",
}

I have also applied the EnvoyFilter, but as soon as I apply it both services (httpbin and knative) stop working.

The last thing I have tried was to explicitly use the suggested annotation (I've looked at Istio documentation), but it doesn't make a difference.

[skonto]

I suggest you try first to see if the old guide here still works (it uses this part of Istio config).
That guide relies on setting an envoy filter plus envoy's configuration here. The XFF stuff btw are quite complex but docs guide you of what to expect (Istio relies on envoy for header setting).

When you run the simple curl above you should get your machine's ip not the ingress one. Could you verify that in the istio ingress logs you see your remote client ip by default (the envoy filter affects ingress <-> envoy sidecar communications afaik)?

Also could you compare with mesh vs non-mesh mode (no sidecar injection).

If you face any issues with the guide above pls create a ticket at the Istio side to discuss further. When this works we can check again the Knative stuff. Also feel free to start a discussion on Knative slack channel so you can get more visibility and others can chime in.

I suspect your goal is to get the remote client's ip at the service side and not use the Proxy protocol end-to-end as described in here.

cc @dprotaso he may have more background info on EKS and Knative intgeration.

[jdiazper]

Hi @skonto ,

Thanks for all the links and guidence provided. I have been trying to make it work, but unfortunatelly I still keep receiveing that upstream connect error or disconnect/reset before headers. reset reason: connection termination.

@dprotaso welcome to this issue!

Based on your experience, do you know if I am missing anything in my installation/configuration that makes the gateway to return that error?

As always, thanks for your help and support!

[jdiazper]

Hi @dprotaso ,

Is there any chance you can look at this issue? I would really appreciate any advice you could give me to get it resolved.

Thanks!