knative / networking

knative/networking

This repository contains the Knative Ingress and Certificate CRDs, as well as their conformance tests. These are our extension points to plugin different Ingress plugins (Ambassador, Contour, Gloo, Istio, Kong and Kourier), as well as different ExternalDomainTLS plugins (CertManager and Knative's own HTTP01 challenge solver).

Knative Ingress aka KIngress

The Knative Ingress CRD is based on the Kubernetes Ingress CRD, with the following additions:

1. Traffic splitting
2. Header modification applied to requests based on the traffic split they are assigned to.
3. Host header rewrite
4. Traffic redirection predicates based on a regexp based condition on headers.

In addition to these, we previously added Timeout and Retry settings but no longer used them and may deprecate these parts in future versions.

See https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/#install-a-networking-layer for a list of the current supported KIngress implementations.

Check out:

• pkg/apis/networking/v1alpha1/ingress_types.go for more information about the KIngress API spec.
• pkg/apis/networking/v1alpha1/ingress_validation.go for more information about the validation logic for KIngress API spec.
• test/conformance/ingress/README.md for the conformance tests and how to run them.

See also:

• http://github.com/knative-extensions/net-contour for the Contour-based implementation of KIngress.
• http://github.com/knative-extensions/net-kourier for a dependency-free implementation of KIngress based on Envoy proxy.
• http://github.com/knative-extensions/net-istio For the Istio-based implementation of KIngress.

Knative Certificate aka KCert

Knative Certificate CRD is a Knative abstraction for various SSL certificate provisioning solutions (such as cert-manager or self-signed SSL certificate).

Check out:

• pkg/apis/networking/v1alpha1/certificate_types.go for more information about the Certificate API spec.
• pkg/apis/networking/v1alpha1/certificate_validation.go for more information about the validation logic for Certificate API spec.

Contributing

If you would like to contribute to Knative, take a look at CLOTRIBUTOR for a list of help wanted issues across the project.

networking's Issues

Prober should ensure that DNS works as expected

When service to service test is running, istio-ingressgateway on Kind returns 502 with no such host error.
We added retry checker by knative/pkg#2351 but prober should ensure that DNS works as expected.

xref: knative/pkg#2351

"--enable-alpha" option must implement all alpha features

When we use --enable-alpha flag, all alpha tests are included.
For example in the current alpha tests:

networking/test/conformance/ingress/run.go

Lines 71 to 75 in 6ae0352

	if test.ServingFlags.EnableAlphaFeatures {
	// Add your conformance test for alpha features
	t.Run("headers/tags", TestTagHeaders)
	t.Run("host-rewrite", TestRewriteHost)
	}

we want to run TestTagHeaders but we did not implement TestRewriteHost yet. In this case, we need to give up both tests to run.

Add a flag to control future trust support

Current tls implementation support internal-encryption without trust
We need to have a roadmap for one with mutual trust which seems achievable using a number of future steps.

Add flag for InternalTrust bool in knative.dev/networking/pkg/config/config.go so we can start alpha it.

/assign davidhadas

Visibility test (proxy image) fails with KinD 0.12.0 + k8s 1.23.x due to DNS issue

When trying to upgrade KinD to 0.12.0, k8s 1.23.x always fails with the following error:

https://github.com/knative-sandbox/net-contour/actions/runs/2164723487
https://github.com/knative-sandbox/net-kourier/actions/runs/2164695709

              lookup ingress-conformance-visibility-path-ijgjgogt.serving-tests.svc.c2164695709.local on 10.96.0.10:53: server misbehaving

For now, the issue happens as:

• All k8s 1.23.x fails with KinD 0.12.0.
• All net-* repo fails with k8s 1.23.x + KinD 0.12.0.
• All k8s 1.23.x does NOT fail with KinD 0.11.1.

Autotls issues with venafi

I tried to set up auto tls with venafi tpp instead of let encrypt and acme. I get the below error

'Failed to request venafi certificate: Certificate requests submitted to Venafi issuers must have the ''commonName'' field or at least one other subject field set.'

I checked the csr, it just had the SAN. For venafi looks like CN is required, please refer to this issue.

Is there anyway we can include common name in the certificate resource.

Deprecate ingress.status.LoadBalancer

There are three LB status in ingress.status, LoadBalancer, PublicLoadBalancer and PrivateLoadBalancer.

networking/pkg/apis/networking/v1alpha1/ingress_types.go

Lines 304 to 315 in f8dacf7

	// LoadBalancer contains the current status of the load-balancer.
	// This is to be superseded by the combination of `PublicLoadBalancer` and `PrivateLoadBalancer`
	// +optional
	LoadBalancer *LoadBalancerStatus `json:"loadBalancer,omitempty"`
	// PublicLoadBalancer contains the current status of the load-balancer.
	// +optional
	PublicLoadBalancer *LoadBalancerStatus `json:"publicLoadBalancer,omitempty"`
	// PrivateLoadBalancer contains the current status of the load-balancer.
	// +optional
	PrivateLoadBalancer *LoadBalancerStatus `json:"privateLoadBalancer,omitempty"`

PublicLoadBalancer and PrivateLoadBalancer make sense but LoadBalancer does not in the status field.

some research in the current KIngress implementations

net-istio and net-kourier

• external LB or internal LB is assigned in the LoadBalancer based on ingress.Spec.Visibility. But ingress.Spec.Visibility was deprecated.

net-contour

• net-contour does not use it and always set LoadBalancer to nil.

Deprecate retry

I believe that route has stopped specifying this, but we still have tests for it which run a pretty sizable matrix of now unexercised functionality.

Consider deprecating the field, and removing the conformance tests.

/assign @tcnghia
/kind cleanup

Conformance tests should not hard code .svc.cluster.local

There are a few instances

networking/pkg/ingress/ingress_test.go

Line 35 in 8af68c7

hosts: sets.NewString("service.name-space.svc.cluster.local"),

networking/test/conformance/ingress/visibility.go

Line 46 in f1e9928

"fqdn": shortName + ".svc.cluster.local",

networking/test/test_images/httpproxy/service.yaml

Line 13 in 4b26211

value: "helloworld-test-image.default.svc.cluster.local"

[HTTP Option support] Support HTTP option for Kourier

After the HTTP option is added into Kingress spec, we can start supporting it within Kourier.

Support post-split RewriteHost

In order to target ExternalName services that rely on Host-based dispatch (the way we do for ksvc), we essentially need to support rewriting every service reference to it's internal hostname in order for it to correctly receive traffic.

We can work around this today for the trivial vanity domain case because we can put the RewriteHost pre-split, since there's a single split, however, if we wanted to correctly traffic split over two ksvc with kingress, then we could not do that with the kingress we have today.

cc @julz @tcnghia

Related: #107
This would need some extension: knative-extensions/net-contour#220

[HTTP Option support] Deprecate `httpProtocol: "Disabled"` in config-network

As per design doc https://docs.google.com/document/d/1yVOGo-j1hI7oCpuWVIi5iBJwX1wKbOzWaV0QNJxCbaY/edit#

Note that we will not support Disabled at least in the beginning because it may break HTTP01 challenge of auto TLS feature.

It has a plan to make "Disabled" unsupported.

Also, Disable should be possible by stop exposing :80.

Add a negative test for ingress.class handling

/kind good-first-issue

If ingress class is omitted or the incorrect value, then the kingress should NOT pick it up.

We should have a conformance test for this.

Drop deprecated generation field

ie.

networking/pkg/apis/networking/v1alpha1/ingress_types.go

Lines 91 to 100 in 52c85a3

	// DeprecatedGeneration was used prior in Kubernetes versions <1.11
	// when metadata.generation was not being incremented by the api server
	//
	// This property will be dropped in future Knative releases and should
	// not be used - use metadata.generation
	//
	// Tracking issue: https://github.com/knative/serving/issues/643
	//
	// +optional
	DeprecatedGeneration int64 `json:"generation,omitempty"`

[HTTP Option support] Support HTTP option in net-contour controller

After HTTP option is added into Kingress spec, we can start the implementation of it within net-contour controller in order to integrate with Contour.

Knative istio client ip is not working

Hi,
I am using istio in knative. Since the client ip did not come, I activated the reverse-prox. I added envoyfilter. but I get "upstream connect error or disconnect/reset before headers. reset reason: connection termination" error in services on knative.
if it's not knative service I can access it.

virtualservice

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: test-entry-route
namespace: default
spec:
gateways:
- knative-serving/knative-ingress-gateway
hosts:
- "test.net"
http:
- match:
- uri:
prefix: /test/ #working
rewrite:
uri: /
route:
- destination:
host: user-api.test.svc.cluster.local
port:
number: 80
- match:
- uri:
prefix: /user/ #not working
route:
- destination:
host: istio-ingressgateway.istio-system.svc.cluster.local
port:
number: 80
weight: 100
rewrite:
authority: user-api.poker-test.k8s.test.net
uri: /

EnvoyFilter

---

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: proxy-protocol
namespace: istio-system
spec:
configPatches:
- applyTo: LISTENER
patch:
operation: MERGE
value:
listener_filters:
- name: envoy.listener.proxy_protocol
- name: envoy.listener.tls_inspector
workloadSelector:
labels:
istio: ingressgateway

service load balancer

---

annotations:
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: 'true'
externalTrafficPolicy: Local

Annotation validation for networking.knative.dev

/kind good-first-issue

Currently we validate that users can't create resources with annotations with the serving.knative.dev prefix in the validateKnativeAnnotations. We currently don't appear to have any such validation for the networking.knative.dev prefix.

I see two useful things to get out of this:

• We can create a file similar to annotation_validation in autoscaler, which performs validation on all the annotations owned by autoscaler, including validating the values of the annotations themselves.
• We could modify the validateKnativeAnnotations to validate all annotation prefixes owned by knative. I suspect this gap exists for other annotation groups as well.

Moved from knative/serving#9192

[HTTP Option support] Add conformance test for HTTP option

We need conformance test for HTTP option behavior to ensure the downstream implementations do the right thing.

Full design: https://docs.google.com/document/d/1yVOGo-j1hI7oCpuWVIi5iBJwX1wKbOzWaV0QNJxCbaY

[HTTP Option support] Support HTTP option in net-istio controller

After httpOption is added into Kingress spec, and is set by Route controller, net-istio controller could start reading the httpOption from the Kingress spec field instead of config-network ConfigMap directly.

Add Retry Checker for conformance test's request?

Currently conformance test for net-* is flaky due to some test failed with connection reset by peer or connection refused.

Example logs:

• net-istio
• net-kourier when it runs e2e in parallel
• net-contour may fix the flak by knative-extensions/net-contour#631 🤔

Serving repo has a response checker and so it does not stop e2e tests by the TCP errors.

https://github.com/knative/serving/blob/ab2fcc63551eecc408deef6e5f06b4defbd46e12/vendor/knative.dev/pkg/test/spoof/spoof.go#L167

So should networking conformance test also implement the retry checker which is same with Serving?

Ingress Conformance Tracking & Doc

/area networking

Our current documentation does not make clear how well a KIngress implementation integrates with Knative. We should establish:

1. clear criterias for being Alpha/Beta/Stable.
2. surface indicators in our Knative documentation to help users aware of how conformant different KIngress implementations are.

Proposal Doc: https://docs.google.com/document/d/1-oFais46gcPyaYYXiTxjWgok97hOtdVZlkbeSPoFaho/edit#

/assign

Bogus README.md

The root /README.md hasn't been updated since the fork from sample-controller, so it isn't appropriate to the repo.

[HTTP Option support] Add E2E test for the case of setting HTTPOption per ksvc

We will support the feature of configuring HTTPOption per ksvc knative/serving#11882

We need an E2E test for this use case after other net-* repo supports HTTPOption.

Test downstream integrations

See: knative/eventing#4260

cc @nak3 @ZhiminXiang @tcnghia

/kind good-first-issue

Document how to write a plugin that fulfils the KIngress spec

Describe the change you'd like to see

I received some feedback along the lines:

I think its still hard to understand integrating with [Knative]

Looking for some docs related to the internal pluggable types ie. KIngress I couldn't find any. We should aim to have some documentation around these types and maybe even some examples.

HTTP Authorization.

We are using Knative with Contour and would like to add authorization to the http requests to the knative services. Contour already has some support for this: https://projectcontour.io/docs/main/config/client-authorization/. This needs to be supported and exposed through net-contour.

config key `httpProtocol` (`http-protocol`) should be `httpOption` (`http-option`)?

Currently annotation for httpOption is networking.knative.dev/httpOption while global config is httpProtocol. It is same config but the name is slightly different.

If #522 will change the config name, it is a good time to change the name.

stream test pod logs back to the test

It would be useful to have the pod logs in our e2e tests. For example, the logs from the httpproxy pod in the visibility tests might be helpful for tracking down this

Intermittent 502s during "visibility/split"

I was originally tracking this in knative-extensions/net-contour#189, but have also seen this in net-istio and net-kourier.

They basically all fail on a DNS lookup (:53 is DNS):

    util.go:860: Error meeting response expectations: got unexpected status: 502, expected map[200:{}]
    util.go:860: HTTP/1.1 502 Bad Gateway
        Content-Length: 164
        Content-Type: text/plain; charset=utf-8
        Date: Wed, 12 Aug 2020 09:37:18 GMT
        Server: istio-envoy
        X-Content-Type-Options: nosniff
        X-Envoy-Upstream-Service-Time: 8
        
        dial tcp: lookup ingress-conformance-visibility-split-pxtddfbt.serving-tests.svc.cluster.local on 10.11.240.10:53: dial udp 10.11.240.10:53: operation was canceled

cc @tcnghia

[HTTP Option support] Add HTTPOption into Kingress spec

This is the first part of the httpOption support work. We need to add httpOption into Kingress spec.

Full design: https://docs.google.com/document/d/1yVOGo-j1hI7oCpuWVIi5iBJwX1wKbOzWaV0QNJxCbaY

net-* should have consistent naming

Going to follow the convention

net-[component]-[component_type] - ie. net-certmanager-webhook

• knative-extensions/net-certmanager#240
• knative-extensions/net-certmanager#241
• knative-extensions/net-http01#232
• knative-extensions/net-istio#683
  • knative-extensions/net-istio#693
  • knative-extensions/net-istio#686
• net-ingressv2 - no changes
• knative-extensions/net-kourier#555
• knative-extensions/net-contour#526

Downstream changes

• knative/operator#669
• knative/serving#11551
• knative/serving#11559
• knative/serving#11560

Adopt apps.kubernetes.io labels across networking repos

Context: knative/community#667
Serving PR: knative/serving#11617

[HTTP Option support] Allow setting HTTP option through annotation

Per the design, the Route controller could also consume a specific annotation on Route, and then set the corresponding annotation value to Kingress spec.

The specific annotation is TBD.

This issue is more suitable to be put under knative serving. But I put it here for easily tracking.

Avoid the Go iterator capture issue

Issue

When table test is running with t.Parallel() like below, the variable tt needs to be created by a new variable before t.Parallel(). Otherwise all tests use the same instance as https://golang.org/doc/faq#closures_and_goroutines.

        for _, tt := range tests {
                t.Run(tt.name, func(t *testing.T) {
                        t.Parallel()

Reproducer

For example, I verified that the issue by inserting a debug log as:

diff --git a/test/conformance/ingress/visibility.go b/test/conformance/ingress/visibility.go
index 77e4b36e..f7d17e34 100644
--- a/test/conformance/ingress/visibility.go
+++ b/test/conformance/ingress/visibility.go
@@ -363,6 +363,7 @@ func TestVisibilityPath(t *testing.T) {
        for path, want := range tests {
                t.Run(path, func(t *testing.T) {
                        t.Parallel()
+                       fmt.Printf("[debug] path %q, want %q\n", path, want)

                        ri := RuntimeRequest(ctx, t, client, "http://"+publicHostName+path)
                        if ri == nil {

Then, all tests in the table test uses the last variable /asdf only. (Please find [debug] path "/asdf", want "ingress-conformance-visibility-path-dvfrawou")

$ go test -v   -tags=e2e -count=1  ./test/conformance/...  -run "TestIngressConformance/visibility/path" --ingressendpoint=172.20.0.3  --ingressClass=kourier.ingress.networking.knative.dev --enable-beta
 ...
=== RUN   TestIngressConformance/visibility/path//foo
=== PAUSE TestIngressConformance/visibility/path//foo
=== RUN   TestIngressConformance/visibility/path//bar
=== PAUSE TestIngressConformance/visibility/path//bar
=== RUN   TestIngressConformance/visibility/path//baz
=== PAUSE TestIngressConformance/visibility/path//baz
=== RUN   TestIngressConformance/visibility/path/#00
=== PAUSE TestIngressConformance/visibility/path/#00
=== RUN   TestIngressConformance/visibility/path//asdf
=== PAUSE TestIngressConformance/visibility/path//asdf
=== CONT  TestIngressConformance/visibility/path//foo
[debug] path "/asdf", want "ingress-conformance-visibility-path-dvfrawou"
=== CONT  TestIngressConformance/visibility/path/#00
[debug] path "/asdf", want "ingress-conformance-visibility-path-dvfrawou"
=== CONT  TestIngressConformance/visibility/path//asdf
[debug] path "/asdf", want "ingress-conformance-visibility-path-dvfrawou"
=== CONT  TestIngressConformance/visibility/path//baz
[debug] path "/asdf", want "ingress-conformance-visibility-path-dvfrawou"
=== CONT  TestIngressConformance/visibility/path//bar
[debug] path "/asdf", want "ingress-conformance-visibility-path-dvfrawou"
=== CONT  TestIngressConformance/visibility/path

The test expects different five cases.

networking/test/conformance/ingress/visibility.go

Lines 355 to 361 in 8925a50

	tests := map[string]string{
	"/foo": fooName,
	"/bar": barName,
	"/baz": bazName,
	"": mainName,
	"/asdf": mainName,
	}

Document compatibility of knative/serving and knative/net-* versions

Since we follow the 6-month deprecation policy, any knative/serving version should work with a net-xxx version released 6 months ago, i.e. backward compatibility.

However, I am not sure we have any guarantee the other way around. Since we only get testing of knative/serving version 0.X.y with knative-sandbox/net-* version 0.X.z, the only forward compatibility is just with patch versions.

I'd love to see some confirmation from @nak3 @ZhiminXiang @mattmoor about this, following by some documentations.

Thanks @houshengbo who raised this very interesting question.

[HTTP Option support] HTTP Option for Kingress design

For the auto TLS feature, we want to support httpProtocol https://github.com/knative/networking/blob/master/config/config-network.yaml#L106.

Currently this feature is only supported by net-istio as net-istio directly consumes the value of config-network configmap. We should instead add the httpProtocol into Kingress spec so that the rest of kingress implementations could follow the conformance and support it.

Establish cherry-pick policy for net-* CRDs

@houshengbo brought it up that it would make things easier for the Knative operator to upgrade KIngress if patch releases don't change the CRDs (add/remove/rename).

Generally, these kinds of cherrypicks are riskier so we don't have many like that. However, it would be good to establish a guideline about cherrypicks to make things easier for the operator.

cc @nak3 @ZhiminXiang @mattmoor @dprotaso @houshengbo

conformance test for omitted timeout

cc @tcnghia @richterdavid

/kind good-first-issue
/kind cleanup

Add mechanism to override the scheme used for Knative routes

Currently, the ability to surface a https:// fronted URL in Knative is bound to the AutoTLS feature. AutoTLS however hooks itself directly into the ingress flow and might not be applicable in all situations. If you were to front the Knative Ingress with an external loadbalancer that's doing the termination for instance, the Route would still be available via HTTPS but we wouldn't enable AutoTLS to achieve that.

As such, I'd love to have a mechanism to override the scheme Knative thinks is appropriate. This could be as simple as adding overrideExternalScheme and overrideInternalScheme to config-network to allow to override the respective URLs published. There could be similarly named annotations as well to override per-service. Initially proposed here knative/serving#7708 (comment).

Require kingress to implement the dataplane contract

We should add a kingress conformance test that validates conformance to our dataplane contract.

Base Case

If we send:

K-Network-Probe: probe
K-Network-Hash: override

Then we should get back a response with K-Network-Hash: {hash reflecting config}.

Inductive Step

If we send:

K-Network-Probe: probe
K-Network-Hash: {hex}

Then we should get back a response with K-Network-Hash: {hex}.

---

This verifies that we can enable composition of kingress using ExternalName services.

Retry Get requests

Use the new reconciler.RetryTestErrors (see: #145) to wrap Get requests in kingress conformance.

I saw this on a recent net-contour test run:

percentage.go:47: Error waiting for Endpoints to contain a Pod IP: rpc error: code = Unavailable desc = etcdserver: leader changed

Change `NamespaceWildcardCertSelector` be of the type `labels.Selector` ?

nit: Any reason why we don't have cfg.Network.NamespaceWildcardCertSelector be of the type labels.Selector ?

Originally posted by @dprotaso in knative/serving#12174 (comment)

[HTTP Option support] Configure HTTP Option in Route controller from config-network ConfigMap

In Knative serving, the Route controller should read the HTTP option from the config-network ConfigMap, and configure Kingress spec accordingly.

Rename ServingFlags to NetworkingFlags

In here

https://github.com/knative/networking/blob/master/test/e2e_flags.go

We still refer to our flags as ServingFlags. Should be NetworkingFlags or just Flags.

Document/Test KIngress use of secrets

I'm wondering if we should have a conformance test to ensure that ingresses can't reach across namespaces to access secrets. The exception would be unless the operator has intervened - ie. with contour there's TLS delegation - https://projectcontour.io/docs/main/config/tls-delegation/

Context for this issue: knative/serving#11250 (comment)

Use cache.DeletedFinalStateUnknown in the status prober's handling of deletion event

In what area(s)?

/area networking

What version of Knative?

v 0.11

Expected Behavior

We use cache.DeletedFinalStateUnknown in the status prober's handling of deletion event

Actual Behavior

We do not.

Steps to Reproduce the Problem

Move network.yaml from serving repo to networking repo

As networking repo is specific to host networking resources, it makes sense to move the network.yaml from serving repo to networking repo.

/cc @tcnghia @nak3

Minor formatting issues

• unused parameter: 'req' - test/test_images/httpproxy

• Need to s/print/printf so the port variable can be printed in the right position

  • test/test_images/retry/main.go#L44

  • test/test_images/runtime/main.go#L523

  • test/test_images/timeout/timeout.go#L57

  • test/test_images/wsserver/echo.go#L91

Cleanup test flags naming

We have 3 kinds of flag names:

• resolvabledomain -> ResolvableDomain
• enable-alpha -> enableAlpha
• ingressClass -> ingressClass

Would be nice to use same style, so test writers don't have to be confused about which way is right.

Clean up DeprecatedDefaultIngressClassKey

clusteringress.class was deprecated for a long time ago, we should clean up DeprecatedDefaultIngressClassKey and related code like:

networking/pkg/network.go

Lines 68 to 69 in 6a77ca6

	// DeprecatedDefaultIngressClassKey Please use DefaultIngressClassKey instead.
	DeprecatedDefaultIngressClassKey = "clusteringress.class"

/kind good-first-issue

Engage with Ingress v2 on security controls

Serving could benefit from an AAA (authentication/identity, authorization, audit) capability on the HTTP data plane.

This is a tracking bug for the security WG.