[LinuxOnly] is egregiously misused at least in sig-network tests about kubernetes

[sebsoto]

/assign @sebsoto

[knabben]

/triage accepted

[claudiubelu]

Hello,

Indeed, there are quite a few tests which are marked as [LinuxOnly], and since then there were some tests which were fixed and / or had this label removed over time, not necessarely all networking-related [1][2][3][4][5][6] (may be others as well). But out of those 45 tests you mentioned, not all of them are marked as [Conformance], or at least as far as I can tell, I see only 8:

ubuntu@ubuntu:~/workdir/kubernetes$ yq '.[] | .codename | select(contains("LinuxOnly")) | select(contains("network"))' test/conformance/testdata/conformance.yaml
[sig-network] DNS should resolve DNS of partial qualified names for services [LinuxOnly] [Conformance]
[sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [LinuxOnly] [NodeConformance] [Conformance]
[sig-network] Networking Granular Checks: Pods should function for node-pod communication: udp [LinuxOnly] [NodeConformance] [Conformance]
[sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Conformance]
[sig-network] Services should have session affinity work for NodePort service [LinuxOnly] [Conformance]
[sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [Conformance]

Also worth mentioning that there may be more tests which are being skipped on Windows through something like SkipIfNodeOSDistroIs("windows"), I'm not sure if they're included in your count or not. I remember that we were not allowed to add skips to [Conformance] tests, which is why that label was necessary in the first place. This label was not typically added to non-conformance tests, but IMO, it's more useful to have it as a label; it makes it easier to grep which tests are supposed to be Linux-only or not. As for the number of Linux-only networking tests, a more complete list of tests could be useful, so we can check whether or not they're mislabeled or not, or which tests exactly we're talking about.

Also note that non-Conformance tests are not typically tested as the Conformance ones, so I can't say much about them passing or not. There were some efforts in the past for promoting more networking tests to Conformance [7], but that effort seems discontinued at this point.

Now, regarding the tests you've mentioned:

• SessionAffinity: Probably requires more in-depth investigation. But from a quick look, it doesn't work [8] (marked all networking tests as [Conformance] so they'd be run by the Windows CI):

Kubernetes e2e suite: [It] [sig-network] Services should have session affinity work for NodePort service [Conformance] 2m47s
{ failed [FAILED] Affinity should hold but didn't.
In [It] at: k8s.io/kubernetes/test/e2e/network/service.go:266 @ 04/22/24 19:31:21.999
}

Kubernetes e2e suite: [It] [sig-network] Networking Granular Checks: Services should function for client IP based session affinity: udp [Conformance]   1m6s
{ failed [FAILED] Unexpected endpoints return: map[netserver-0:{} netserver-1:{}], expect 1 endpoints
In [It] at: k8s.io/kubernetes/test/e2e/network/networking.go:447 @ 04/22/24 19:34:02.49
}

• dual-stack: Indeed, according to James [9], these tests could be enabled for WS 2022 jobs but not for WS 2019 jobs. I assume there should be a dedicated dual-stack WS 2022 job for this, which would also require e2e to receive a dual-stack-related config flag. From what I can see, they're labeled as [Feature:IPv6DualStack], so we can use that label to exclude it from other jobs.

• UDP support in agnhost: not sure which tests are those. should function for node-pod communication: udp?

• hostNetwork tests: agreed. Though, I thought there were more than 2 tests.

• hostPorts test seems to be failing on Windows [7], may require further investigation:

Kubernetes e2e suite: [It] [sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance]   45s
{ failed [FAILED] Failed to connect to exposed host ports
In [It] at: k8s.io/kubernetes/test/e2e/network/hostport.go:161 @ 04/22/24 19:29:22.484
}

• IPv6 test: I assume you refer to should provide Internet connection for containers? It might require a bit more investigation. It seems that there's a failure [10]: forward host lookup failed: h_errno 11001: HOST_NOT_FOUND

• SCTP: indeed, it seems that Windows doesn't have it [11]. But there seem to be some 3rd party implementations. WDYT, should we consider one of those and say it's officially supported on Kubernetes for Windows?

• partial DNS names: yes, that is correct. You may either use only the FQDNs, or only the hostname part.

[1] #101063
[2] #72729
[3] #97045
[4] #85453
[5] #78731
[6] #75591

[7] #73425
[8] https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124447/pull-kubernetes-e2e-capz-windows-master/1782468233807269888
[9] #100870 (comment)
[10] https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124447/pull-kubernetes-e2e-capz-windows-master/1782725964745150464
[11] https://learn.microsoft.com/en-us/answers/questions/778329/sctp-driver

[danwinship]

But out of those 45 tests you mentioned, not all of them are marked as [Conformance]

Sure, but they all involve documented features that users might want to use. And as you said, we do eventually want to move more tests toward Conformance, so Windows really ought to be aiming to pass everything that isn't truly linux-specific.

Also worth mentioning that there may be more tests which are being skipped on Windows through something like SkipIfNodeOSDistroIs("windows"), I'm not sure if they're included in your count or not.

No... I was only looking at [LinuxOnly].

• UDP support in agnhost: not sure which tests are those. should function for node-pod communication: udp?

No, sorry, I should have listed these out. The NetworkPolicy between server and client using UDP tests in test/e2e/network/netpol/network_policy.go. (In fact, they are both [LinuxOnly] and SkipIfNodeOSDistroIs("windows")!)

• hostNetwork tests: agreed. Though, I thought there were more than 2 tests.

Perhaps the others are skipped rather than labeled.

• IPv6 test: I assume you refer to should provide Internet connection for containers?

Yes

• SCTP: indeed, it seems that Windows doesn't have it [11]. But there seem to be some 3rd party implementations. WDYT, should we consider one of those and say it's officially supported on Kubernetes for Windows?

SCTP isn't even implemented by most Linux network plugins and it seems like sig-windows already has too much to do anyway, so it seems reasonable to just say that it's not supported on Windows. If someone wants to get it to be officially-supported then they can do the work...

[claudiubelu]

Ran the [LinuxOnly] tests in a few jobs [1][2]. Here are some results:

• should test kubelet managed /etc/hosts file - containerd supports single file mappings in containers, so this should work. Sent a PR fixing the test for Windows here: #124852

• validates that there is no conflict between pods with same hostPort but different hostIP and protocol: Failed to connect to exposed host ports:

I0513 21:12:00.371224 129567 exec_util.go:83] ExecWithOptions: execute(POST https://capz-conf-hk7naa-c91daf83.canadacentral.cloudapp.azure.com:6443/api/v1/namespaces/hostport-7358/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+--interface+10.1.0.5+http%3A%2F%2F127.0.0.1%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
I0513 21:12:00.689775 129567 hostport.go:129] Can not connect from e2e-host-exec to pod(pod1) to serverIP: 127.0.0.1, port: 54323

• Network Policy tests (should ensure an IP overlapping both IPBlock.CIDR and IPBlock.Except is allowed [Feature:NetworkPolicy], should allow egress access on one named port [Feature:NetworkPolicy], should allow ingress access on one named port [Feature:NetworkPolicy]:

I0426 11:41:47.163106 82743 probe.go:104] Expected allowed pod connection was instead BLOCKED --- run 'kubectl exec a -c cont-80-tcp -n netpol-y-4052 -- /agnhost connect 10.96.53.156:80 --timeout=3s --protocol=tcp'
...
I0426 11:53:05.731060 82747 reachability.go:178] reachability: correct:48, incorrect:24, result=false

• should function for client IP based session affinity: udp, should function for client IP based session affinity: http: Unexpected endpoints return: map[netserver-0:{} netserver-1:{}], expect 1 endpoints:

I0426 11:10:47.846110 82747 utils.go:372] Tries: 10, in try: 0, stdout: {"responses":["netserver-0"]}, stderr: , command run in Pod { "name: test-container-pod, namespace: nettest-8008, hostIp: 10.1.0.4, podIp: 192.168.9.52, conditions: [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:44 +0000 UTC  } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:40 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:44 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:44 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:40 +0000 UTC  }]" }
I0426 11:11:02.246169 82747 utils.go:372] Tries: 10, in try: 6, stdout: {"responses":["netserver-1"]}, stderr: , command run in Pod { "name: test-container-pod, namespace: nettest-8008, hostIp: 10.1.0.4, podIp: 192.168.9.52, conditions: [{PodReadyToStartContainers True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:44 +0000 UTC  } {Initialized True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:40 +0000 UTC  } {Ready True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:44 +0000 UTC  } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:44 +0000 UTC  } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2024-04-26 11:10:40 +0000 UTC  }]" }

• should have session affinity work for service with type clusterIP, should have session affinity timeout work for NodePort service, should be able to switch session affinity for service with type clusterIP: Affinity should hold but didn't.

• should fail health check node port if there are only terminating endpoints: curl timeouts:

I0426 12:31:54.822266 82743 service.go:2756] unexpected error trying to connect to nodeport 10.1.0.4:30904 : error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/k8s.io/windows-testing/capz/capz-conf-7w0a3a.kubeconfig --namespace=services-2600 exec pause-pod-0 -- /bin/sh -x -c curl -s -o /dev/null -w "%{http_code}" --max-time 5 http://10.1.0.4:30904/healthz:
Command stdout:
000
stderr:
+ curl -s -o /dev/null -w '%{http_code}' --max-time 5 http://10.1.0.4:30904/healthz
command terminated with exit code 28

• internalTrafficPolicy and externalTrafficPolicy tests: curl exit code 7 (couldn't connect to host or proxy):

I0426 11:19:10.429603 82745 util.go:166] got err: error running /usr/local/bin/kubectl --kubeconfig=/home/prow/go/src/k8s.io/windows-testing/capz/capz-conf-7w0a3a.kubeconfig --namespace=services-7252 exec pause-pod-0 -- /bin/sh -x -c curl -q -s --max-time 30 10.107.35.46:80/hostname:
Command stdout:

stderr:
+ curl -q -s --max-time 30 10.107.35.46:80/hostname
command terminated with exit code 7

error:
exit status 7, retry until timeout

[1] https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124447/pull-kubernetes-e2e-capz-windows-master/1783804706359873536
[2] https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/124447/pull-kubernetes-e2e-capz-windows-master/1790109519284539392

[claudiubelu]

Also worth mentioning that there may be more tests which are being skipped on Windows through something like SkipIfNodeOSDistroIs("windows"), I'm not sure if they're included in your count or not.

No... I was only looking at [LinuxOnly].

FWIW, there aren't many SkipIfNodeOSDistroIs("windows") tests, most of them are storage-related (requring RunAsUser or fs group), or sysctl-related. Most of these tests are also labeled with [LinuxOnly] as well.

[aravindhp]

/cc @sbangari

[danwinship]

• Network Policy tests (should ensure an IP overlapping both IPBlock.CIDR and IPBlock.Except is allowed [Feature:NetworkPolicy]

That test was specifically added because multiple implementations got the behavior wrong, so this probably points to a bug in your NetworkPolicy implementation

should allow egress access on one named port [Feature:NetworkPolicy], should allow ingress access on one named port [Feature:NetworkPolicy]

And named ports are a slightly-obscure feature that lots of people skip in their initial NetworkPolicy implementation (and then sometimes never get back to).

• should fail health check node port if there are only terminating endpoints: curl timeouts:

The actual behavior being tested here is entirely in the platform-independent part of kube-proxy, so this is likely a bug/Linux-ism in the e2e test. (Possibly a bad assumption about pod-to-node connections? I always forget what is and isn't allowed.)

• internalTrafficPolicy and externalTrafficPolicy tests: curl exit code 7 (couldn't connect to host or proxy):

Possibly the same thing, but also, winkernel has its own implementation of figuring out which endpoints to use rather than using proxy.CategorizeEndpoints so it may be screwing up some edge cases