Unable to reproduce on local cluster about cilium-perf-networking

[prateeksahu]

Hey @pchaigno, thanks for all the tips and directions.
There was a rogue process on the master which was hogging up the port on which vxlan was supposed to run. Upon reboot and cleaning up the firewall rules, the experiments ran smoothly.

[prateeksahu]

I tried out the direct-routing mode and that successfully ran the kubenetbench benchmark. It seems tunnelling/vxlan setting is causing some issue with pod2pod communication.
Any insights in this regard will be helpful.

[pchaigno]

Could you share a Cilium sysdump? Did you check for pakcet drops reported by Cilium? Did you try to trace the failing connection with Cilium and/or tcpdump?

[prateeksahu]

Hi @pchaigno, I have attached the sysdump here. I did find some failed tasks while generating the sysdump mostly pertaining to hubble, but that was expected since I am not running hubble while doing the perf testing.
I did find one failed task that I am not completely certain: [10] Collecting Cilium egress NAT policies: failed to collect Cilium egress NAT policies: the server could not find the requested resource (get ciliumegressnatpolicies.cilium.io)

I will check tcpdump by running the server and
client independently and report back. Unfortunately, I raised the issue at a bad time since I am travelling for about a week, but will update asap.
cilium-sysdump-20220215-171559.zip

[pchaigno]

It looks like both nodes can't reach pods on the other node. That traffic should go through the tunnel, so most likely explanation is that the encapsulated traffic is dropped. Maybe you need to open a hole in some firewall to allow the VXLAN traffic through?

[prateeksahu]

@pchaigno. Apologies for the delay.
I see that VXLAN port 8472 is open on both the nodes

master
sudo netstat -a -n | grep 8472                                                                                                                                                                        12:35:38
udp        0      0 0.0.0.0:8472            0.0.0.0:*

node
sudo netstat -a -n | grep 8472                                                                                                                                     
udp        0      0 0.0.0.0:8472            0.0.0.0:*                          
udp6       0      0 :::8472                 :::*

While trying to do a tcpdump, I did notice the 'cilium_vxlan' interface was not up on the master.
master:

sudo tcpdump -D                                                                                                                                                                                       12:34:07
1.cilium_net [Up, Running]
2.cilium_host [Up, Running]
3.enp2s0 [Up, Running]
4.flannel.1 [Up, Running]
5.lo [Up, Running, Loopback]
6.any (Pseudo-device that captures on all interfaces) [Up, Running]
7.virbr0 [Up]
8.docker0 [Up]
9.virbr1 [Up]
10.virbr2 [Up]
11.bluetooth-monitor (Bluetooth Linux Monitor) [none]
12.nflog (Linux netfilter log (NFLOG) interface) [none]
13.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
14.virbr1-nic [none]
15.virbr2-nic [none]
16.virbr0-nic [none]
17.cilium_vxlan [none]

the same is up and running on the worker node

sudo tcpdump -D                                                                                                                                                    Thu 02/24 12:35 PM CST
[sudo] password for prateek: 
1.cilium_net [Up, Running]
2.cilium_host [Up, Running]
3.cilium_vxlan [Up, Running]
4.enp5s0 [Up, Running]
5.lo [Up, Running, Loopback]
6.any (Pseudo-device that captures on all interfaces) [Up, Running]
7.enp3s0 [Up]
8.virbr0 [Up]
9.docker0 [Up]
10.bluetooth-monitor (Bluetooth Linux Monitor) [none]
11.nflog (Linux netfilter log (NFLOG) interface) [none]
12.nfqueue (Linux netfilter queue (NFQUEUE) interface) [none]
13.virbr0-nic [none]

Could you provide instructions on debugging why the interface is not showing up?

[pchaigno]

Did you check the Cilium agents and dmesg on the node for errors?