Not able to create containers when using 4.13.4 microshift bundle about crc

[praveenkumar]

Looking at the node description, looks like no cni plugin. will check with microshift eng.

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 27 Jul 2023 13:58:56 +0530   Thu, 27 Jul 2023 13:48:56 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 27 Jul 2023 13:58:56 +0530   Thu, 27 Jul 2023 13:48:56 +0530   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 27 Jul 2023 13:58:56 +0530   Thu, 27 Jul 2023 13:48:56 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Thu, 27 Jul 2023 13:58:56 +0530   Thu, 27 Jul 2023 13:48:56 +0530   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/cni/net.d/. Has your network provider started?

[praveenkumar]

Following service is failing.

$ systemctl status ovsdb-server.service 
× ovsdb-server.service - Open vSwitch Database Unit
     Loaded: loaded (/usr/lib/systemd/system/ovsdb-server.service; static)
    Drop-In: /etc/systemd/system/ovsdb-server.service.d
             └─microshift-cpuaffinity.conf
     Active: failed (Result: exit-code) since Thu 2023-07-27 05:34:17 EDT; 28s ago
    Process: 3436 ExecStartPre=/usr/bin/rm -f /run/openvswitch.useropts (code=exited, status=0/SUCCESS)
    Process: 3437 ExecStartPre=/usr/bin/chown ${OVS_USER_ID} /run/openvswitch /var/log/openvswitch (code=exited, status=0/SUCCESS)
    Process: 3438 ExecStartPre=/bin/sh -c /usr/bin/echo "OVS_USER_ID=${OVS_USER_ID}" > /run/openvswitch.useropts (code=exited, status=0/SUCCESS)
    Process: 3440 ExecStartPre=/bin/sh -c if [ "$${OVS_USER_ID/:*/}" != "root" ]; then /usr/bin/echo "OVS_USER_OPT=--ovs-user=${OVS_USER_ID}" >> /run/openvswitch.useropts; fi (code=exited, status=0/SUCCESS)
    Process: 3442 ExecStart=/usr/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd --no-monitor --system-id=random ${OVS_USER_OPT} start $OPTIONS (code=exited, status=1/FAILURE)
        CPU: 64ms

Jul 27 05:34:17 api.crc.testing systemd[1]: ovsdb-server.service: Control process exited, code=exited, status=1/FAILURE
Jul 27 05:34:17 api.crc.testing systemd[1]: ovsdb-server.service: Failed with result 'exit-code'.
Jul 27 05:34:17 api.crc.testing systemd[1]: Failed to start Open vSwitch Database Unit.
Jul 27 05:34:17 api.crc.testing systemd[1]: ovsdb-server.service: Scheduled restart job, restart counter is at 4.
Jul 27 05:34:17 api.crc.testing systemd[1]: Stopped Open vSwitch Database Unit.
Jul 27 05:34:17 api.crc.testing systemd[1]: ovsdb-server.service: Start request repeated too quickly.
Jul 27 05:34:17 api.crc.testing systemd[1]: ovsdb-server.service: Failed with result 'exit-code'.
Jul 27 05:34:17 api.crc.testing systemd[1]: Failed to start Open vSwitch Database Unit.

Journalctl logs

Jul 27 05:36:28 api.crc.testing systemd[1]: ovsdb-server.service: Scheduled restart job, restart counter is at 4.
Jul 27 05:36:28 api.crc.testing systemd[1]: Stopped Open vSwitch Database Unit.
Jul 27 05:36:28 api.crc.testing systemd[1]: Starting Open vSwitch Database Unit...
Jul 27 05:36:29 api.crc.testing ovsdb-server[4771]: ovs|00001|daemon_unix|EMER|(null): Invalid --user option openvswitch:hugetlbfs (user openvswitch is not in group hugetlbfs), aborting.
Jul 27 05:36:29 api.crc.testing ovs-ctl[4771]: ovsdb-server: (null): Invalid --user option openvswitch:hugetlbfs (user openvswitch is not in group hugetlbfs), aborting.
Jul 27 05:36:29 api.crc.testing ovs-ctl[4724]: Starting ovsdb-server ... failed!
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Control process exited, code=exited, status=1/FAILURE
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Failed with result 'exit-code'.
Jul 27 05:36:29 api.crc.testing systemd[1]: Failed to start Open vSwitch Database Unit.
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Scheduled restart job, restart counter is at 5.
Jul 27 05:36:29 api.crc.testing systemd[1]: Stopped Open vSwitch Database Unit.
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Start request repeated too quickly.
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Failed with result 'exit-code'.
Jul 27 05:36:29 api.crc.testing systemd[1]: Failed to start Open vSwitch Database Unit.
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Start request repeated too quickly.
Jul 27 05:36:29 api.crc.testing systemd[1]: ovsdb-server.service: Failed with result 'exit-code'.
Jul 27 05:36:29 api.crc.testing systemd[1]: Failed to start Open vSwitch Database Unit.
lines 411-463/463 (END)

[praveenkumar]

For 4.13.3 where it all worked have

$ ssh crc -- cat /usr/lib/group | grep open
Warning: Permanently added '192.168.130.11' (ED25519) to the list of known hosts.
openvswitch:x:992:
hugetlbfs:x:1000:openvswitch

but for 4.13.4 have

$ ssh crc -- cat /usr/lib/group | grep open
Warning: Permanently added '192.168.130.11' (ED25519) to the list of known hosts.
openvswitch:x:992:

[praveenkumar]

So https://issues.redhat.com/browse/OCPBUGS-15948 is actual issue and looks like it is with openvswitch side and for microshift they recently pinned the version to working one openshift/microshift@8ba6e57

[cfergeau]

Workaround is openshift/os#1318

[praveenkumar]

We don't need workaround in snc or crc side. microshift already taken that into account and I created 4.13.6 bundles which works as expected now. So I am closing this issue.

[gbraad]

@praveenkumar so this was fixed before, but not in the release we used?