Conceptual problem: Pods still need to know device pool names about network-resources-injector

[pperiyasamy]

/cc @dougbtv @s1061123

[JanScheurich]

NB: For Mellanox VF PCI devices, the identification problem can be solved as these are always bound to their kernel driver and visible as network devices in the pod namespace, even when used as DPDK interfaces. A simple call to "ethtool -i" reveals the PCI address:

testpmd0:~ # ethtool -i net1
driver: mlx5_core
version: 4.7-3.2.9
firmware-version: 16.26.4012 (MT_0000000012)
expansion-rom-version:
bus-info: 0000:18:00.2

But Intel SRIOV VFs configured for DPDK use are bound to vfio_pci or similar driver and not visible as netdevs. Today containers must rely on the environment variables provided by the device plugin to identify the assigned PCI devices.

[zshi-redhat]

I don't know if there is any easy way for Multus to update env var for container/pod.
Another approach might be to have Multus update the mapping between NAD and DeviceID in pod network-status annotation and use downward API volume to map pod annotation to container/pod as file or env variable. Application can parse the mapped file or env variable to get the right info.

[JanScheurich]

@zshi-redhat: Could Multus still add pod annotations when executing? Through the kubeclient? And yes, would such updated pod annotations be visible to the containers through the downward API volume?

[zshi-redhat]

@zshi-redhat: Could Multus still add pod annotations when executing? Through the kubeclient? And yes, would such updated pod annotations be visible to the containers through the downward API volume?

Yes, that's how Multus publishes network-status to pod annotation, through kubeclient.
If downward API volume is injected using network-resources-injecotr, then pod annotation will be continuously updated to container via downward API.

[pperiyasamy]

@zshi-redhat: Could Multus still add pod annotations when executing? Through the kubeclient? And yes, would such updated pod annotations be visible to the containers through the downward API volume?

Yes, that's how Multus publishes network-status to pod annotation, through kubeclient.
If downward API volume is injected using network-resources-injecotr, then pod annotation will be continuously updated to container via downward API.

@zshi-redhat Thanks! Let's propose a PR in multus-cni updating pod annotations for network name to device id(s).

[JanScheurich]

We looked at the existing network-status annotation. The cleanest way would in our opinion be to add an optional device_id field to the JSON map that Multus could populate if a net-attachment is associated with a PCI device. Is that possible?

We suspect that the network status struct may be governed by the K8sNWPGs type definition (https://github.com/k8snetworkplumbingwg/network-attachment-definition-client/blob/0ee521d560613bfe2d2c775159f93c2413a39db2/pkg/apis/k8s.cni.cncf.io/v1/types.go#L43) and it may be difficult to add a field there. Is that the case?

[akalambu]

Would this patch only update the network status with DeviceID only when explicitly asked for it from NAD?
Because in case of fail over the PCI address would change. So wondering if i have a NAD for SRIOV/DPDK and dont specify the DeviceID can i use these patches to gather the DeviceID to network name mapping

[pperiyasamy]

The spec proposal for adding device id into network status pod annotation is underway in this document . The implementation will follow across components like network-attachment-definition-client, sriov-network-device-plugin, multus-cni and plugins.

once that's available, pod container can see device id assigned to a network through DownwardAPI mount point on network status annotation (/etc/podnetinfo/annotations directory). NRI already does it.

@JanScheurich can we close this issue ?

[JanScheurich]

@pperiyasamy Yes, I agree to close this bug with reference to systemized solution agreed in NWPWG.