How can we better handle feature control switches in NRI? about network-resources-injector

[pperiyasamy]

Looks config map would be a good fit for passing these config options like proposal done for issue #57, we could reuse the same config map with adding config.json as below:

apiVersion: v1
data:
  network-resource-injector-environment-variable: '{"spec":{"containers":[{"image":"docker.io/nfvpe/sriov-device-plugin","env":[{"name":"APP_ROLE","value":"client"}],"name":"application"}]}}'
  config.json: |
    {
	  "webhookconfig": {
	    "injectHugepageDownApi": true,
	    "networkResourceNameKeys": ["k8s.v1.cni.cncf.io/resourceName", "k8s.v1.cni.cncf.io/bridgeName"],
	    "honorResources": false
	  }
     }
kind: ConfigMap
metadata:
  name: network-resource-injector-customized-injection-config
  namespace: network-resource-injector-namespace

and mount config.json as config-volume inside the NRI pod, load it into a struct and configure the webhook accordingly.

[MichalGuzieniuk]

I read previous comments and tried to create final ConfigMap that would have section for control of features state and in addition possibility to define additional properties for features. Please find it below.

Within this issue, I'm not going to change implementation of for instance custom injections. I will focus only on ConfigMap handling and dynamic features toggle. It means that user will be able to enable / disable features on the fly.

This change also should take care of restoring default state of feature when ConfigMap will be removed. It means that the state before ConfigMap should be restored for instance defined by CLI flags.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nri-feature-control-map-1
  namespace: kube-system
data:
  # JSON at the beginning defines features with the status. The later sections may define additional arguments to each feature.
  # For instance resource key names, or custom injections which right now are stored in separate configMap
  config.json: |
    {
      "features": {
        "enableHugePageDownApi": false
        "enableHonorExistingResources": false
        "enableCustomizedInjection": false
        "enableResourceName": false
      },
      "networkResourceNameKeys": ["k8s.v1.cni.cncf.io/resourceName", "k8s.v1.cni.cncf.io/bridgeName"],
      "customInjection": {
        network-resource-injector-pod-annotation: '{"op": "add", "path": "/metadata/annotations", "value": {"k8s.v1.cni.cncf.io/networks": "sriov-net-attach-def"} }'
      }
    }

[zshi-redhat]

I think the issue has two aspects, one is to pass config to NRI, the other is to have structured config for various features.

For passing config to NRI, as @pperiyasamy mentioned, we could use configMap, or environment variable or a regular config file (mounted via volume) or cli cmd options. configMap and regular config file have advantage that features can be toggled on the fly when NRI is running, while environment variable and cli cmd can only be configured when NRI starts. If all of them are to be supported, there needs to be an order of preference.

For stuctured config data, as feature grows, we probably can define sections to organize the configurations, just like ini file, which may contain multiple sections, with each section contains key:value pair. For example:

[default]
enableCustomizedInjection = false

[resource]
injectResourceName = true
honorExistingResources = true
resourceNameKeys = ["key 1", "key 2"]

[volume] or [downward]
injectHugepageDownApi = true
injectNetInfo = true

[customInjection]
network-resource-injector-pod-annotation = '{"op": "add", "path": "/metadata/annotations", "value": {"k8s.v1.cni.cncf.io/networks": "sriov-net-attach-def"} }'

Section name can be different, depending on whether we want to separate them or have single section.

[martinkennelly]

Proposed Order of Precedence of NRI at start time:

1. CLI flags
2. Env vars
3. ConfigMap
4. Defaults

During run time, configMap updates would override.
I like the sections - Clearer for the user to determine what each key-value is related to.

[martinkennelly]

@zshi-redhat Are you ok with the order of precedence above?

[zshi-redhat]

@zshi-redhat Are you ok with the order of precedence above?

@martinkennelly the order looks good to me.

[MichalGuzieniuk]

I'm going to work on this issue. Please comment if someone have something against it.

[MichalGuzieniuk]

Can you explain me why definition of key, pairs for customInjections are within proposed proposed config map data section #61 (comment) ? I would expect that config map in that case have only control switch to enable or disable custom injections or any other feature of NRI. While for instance the user defined injections are stored in another config map - nri-user-defined-injections like it is done now.

[zshi-redhat]

Can you explain me why definition of key, pairs for customInjections are within proposed proposed config map data section #61 (comment) ? I would expect that config map in that case have only control switch to enable or disable custom injections or any other feature of NRI. While for instance the user defined injections are stored in another config map - nri-user-defined-injections like it is done now.

Good question!
While writing that comment, I was thinking putting all the supported configuration in the single configMap. but when I implemented the user-defined-injections, this feature control configMap didn't exist, so I used the separate configMap(nri-user-defined-injections).

For the feature control configmap, I think we can start by adding only enable/disable key/value pairs. then consider if it's needed to move user-defined-injection to this feature control configMap.