Can we have the option for a CCM implementation to add cloud-provider-specific tags to nodes when they are created?
NOTE: This is the result of a slack discussion on the cloud-provider
channel here and the follow-on thread with @andrewsykim here
The way I had envisioned this is to extend the Instances interface
as follows:
// InstanceTags returns a map of cloud-provider specific tags for the specified instance.
// May be called multiple times. The keys of the map always will be prefixed with the
// name of the cloud provider as "cloudprovider.kubernetes.io/<providername>/".
InstanceTags(ctx context.Context, name types.NodeName) (map[string]string, error)
// InstanceTagsByProviderID returns a map of cloud-provider specific tags for the specified instance.
// May be called multiple times. The keys of the map always will be prefixed with the
// name of the cloud provider as "cloudprovider.kubernetes.io/<providername>/".
InstanceTagsByProviderID(ctx context.Context, providerID string) (map[string]string, error)
These would be idempotent, similar to (most of) the other functions in the interface, e.g. InstanceType()
or InstanceTypeByProviderID()
.
Also note that the tags would have their keys prefixed by a cloud-provider-specific prefix, so as to prevent returned tags from trashing anything the user put on, or k8s-native tags. I picked cloudprovider.kubernetes.io/<providername>/
but anything will do.
@andrewsykim raised the valid issue that this might lead to all users of a CCM getting all of the tags. I think this is a problem for each CCM provider to solve in its own way. The cloud-provider implementation here would give each CCM the option to add tags; each CCM implementor would choose how to handle it: some would never add tags, because their user base wouldn't want it; others would always add tags, because their user base does; others would have tags, but controllable via config, whether CLI or env var options in the manifest that deploys the CCM, a CCM-controlling ConfigMap
or some other mechanism. The key point is to create the option for each provider.
@andrewsykim also raised a possible alternate option, which may be complementary, specifically that we have an "add node" hook, also likely under Instances interface
that would pass the node definition and allow the CCM to do whatever it needs with the definition of the node in kubernetes. This has the same tag (and other) issues as above, which could be involved in the same way. It is less idempotent, but has more options for CCM control of the node addition.
We could do both.
Finally, the implementation of it in cloud-provider is fairly straightforward. We would add the two funcs to the Instances interface
, as above, and then extend getNodeModifiersFromCloudProvider
to get the tags as modifiers, see here.
Looking forward to comments and feedback.