Many CCM in one cluster. about cloud-provider

[cheftako]

Having multiple copies of the same controller operating simultaneously is asking for race conditions. Currently we prevent this using controller manager level leader election. I am assuming that you have already given one of the two CCMs an alternate leader election name to circumvent the leader election protection. If you want to go that route I would suggest using the controllers flag (https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/controller-manager/options/generic.go#L65) to control the controllers in each CCM to ensure that each controller is only running in one of the two CCMs.

[sergelogvinov]

Sorry, I'll add more details:

I have two cloud providers with a different ProviderName (nameA/NameB). I have deployments for each cloud-provider.
Deployment(CCM) has its own lock resource --leader-elect-resource-name=nameA/nameB. It solves race conditions for one type of CCM (for one cloud provider).

All CCMs can initialize the nodes and set providerID (nameA://instanceId or nameB://instanceId)

And based on providerName(nameX://), CCMs can choose the right node and make a node lifecycle.
In this case CCM have to skip the nodes from another cloud provider. sergelogvinov@8468b74#diff-807a869d357013a377e7a6153dc3133491ca7dfb909690f0dfa2b1f2b873fa60R131-R135

And problem her - when the node joins the cluster, it does not have a providerID yet.
One of CCMs will delete the node because a cloud provider does not have this node. It belongs to another cloud provider.
If wait a few second, another CCM initialize the node and set providerID. sergelogvinov@8468b74#diff-807a869d357013a377e7a6153dc3133491ca7dfb909690f0dfa2b1f2b873fa60R162-R169

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[james-callahan]

I have this same question: how has anyone implemented a multi-cloud kubernetes cluster without this?

Related: #35