Multiple ENIs is confusing cloud-provider-aws controller about cloud-provider-aws

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kmala]

Once the Node object internal IP is set ; it should not be replaced by the one of the other ENI.

I don't think IP is replaced. kubectl shows just the one IP but the node object should have all the IP's as that is the default behavior and they should be ordered based on the interface number

cloud-provider-aws/pkg/providers/v1/aws.go

Lines 735 to 757 in cea2af6

	// extractIPv4NodeAddresses maps the instance information from EC2 to an array of NodeAddresses.
	// This function will extract private and public IP addresses and their corresponding DNS names.
	func extractIPv4NodeAddresses(instance *ec2.Instance) ([]v1.NodeAddress, error) {
	// Not clear if the order matters here, but we might as well indicate a sensible preference order
	if instance == nil {
	return nil, fmt.Errorf("nil instance passed to extractNodeAddresses")
	}
	addresses := []v1.NodeAddress{}
	// sort by device index so that the first address added to the addresses list is from the first (primary) device
	sort.Slice(instance.NetworkInterfaces, func(i, j int) bool {
	// These nil checks should cause interfaces with non-nil attachments to sort before those with nil attachments
	if instance.NetworkInterfaces[i].Attachment == nil {
	return false
	}
	if instance.NetworkInterfaces[j].Attachment == nil {
	return true
	}
	return aws.Int64Value(instance.NetworkInterfaces[i].Attachment.DeviceIndex) < aws.Int64Value(instance.NetworkInterfaces[j].Attachment.DeviceIndex)
	})

from cloud provider release 1.29.3. Can you upgrade and test?

Node controller will make sure that addresses of the instance is always same as node object addresses https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cloud-provider/controllers/node/node_controller.go#L193-L197.

What I cannot explain though, is why this secondary private IP attached to the same instance is not answering properly those calls even though the firewall was allowing any kind of traffic from any source.

what is the error you are facing , did you check the apisever logs for the reason? It could be because of cert verification also.

[cartermckinnon]

Do you pass the --node-ip flag to kubelet?

[MadJlzz]

what is the error you are facing , did you check the apisever logs for the reason? It could be because of cert verification also.

It's been quite some time, I have to dig back into it to get extra details. I had problem getting back results from commands like kubectl logs or kubectl exec being proxified by the api-server to the correct node's kubelet.

Do you pass the --node-ip flag to kubelet?

I had to do that as a workaround, yes. The IP I have set is the primary IP of initial network interface of the EC2 instance.

As soon as I have time, I'll try to get some more informations and put them here.

[cartermckinnon]

There's been some recent discussions about --node-ip and how the external CCM should handle it. At this point, passing --node-ip to kubelet is the right thing to do, for AWS at least. Here's how we do it for the AL2-based EKS AMI: https://github.com/awslabs/amazon-eks-ami/blob/e50acfb7e6be088dde823dc80b21c50651e71b01/templates/al2/runtime/bootstrap.sh#L490-L495

More: kubernetes/kubernetes#125337