patuzov / terraform-private-aks Goto Github PK

I followed your great article, very well written! But once the cluster is created, I cannot create an internal LoadBalancer service (with annotation: service.beta.kubernetes.io/azure-load-balancer-internal: "true") because the AKS identity lacks the Network Contributor role on the subnet:

  Normal   EnsuringLoadBalancer    2s (x3 over 17s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  2s (x3 over 17s)  service-controller  Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '30e14e7e-3eeb-4340-9003-7935d1007782' with object id '30e14e7e-3eeb-4340-9003-7935d1007782' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/---/resourceGroups/fullyprivx/providers/Microsoft.Network/virtualNetworks/spoke1-kubevnet/subnets/aks-subnet' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

I'm submitting a PR that includes:

resource "azurerm_role_assignment" "netcontributor" {
  role_definition_name = "Network Contributor"
  scope                = module.kube_network.subnet_ids["aks-subnet"]
  principal_id         = azurerm_kubernetes_cluster.privateaks.identity[0].principal_id
}

to fix this. Please note that it might take considerable amount of time for propagation of the role assignment. I witnessed up to 20 minutes before the loadbalancer got an internal IP (during that time, the above error is reported in the service events).

Interesting behavior: even in fully private clusters it's still possible to create Public IPs with services of type loadbalancer (without the annotations); you need an OPA policy to prevent that to happen. The service is anyway not functional (asymmetric path). Also, for now this works with AzureCNI only until this issue is resolved (it gives the following error):

Error: creating Managed Kubernetes Cluster "private-aks" (Resource Group "nopublicipaks"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="CustomRouteTableWithMSINotSupported" Message="Clusters using managed identity do not support bringing your own route table. Please see https://aka.ms/aks/customrt for more information"

I used this tool to create new vnet and subnets http://jodies.de/ipcalc?host=10.1.5.0&mask1=22&mask2=24

Thank you

running the code with version:
Terraform v0.15.3
on linux_amd64

• provider registry.terraform.io/hashicorp/azurerm v2.63.0
• provider registry.terraform.io/hashicorp/random v3.1.0

returns the error
PrivateEndpointNetworkPoliciesCannotBeEnabledOnPrivateEndpointSubnet

What am I doing wrong?

Many thanks for this project and your associated article - it has been very helpful.

Please could you add a LICENSE file to the repo to make the licensing terms explicit.

I have tried to run pods based on images stored on an Azure Container Registry that I have created, however, the pod always fails due to an authorisation error pulling the image.

I have tried installing docker on the jumpbox VM and logging to the ACR using the command
docker login loginServer -u 00000000-0000-0000-0000-000000000000 -p accessToken
As well as connecting the AKS to the ACR using the Kubernetes Secret method mentioned here but to no avail.

Could you please tell me how it is possible to run pods from images taken from an Azure Container Registry?