Private AKS endpoint don't use PrivateDnsZoneGroup about private-endpoints-topologies

[paolosalvatori]

Hi @jkroepke thanks for the positive feedback. I spent some time enabling the use of a private AKS cluster in the hub and spoke topology described in this article and I also thought to publish an article on this subject. However, I know that the product group is working in an easier way to make it possible to resolve the control plane: Azure/AKS#2176. So I put my idea on hold waiting for the AKS team to release a public preview for this feature which is a work in progress 😄
Today, here are the options:

• standard installation: you need an event-driven solution to create a virtual network link between the Private DNS Zone (PDZ) created by the private AKS cluster deployment and the hub virtual network
• Hosts files solution: you need to explicitly create a mapping between the FQDN of the API server and the IP address of the private endpoint to the API Server in any jump box VM or Azure DevOps self-hosted agent that needs to use the kubectl command or access the Control Plane
• BYO PDZ:
  • BYO PDZ in the hub virtual network: app teams users that deploy a private AKS cluster in their landing zone subscription and the service principal/managed identity used by the AKS cluster need to have write permissions on the PDZ in the hub VNET
  • BYO PDZ in the landing zone subscription: the naming convention for the BYO PDZ is very unfortunate as it does not allow to differentiate it from the name of the BYO DNS in another landing zone subscription. Now, all the spoke vnets are configured to send DNS queries to the DNS servers in the hub vnets, hence you should create a virtual network link between the BYO PDZ used by the private AKS cluster in the landing zone subscription and the hub virtual network that hosts the DNS forwarders. The issue is that you can't create more than one virtual network link between a virtual network and a PDZ with the same name. Hence, this solution works for the first private AKS cluster, but if you want to create another private AKS cluster in another landing zone subscription with a BYO PDZ that has fixed name, and then you create a virtual network link between this and hub virtual network, you get an error. Failed to create virtual network link 'link_to_hub_ne'. Error: A virtual network cannot be linked to multiple zones with overlapping namespaces. You tried to link the virtual network with 'privatelink.northeurope.azmk8s.io' and 'privatelink.northeurope.azmk8s.io' zones.

[jkroepke]

Hi @paolosalvatori ,

thanks for your input. Azure/AKS#2176 looks empty for me. I have no idea whats planned here.

About your points:

standard installation: you need an event-driven solution to create a virtual network link between the Private DNS Zone (PDZ) created by the private AKS cluster deployment and the hub virtual network

I guess, thats possible with Azure Policy using the deployIfNotExists. Currently I have some personal issues the Azure policies. Hard to debug, high latency (up to 30 Minutes after creation, up to 15 minutes to see newly creates resources). I do not like them.

Hosts files solution: you need to explicitly create a mapping between the FQDN of the API server and the IP address of the private endpoint

This looks not bullet proof. Reading https://docs.microsoft.com/en-us/azure/aks/private-clusters#limitations

In the case of maintenance on the control plane, your AKS IP might change. In this case you must update the A record pointing to the API server private IP on your custom DNS server and restart any custom pods or deployments using hostNetwork.

This is not an option.

BYO PDZ in the hub virtual network: app teams users that deploy a private AKS cluster in their landing zone subscription and the service principal/managed identity used by the AKS cluster need to have write permissions on the PDZ in the hub VNET

As I know, I could define a dns prefix for AKS clusters.

Is it possible the restrict this permissions, e.g. by Azure Policies? If every teams get an own dns-prefix, I would like to restrict teams to only manage dns entries they belongs to.

[paolosalvatori]

Can you see https://github.com/Azure/AKS/projects/1?

I tried to use a deployIfNotExists policy, and it works correctly, but it takes about 15 minutes to complete while the custom script extension run by VMSS nodes times out after 5 minutes. I also asked the AKS team if they could increase the timeout but they replied this was not possible.

The host-file based solution works, but then you should create the same mapping in the /etc/hosts file of the jumpbox VM and any Azure DevOps self-hosted agent or GitHub Actions self-hosted runner VM that you deploy in the AKS VNET or a in peered VNET.

The DNS prefix you referred has nothing to do with the prefix of the private DNS zone.

[jkroepke]

Can you see https://github.com/Azure/AKS/projects/1?
Yes, I'm able to see the full roadmap. I could see the issue, but the issue does not have any content.

I tried to use a deployIfNotExists policy, and it works correctly, but it takes about 15 minutes to complete while the custom script extension run by VMSS nodes times out after 5 minutes. I also asked the AKS team if they could increase the timeout but they replied this was not possible.

Would you like to share the policy?

The host-file based solution works, but then you should create the same mapping in the /etc/hosts file of the jumpbox VM and any Azure DevOps self-hosted agent or GitHub Actions self-hosted runner VM that you deploy in the AKS VNET or a in peered VNET.

In our current concept there is not bastion host/jumpbox VM. The API of AKS should be accessible from clients (connected via S2S VPN). We are not able to set host-file on the (windows) clients. Additionally, the DNS Server of the vnet links to an central dns forwarder. Pods should able to resolve on-premise names.

The DNS prefix you referred has nothing to do with the prefix of the private DNS zone.

This is correct. I was not talking about PDZ itself.

Given you a deeper example:

• I have a central zone called privatelink.northeurope.azmk8s.io.
• Additionally, teams have a pre provisioned managed identity. The MI has the "Private DNS Zone Contributor" role inside privatelink.northeurope.azmk8s.io PDZ.

I would like to control the creation/deletion of A Records (through AKS), to prevent some mistakes (like Team A should not be able to manage Records from Team B).

Is it possible to have Policy (assigned to a subscription) that enforce a name pattern on the dns records created inside the PDF?

Like:

• If the DNS A Records begins with "team1-", move forward else the action is denied.

[paolosalvatori]

This is the policy definition. As I said, it works as expected but it doesn't trigger on time, and the setup of the AKS cluster will fail. Instead, an event-driven solution like https://github.com/rhummelmose/private-aks-dns-zone-linker-function-app is able to create the necessary virtual network link on time during the AKS cluster setup.

{
  "name": "Create-Virtual-Network-Link-to-AKS-Private-DNS-Zone",
  "properties": {
    "displayName": "Create-Virtual-Network-Link-to-AKS-Private-DNS-Zone",
    "description": "This policy automatically creates a virtual network link between the private DNS zone of a private AKS cluster and the hub virtual networks.",
    "policyType": "Custom",
    "mode": "Indexed",
    "metadata": {
      "version": "1.0.0",
      "author": "Paolo Salvatori",
      "company": "Microsoft"
    },
    "parameters": {
      "virtualNetworkId": {
        "type": "string",
        "metadata": {
          "description": "Specifies the resource id of the virtual network.",
          "displayName": "Virtual Network Resource Id",
          "strongType": "Microsoft.Network/virtualNetworks"
        }
      }
    },
    "policyRule": {
      "if": {
        "allOf": [{
            "field": "type",
            "equals": "Microsoft.Network/privateDnsZones"
          },
          {
            "allOf": [{
                "field": "name",
                "like": "*.azmk8s.io"
              },
              {
                "field": "name",
                "contains": ".privatelink."
              }
            ]
          }
        ]
      },
      "then": {
        "effect": "deployIfNotExists",
        "details": {
          "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
          "roleDefinitionIds": [
            "/providers/Microsoft.Authorization/roleDefinitions/4d97b98b-1d4f-4787-a291-c67834d212e7"
          ],
          "existenceCondition": {
            "field": "Microsoft.Network/privateDnsZones/virtualNetworkLinks/virtualNetwork.id",
            "equals": "[parameters('virtualNetworkId')]"
          },
          "deployment": {
            "properties": {
              "mode": "incremental",
              "parameters": {
                "virtualNetworkId": {
                  "value": "[parameters('virtualNetworkId')]"
                },
                "privateDnsZoneName": {
                  "value": "[field('name')]"
                },
                "location": {
                  "value": "[field('location')]"
                }
              },
              "template": {
                "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
                "contentVersion": "1.0.0.0",
                "parameters": {
                  "virtualNetworkId": {
                    "type": "string"
                  },
                  "privateDnsZoneName": {
                    "type": "string"
                  },
                  "location": {
                    "type": "string"
                  }
                },
                "resources": [{
                  "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
                  "apiVersion": "2018-09-01",
                  "name": "[concat(parameters('privateDnsZoneName'), '/link_to_', if(greaterOrEquals(lastIndexOf(parameters('virtualNetworkId'), '/'), 0), substring(parameters('virtualNetworkId'), add(lastIndexOf(parameters('virtualNetworkId'), '/'), 1)), 'HubVirtualNetwork'))]",
                  "location": "global",
                  "properties": {
                    "registrationEnabled": false,
                    "virtualNetwork": {
                      "id": "[parameters('virtualNetworkId')]"
                    }
                  }
                }]
              }
            }
          }
        }
      }
    }
  }
}

and this is the policy assignment. You can assign a policy at a management group level, like below, or at the subscription level.

{
    "name": "VNL2AKSPDZ",
    "location": "westurope",
    "Identity": {
      "type": "SystemAssigned"
    },
    "properties": {
      "displayName": "VNL2AKSPDZ-NE-Classic",
      "description": "This policy automatically creates a virtual network link between the private DNS zone of a private AKS cluster and the hub virtual network in North Europe.",
      "metadata": {
        "version": "1.0.0",
        "author": "Paolo Salvatori",
        "company": "Microsoft"
      },
      "policyDefinitionId": "/providers/Microsoft.Management/managementgroups/Main/providers/Microsoft.Authorization/policyDefinitions/Create-Virtual-Network-Link-to-AKS-Private-DNS-Zone",
      "scope": "/providers/Microsoft.Management/managementgroups/Main",
      "notScopes": [],
      "parameters": {
        "virtualNetworkId": {
          "value": "/subscriptions/e363a06e-xxxx-xxxx-xxxx-1aa1f5db8b1a/resourceGroups/NetworkRG/providers/Microsoft.Network/virtualNetworks/HubVirtualNetwork"
        }
      }
    }
  }