Quick-Start Guide to Azure Private Endpoints with AKS & Storage

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.

Introduction

Azure Private Endpoints (PE) offer a robust and secure method for establishing connections via a private link. This blog focuses on utilizing PEs to link a Private Azure Kubernetes Service (AKS) cluster with a Storage account, aiming to assist in quick Proof-of-Concept setups. Although we spotlight the Storage service, the insights can be seamlessly applied to other Azure services. We will explore two specific scenarios for setting up Private Endpoints for AKS to interface with Azure Storage:

  1. A Private Endpoint part of the same Virtual Network (VNet) as AKS and in a distinct subnet.
  2. A Private Endpoint in a dedicated VNet, using VNet peering between the AKS and the PE VNet.

 

Pre-requisites

  • Basic understanding of Azure Kubernetes Service (AKS), Virtual Networks (VNet), and Azure Storage
  • An Azure subscription with Azure VM set up to access the Private AKS cluster, either using bastion or VM VNet peered to AKS VNet.
  • Azure CLI and PowerShell installed and configured.
  • PowerShell environment to run below CLI commands. For PowerShell CLI reference, check this link out.
  • GitHub with code access, check this link out.

 

1. Private AKS with Private Endpoint Sharing AKS VNet

In this approach, the Private Endpoint (PE) for the Azure Storage Account will be created within the same VNet as the AKS cluster but in a separate subnet. This allows for a simplified network topology while ensuring that the traffic between the AKS cluster and the Storage Account remains private and within the Azure network backbone.

The following diagram depicts the configuration with Storage PE located within the AKS VNet. Connectivity is established via a bastion host linked to a VM in the AKS VNet or to a VM in a separate VNet that is peered with the AKS VNet. The highlights emphasize the primary focus of this section.varghesejoji_2-1696272561494.png

 

Setting Up Variables

 

# Use PowerShell $resourceGroup="privateep-sharedvnet-rg" $aksResourceGroup="aks-private" <- Replace with existing $aksClusterName="aks-private" <- Replace with existing $storageAccountName="saprivatesharedvnet" $PESubnetName="pe-subnet" $region="eastus" $subnetPrefix='10.225.0.0/24' <- Replace with what applies to existing AKS VNet $StoragePrivateEndpoint="StoragePESharedVNet" $PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net"

 

 

Resource Group and Storage Account Creation

In this section, we create a resource group and a storage account. The storage account is then configured with a container, and public access is blocked to enhance security.

 

# Create resource group az group create --name $resourceGroup --location $region # Create storage account az storage account create --name $storageAccountName --resource-group $resourceGroup --location $region --sku Standard_LRS # Create container in above SA $storageAccountKey = az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv az storage container create --name "container01" --account-name $storageAccountName --account-key $storageAccountKey # Block public access after container creation az storage account update --name $storageAccountName --resource-group $resourceGroup --public-network-access Disabled

 

 

AKS Configuration

Here, we retrieve the AKS VNet and disable its public FQDN. This ensures that our AKS cluster is private and can only be accessed within the VNet.

 

# Get Private AKS VNet $aksMCResourceGroup=az aks show --resource-group $aksResourceGroup --name $aksClusterName --query "nodeResourceGroup" --output tsv $vnetInfo=az network vnet list --resource-group $aksMCResourceGroup --query '[0].{name:name, id:id}' --output json | ConvertFrom-Json $aksVnetId=$vnetInfo.id $aksVnetName=$vnetInfo.name echo $aksVnetName # Disable public FQDN on existing AKS Private cluster az aks update -n $aksClusterName -g $resourceGroup --disable-public-fqdn

 

 

Private Endpoint Creation

We create a subnet within the AKS VNet dedicated for the private endpoint. Then, a private endpoint for the storage account is created within this subnet. This allows secure access to the storage account from the AKS cluster for blob storage types.

 

# Create Storage PE subnet on AKS VNet and get PE’s Subnet ID az network vnet subnet create --name $PESubnetName --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --address-prefixes $subnetPrefix --disable-private-endpoint-network-policies true $PESubnetId = az network vnet subnet show --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --name $PESubnetName --query 'id' --output tsv echo $PESubnetId # Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE az network private-endpoint create ` --resource-group $resourceGroup ` --name $StoragePrivateEndpoint ` --vnet-name $aksVnetName ` --subnet $PESubnetId ` --private-connection-resource-id $(az storage account show --name storageAccountName --resource-group $resourceGroup --query "id" --output tsv) ` --group-ids "blob" ` --connection-name "StoragePESharedVNetConnection"

 

 

Private DNS Configuration

A private DNS zone is created and a link to the AKS VNet is established. An A-record pointing to the storage private endpoint is added to this DNS zone, enabling name resolution within the VNet.

 

# Create a Private DNS Zone to associate with Storage PE az network private-dns zone create --resource-group $resourceGroup --name $PrivateLinkNameSharedVNet # Link AKS VNet to the Private DNS Zone. Using $aksVnetId, private-link will be in different RG than AKS Vnet (in MC Resource Group) az network private-dns link vnet create --resource-group $resourceGroup --virtual-network $aksVnetId --name "AksPrivateDnsLink" --zone-name "$PrivateLinkNameSharedVNet" --registration-enabled false # Get the Private IP Address of the Storage Private Endpoint: $privateIpAddress = az network private-endpoint show --name $StoragePrivateEndpoint --resource-group $resourceGroup --query 'customDnsConfigs[0].ipAddresses[0]' --output tsv echo $privateIpAddress # Add the A-Record of Storage PE to the Private DNS Zone table az network private-dns record-set a add-record --resource-group $resourceGroup --zone-name $PrivateLinkNameSharedVNet --record-set-name $storageAccountName --ipv4-address $privateIpAddress

 

 

Testing and Validation

After setting up the Private Endpoint and configuring the necessary networking components, it is essential to test and validate that the AKS cluster can communicate with the Storage Account over the Private Endpoint.

 

Basic Connectivity Testing

We deploy a pod within the AKS cluster to test connectivity to the storage account. This verifies that our configurations are correct, and that the AKS cluster can access the storage securely.

 

# Test to validate AZ CLI access from AKS Pod $yaml = @" apiVersion: v1 kind: Pod metadata: name: storage-connectivity-tester spec: containers: - name: azure-cli image: mcr.microsoft.com/azure-cli command: - sleep - "3600" "@ $yaml | kubectl apply -f - # Get shell access to the Pod for interactive command run kubectl exec storage-connectivity-tester -it -- bash # Run below in shell to validate NW connectivity to storage accounts from AKS Pod PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net" storageAccountName="saprivatesharedvnet" # Lookup should return PE IP address of Storage FQDN nslookup $storageAccountName.$PrivateLinkNameSharedVNet ## Test for AZ CLI Storage connectivity az login resourceGroup="privateep-sharedvnet-rg" storageAccountName="saprivatesharedvnet" storageAccountKey=$(az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv) echo $storageAccountKey # List the containers in the storage account az storage container list --account-name $storageAccountName --account-key $storageAccountKey

 

 

Storage Mount Testing

Deploying a stateful set Pod using below script verifies the ability to mount Azure Blob Storage as a file system using the Blob Container Storage Interface (CSI) driver. This will use the PE to access the storage account.

 

# Test to validate File mount using Blob CSI driver $yaml = @" apiVersion: apps/v1 kind: StatefulSet metadata: name: statefulset-blob-nfs labels: app: nginx spec: serviceName: statefulset-blob-nfs replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: statefulset-blob-nfs image: mcr.microsoft.com/oss/nginx/nginx:1.19.5 command: - "/bin/sh" - "-c" - while true; do echo $(date) >> /mnt/azureblob/data; sleep 60; done volumeMounts: - name: persistent-storage mountPath: /mnt/azureblob volumeClaimTemplates: - metadata: name: persistent-storage annotations: volume.beta.kubernetes.io/storage-class: azureblob-nfs-premium spec: accessModes: ["ReadWriteMany"] resources: requests: storage: 100Gi "@ $yaml | kubectl apply -f - # Validate that the Pod is running and verify from SA that /mnt/azureblob/data has contents kubectl get pods,pv,pvc

 

 

Deletion on Completion

 

# Delete custom resource group on completion to remove all resources created above (other than AKS) az group delete --name $resourceGroup --yes --no-wait

 

 

2. Private AKS with Private Endpoint in Dedicated VNet

In this scenario, the Private Endpoint (PE) for the Azure Storage Account will be created in a dedicated VNet separate from the AKS cluster's VNet, making it ideal for hub-spoke configurations.. VNet peering will be established between the AKS VNet and the dedicated VNet to allow private communication between the AKS cluster and the Storage Account.

The following diagram depicts the setup where the Storage Private Endpoint (PE) resides in a VNet that is peered with the AKS VNet. Access to the AKS VNet is facilitated either via a bastion host linked to a VM within the AKS VNet or through a VM in a separate VNet that is peered with the AKS VNet.varghesejoji_3-1696272561499.png

 

Setting Up Variables

 

# Use PowerShell $resourceGroup="privateep-dedicatedvnet-rg" $aksResourceGroup="aks-private" <- Replace with existing $aksClusterName="aks-private" <- Replace with existing $storageAccountName="saprivatededicatedvnet" $PEVNetName="pe-vnet" $PESubnetName="pe-subnet" $region="eastus" $vnetPrefix='10.208.0.0/12' <- Replace as needed $subnetPrefix='10.208.0.0/14' <- Replace as needed $StoragePrivateEndpoint="StoragePEDedicatedVNet" $PrivateLinkNameDedicatedVNet="privatelink.blob.core.windows.net"

 

 

Resource Group, Storage Account, and VNet Creation

Like the previous use case, we start by creating a resource group, storage account, and a dedicated VNet for the private endpoint.

 

# Create resource group az group create --name $resourceGroup --location $region # Create storage account az storage account create --name $storageAccountName --resource-group $resourceGroup --location $region --sku Standard_LRS # Create container in above SA $storageAccountKey = az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv az storage container create --name "container01" --account-name $storageAccountName --account-key $storageAccountKey # Block public access after container creation az storage account update --name $storageAccountName --resource-group $resourceGroup --public-network-access Disabled

 

 

Private Endpoint Creation in Dedicated VNet

A subnet is created within the dedicated VNet for the private endpoint. A private endpoint for the storage account is then created within this subnet.

 

# Create a new PE VNet az network vnet create --resource-group $resourceGroup --name $PEVNetName --address-prefix $vnetPrefix --location $region # Create a subnet within the VNet dedicated for the private endpoint az network vnet subnet create --resource-group $resourceGroup --vnet-name $PEVNetName --name $PESubnetName --address-prefix $subnetPrefix --disable-private-endpoint-network-policies true --disable-private-link-service-network-policies true $PESubnetId = az network vnet subnet show --resource-group $resourceGroup --vnet-name $PEVNetName --name $PESubnetName --query 'id' --output tsv echo $PESubnetId # Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE az network private-endpoint create ` --resource-group $resourceGroup ` --name $StoragePrivateEndpoint ` --vnet-name $PEVNetName ` --subnet $PESubnetId ` --private-connection-resource-id $(az storage account show --name $storageAccountName --resource-group $resourceGroup --query "id" --output tsv) ` --group-ids "blob" ` --connection-name "StoragePEDedicatedVNetConnection"

 

 

Private DNS Configuration in Dedicated VNet

A private DNS zone is created and linked to both the AKS VNet and the dedicated VNet. This ensures name resolution for the storage account across both VNets.

 

# Create a Private DNS Zone to associate with Storage PE ## Create a new Private DNS Zone for PE az network private-dns zone create --resource-group $resourceGroup --name $PrivateLinkNameDedicatedVNet # Link AKS VNet to the Storage Private DNS Zone. Using $aksVnetId, private-link will be in different RG than AKS Vnet (in MC Resource Group) ## Get Private AKS VNet $aksMCResourceGroup=az aks show --resource-group $aksResourceGroup --name $aksClusterName --query "nodeResourceGroup" --output tsv $vnetInfo=az network vnet list --resource-group $aksMCResourceGroup --query '[0].{name:name, id:id}' --output json | ConvertFrom-Json $aksVnetId=$vnetInfo.id $aksVnetName=$vnetInfo.name echo $aksVnetName ## Create a VNet link to AKS VNet az network private-dns link vnet create --resource-group $resourceGroup --virtual-network $aksVnetId --name "AksPrivateDnsLink" --zone-name $PrivateLinkNameDedicatedVNet --registration-enabled false # Add A-record pointing to Storage PE ## Get the Private IP Address of the Storage Private Endpoint: $privateIpAddress = az network private-endpoint show --name $StoragePrivateEndpoint --resource-group $resourceGroup --query 'customDnsConfigs[0].ipAddresses[0]' --output tsv ## Add the A-Record of Storage PE to the Private DNS Zone table az network private-dns record-set a add-record --resource-group $resourceGroup --zone-name $PrivateLinkNameDedicatedVNet --record-set-name $storageAccountName --ipv4-address $privateIpAddress

 

 

VNet Peering Configuration

VNet peering is established between the AKS VNet and the dedicated VNet. This allows resources in the AKS VNet to communicate with resources in the dedicated VNet.

 

# Create VNet peering between AKS and Storage VNets ## Create VNet Peering from AKS VNet to Storage VNet az network vnet peering create --name peer-aks-storage --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --remote-vnet $(az network vnet show --resource-group $resourceGroup --name $PEVNetName --query id -o tsv) --allow-vnet ## Create VNet Peering from Storage VNet to AKS VNet az network vnet peering create --name peer-storage-aks --resource-group $resourceGroup --vnet-name $PEVNetName --remote-vnet $(az network vnet show --resource-group $aksMCResourceGroup --name $aksVnetName --query id --out tsv) --allow-vnet-access # Verification of VNet Peering should say 'Connected' ## Check peering status for AKS VNet az network vnet peering show --name peer-aks-storage --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --query peeringState ## Check peering status for Storage VNet az network vnet peering show --name peer-storage-aks --resource-group $resourceGroup --vnet-name $PEVNetName --query peeringState

 

 

Testing and Validation in Dedicated VNet

Like the first scenario, after setting up the Private Endpoint in a dedicated VNet and configuring VNet peering, it is crucial to test and validate the connectivity between the AKS cluster and the Storage Account. In a similar manner  we deploy a pod within the AKS cluster to test connectivity to the storage account in the dedicated VNet. Follow the steps in earlier Testing section, with summary listing below.

 

# Test to validate NW connectivity to storage accounts from AKS Pod PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net" storageAccountName="saprivatededicatedvnet" nslookup $storageAccountName.$PrivateLinkNameSharedVNet # from Pod CLI run below az login resourceGroup="privateep-dedicatedvnet-rg" storageAccountName="saprivatededicatedvnet" storageAccountKey=$(az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv) echo $storageAccountKey # List the containers in the storage account az storage container list --account-name $storageAccountName --account-key $storageAccountKey

 

Deletion on Completion

 

# Delete custom resource group on completion to remove all resources created above (other than AKS) az group delete --name $resourceGroup --yes --no-wait

 

 

References

  1. Create a private endpoint by using the Azure CLI
  2. Azure services DNS zone configuration
  3. Connect to a storage account using an Azure Private Endpoint
  4. Connect privately to an Azure container registry using Azure Private Link
  5. Create a private Azure Kubernetes Service (AKS) cluster

 

Conclusion

Azure Private Endpoints provide a secure and private method to connect Azure services like AKS and Storage Accounts. By leveraging Private Endpoints, organizations can ensure that their data remains within the Azure network, reducing the exposure to potential threats. Whether you choose to use a shared VNet or a dedicated VNet for your Private Endpoint, the steps outlined in this blog post will guide you in setting up and validating the connectivity between your AKS cluster and Azure Storage Account.

 

Disclaimer

The sample scripts are not supported by any Microsoft standard support program or service. The sample scripts are provided AS IS without a warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.