This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
Introduction
Azure Private Endpoints (PE) offer a robust and secure method for establishing connections via a private link. This blog focuses on utilizing PEs to link a Private Azure Kubernetes Service (AKS) cluster with a Storage account, aiming to assist in quick Proof-of-Concept setups. Although we spotlight the Storage service, the insights can be seamlessly applied to other Azure services. We will explore two specific scenarios for setting up Private Endpoints for AKS to interface with Azure Storage:
- A Private Endpoint part of the same Virtual Network (VNet) as AKS and in a distinct subnet.
- A Private Endpoint in a dedicated VNet, using VNet peering between the AKS and the PE VNet.
Pre-requisites
- Basic understanding of Azure Kubernetes Service (AKS), Virtual Networks (VNet), and Azure Storage
- An Azure subscription with Azure VM set up to access the Private AKS cluster, either using bastion or VM VNet peered to AKS VNet.
- Azure CLI and PowerShell installed and configured.
- PowerShell environment to run below CLI commands. For PowerShell CLI reference, check this link out.
- GitHub with code access, check this link out.
1. Private AKS with Private Endpoint Sharing AKS VNet
In this approach, the Private Endpoint (PE) for the Azure Storage Account will be created within the same VNet as the AKS cluster but in a separate subnet. This allows for a simplified network topology while ensuring that the traffic between the AKS cluster and the Storage Account remains private and within the Azure network backbone.
The following diagram depicts the configuration with Storage PE located within the AKS VNet. Connectivity is established via a bastion host linked to a VM in the AKS VNet or to a VM in a separate VNet that is peered with the AKS VNet. The highlights emphasize the primary focus of this section.
Setting Up Variables
# Use PowerShell
$resourceGroup="privateep-sharedvnet-rg"
$aksResourceGroup="aks-private" <- Replace with existing
$aksClusterName="aks-private" <- Replace with existing
$storageAccountName="saprivatesharedvnet"
$PESubnetName="pe-subnet"
$region="eastus"
$subnetPrefix='10.225.0.0/24' <- Replace with what applies to existing AKS VNet
$StoragePrivateEndpoint="StoragePESharedVNet"
$PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net"
Resource Group and Storage Account Creation
In this section, we create a resource group and a storage account. The storage account is then configured with a container, and public access is blocked to enhance security.
# Create resource group
az group create --name $resourceGroup --location $region
# Create storage account
az storage account create --name $storageAccountName --resource-group $resourceGroup --location $region --sku Standard_LRS
# Create container in above SA
$storageAccountKey = az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv
az storage container create --name "container01" --account-name $storageAccountName --account-key $storageAccountKey
# Block public access after container creation
az storage account update --name $storageAccountName --resource-group $resourceGroup --public-network-access Disabled
AKS Configuration
Here, we retrieve the AKS VNet and disable its public FQDN. This ensures that our AKS cluster is private and can only be accessed within the VNet.
# Get Private AKS VNet
$aksMCResourceGroup=az aks show --resource-group $aksResourceGroup --name $aksClusterName --query "nodeResourceGroup" --output tsv
$vnetInfo=az network vnet list --resource-group $aksMCResourceGroup --query '[0].{name:name, id:id}' --output json | ConvertFrom-Json
$aksVnetId=$vnetInfo.id
$aksVnetName=$vnetInfo.name
echo $aksVnetName
# Disable public FQDN on existing AKS Private cluster
az aks update -n $aksClusterName -g $resourceGroup --disable-public-fqdn
Private Endpoint Creation
We create a subnet within the AKS VNet dedicated for the private endpoint. Then, a private endpoint for the storage account is created within this subnet. This allows secure access to the storage account from the AKS cluster for blob storage types.
# Create Storage PE subnet on AKS VNet and get PE’s Subnet ID
az network vnet subnet create --name $PESubnetName --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --address-prefixes $subnetPrefix --disable-private-endpoint-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId
# Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE
az network private-endpoint create `
--resource-group $resourceGroup `
--name $StoragePrivateEndpoint `
--vnet-name $aksVnetName `
--subnet $PESubnetId `
--private-connection-resource-id $(az storage account show --name storageAccountName --resource-group $resourceGroup --query "id" --output tsv) `
--group-ids "blob" `
--connection-name "StoragePESharedVNetConnection"
Private DNS Configuration
A private DNS zone is created and a link to the AKS VNet is established. An A-record pointing to the storage private endpoint is added to this DNS zone, enabling name resolution within the VNet.
# Create a Private DNS Zone to associate with Storage PE
az network private-dns zone create --resource-group $resourceGroup --name $PrivateLinkNameSharedVNet
# Link AKS VNet to the Private DNS Zone. Using $aksVnetId, private-link will be in different RG than AKS Vnet (in MC Resource Group)
az network private-dns link vnet create --resource-group $resourceGroup --virtual-network $aksVnetId --name "AksPrivateDnsLink" --zone-name "$PrivateLinkNameSharedVNet" --registration-enabled false
# Get the Private IP Address of the Storage Private Endpoint:
$privateIpAddress = az network private-endpoint show --name $StoragePrivateEndpoint --resource-group $resourceGroup --query 'customDnsConfigs[0].ipAddresses[0]' --output tsv
echo $privateIpAddress
# Add the A-Record of Storage PE to the Private DNS Zone table
az network private-dns record-set a add-record --resource-group $resourceGroup --zone-name $PrivateLinkNameSharedVNet --record-set-name $storageAccountName --ipv4-address $privateIpAddress
Testing and Validation
After setting up the Private Endpoint and configuring the necessary networking components, it is essential to test and validate that the AKS cluster can communicate with the Storage Account over the Private Endpoint.
Basic Connectivity Testing
We deploy a pod within the AKS cluster to test connectivity to the storage account. This verifies that our configurations are correct, and that the AKS cluster can access the storage securely.
# Test to validate AZ CLI access from AKS Pod
$yaml = @"
apiVersion: v1
kind: Pod
metadata:
name: storage-connectivity-tester
spec:
containers:
- name: azure-cli
image: mcr.microsoft.com/azure-cli
command:
- sleep
- "3600"
"@
$yaml | kubectl apply -f -
# Get shell access to the Pod for interactive command run
kubectl exec storage-connectivity-tester -it -- bash
# Run below in shell to validate NW connectivity to storage accounts from AKS Pod
PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net"
storageAccountName="saprivatesharedvnet"
# Lookup should return PE IP address of Storage FQDN
nslookup $storageAccountName.$PrivateLinkNameSharedVNet
## Test for AZ CLI Storage connectivity
az login
resourceGroup="privateep-sharedvnet-rg"
storageAccountName="saprivatesharedvnet"
storageAccountKey=$(az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv)
echo $storageAccountKey
# List the containers in the storage account
az storage container list --account-name $storageAccountName --account-key $storageAccountKey
Storage Mount Testing
Deploying a stateful set Pod using below script verifies the ability to mount Azure Blob Storage as a file system using the Blob Container Storage Interface (CSI) driver. This will use the PE to access the storage account.
# Test to validate File mount using Blob CSI driver
$yaml = @"
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-blob-nfs
labels:
app: nginx
spec:
serviceName: statefulset-blob-nfs
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: statefulset-blob-nfs
image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
command:
- "/bin/sh"
- "-c"
- while true; do echo $(date) >> /mnt/azureblob/data; sleep 60; done
volumeMounts:
- name: persistent-storage
mountPath: /mnt/azureblob
volumeClaimTemplates:
- metadata:
name: persistent-storage
annotations:
volume.beta.kubernetes.io/storage-class: azureblob-nfs-premium
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 100Gi
"@
$yaml | kubectl apply -f -
# Validate that the Pod is running and verify from SA that /mnt/azureblob/data has contents
kubectl get pods,pv,pvc
Deletion on Completion
# Delete custom resource group on completion to remove all resources created above (other than AKS)
az group delete --name $resourceGroup --yes --no-wait
2. Private AKS with Private Endpoint in Dedicated VNet
In this scenario, the Private Endpoint (PE) for the Azure Storage Account will be created in a dedicated VNet separate from the AKS cluster's VNet, making it ideal for hub-spoke configurations.. VNet peering will be established between the AKS VNet and the dedicated VNet to allow private communication between the AKS cluster and the Storage Account.
The following diagram depicts the setup where the Storage Private Endpoint (PE) resides in a VNet that is peered with the AKS VNet. Access to the AKS VNet is facilitated either via a bastion host linked to a VM within the AKS VNet or through a VM in a separate VNet that is peered with the AKS VNet.
Setting Up Variables
# Use PowerShell
$resourceGroup="privateep-dedicatedvnet-rg"
$aksResourceGroup="aks-private" <- Replace with existing
$aksClusterName="aks-private" <- Replace with existing
$storageAccountName="saprivatededicatedvnet"
$PEVNetName="pe-vnet"
$PESubnetName="pe-subnet"
$region="eastus"
$vnetPrefix='10.208.0.0/12' <- Replace as needed
$subnetPrefix='10.208.0.0/14' <- Replace as needed
$StoragePrivateEndpoint="StoragePEDedicatedVNet"
$PrivateLinkNameDedicatedVNet="privatelink.blob.core.windows.net"
Resource Group, Storage Account, and VNet Creation
Like the previous use case, we start by creating a resource group, storage account, and a dedicated VNet for the private endpoint.
# Create resource group
az group create --name $resourceGroup --location $region
# Create storage account
az storage account create --name $storageAccountName --resource-group $resourceGroup --location $region --sku Standard_LRS
# Create container in above SA
$storageAccountKey = az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv
az storage container create --name "container01" --account-name $storageAccountName --account-key $storageAccountKey
# Block public access after container creation
az storage account update --name $storageAccountName --resource-group $resourceGroup --public-network-access Disabled
Private Endpoint Creation in Dedicated VNet
A subnet is created within the dedicated VNet for the private endpoint. A private endpoint for the storage account is then created within this subnet.
# Create a new PE VNet
az network vnet create --resource-group $resourceGroup --name $PEVNetName --address-prefix $vnetPrefix --location $region
# Create a subnet within the VNet dedicated for the private endpoint
az network vnet subnet create --resource-group $resourceGroup --vnet-name $PEVNetName --name $PESubnetName --address-prefix $subnetPrefix --disable-private-endpoint-network-policies true --disable-private-link-service-network-policies true
$PESubnetId = az network vnet subnet show --resource-group $resourceGroup --vnet-name $PEVNetName --name $PESubnetName --query 'id' --output tsv
echo $PESubnetId
# Create a Private Endpoint for the Storage Account using Subnet ID of Storage PE
az network private-endpoint create `
--resource-group $resourceGroup `
--name $StoragePrivateEndpoint `
--vnet-name $PEVNetName `
--subnet $PESubnetId `
--private-connection-resource-id $(az storage account show --name $storageAccountName --resource-group $resourceGroup --query "id" --output tsv) `
--group-ids "blob" `
--connection-name "StoragePEDedicatedVNetConnection"
Private DNS Configuration in Dedicated VNet
A private DNS zone is created and linked to both the AKS VNet and the dedicated VNet. This ensures name resolution for the storage account across both VNets.
# Create a Private DNS Zone to associate with Storage PE
## Create a new Private DNS Zone for PE
az network private-dns zone create --resource-group $resourceGroup --name $PrivateLinkNameDedicatedVNet
# Link AKS VNet to the Storage Private DNS Zone. Using $aksVnetId, private-link will be in different RG than AKS Vnet (in MC Resource Group)
## Get Private AKS VNet
$aksMCResourceGroup=az aks show --resource-group $aksResourceGroup --name $aksClusterName --query "nodeResourceGroup" --output tsv
$vnetInfo=az network vnet list --resource-group $aksMCResourceGroup --query '[0].{name:name, id:id}' --output json | ConvertFrom-Json
$aksVnetId=$vnetInfo.id
$aksVnetName=$vnetInfo.name
echo $aksVnetName
## Create a VNet link to AKS VNet
az network private-dns link vnet create --resource-group $resourceGroup --virtual-network $aksVnetId --name "AksPrivateDnsLink" --zone-name $PrivateLinkNameDedicatedVNet --registration-enabled false
# Add A-record pointing to Storage PE
## Get the Private IP Address of the Storage Private Endpoint:
$privateIpAddress = az network private-endpoint show --name $StoragePrivateEndpoint --resource-group $resourceGroup --query 'customDnsConfigs[0].ipAddresses[0]' --output tsv
## Add the A-Record of Storage PE to the Private DNS Zone table
az network private-dns record-set a add-record --resource-group $resourceGroup --zone-name $PrivateLinkNameDedicatedVNet --record-set-name $storageAccountName --ipv4-address $privateIpAddress
VNet Peering Configuration
VNet peering is established between the AKS VNet and the dedicated VNet. This allows resources in the AKS VNet to communicate with resources in the dedicated VNet.
# Create VNet peering between AKS and Storage VNets
## Create VNet Peering from AKS VNet to Storage VNet
az network vnet peering create --name peer-aks-storage --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --remote-vnet $(az network vnet show --resource-group $resourceGroup --name $PEVNetName --query id -o tsv) --allow-vnet
## Create VNet Peering from Storage VNet to AKS VNet
az network vnet peering create --name peer-storage-aks --resource-group $resourceGroup --vnet-name $PEVNetName --remote-vnet $(az network vnet show --resource-group $aksMCResourceGroup --name $aksVnetName --query id --out tsv) --allow-vnet-access
# Verification of VNet Peering should say 'Connected'
## Check peering status for AKS VNet
az network vnet peering show --name peer-aks-storage --resource-group $aksMCResourceGroup --vnet-name $aksVnetName --query peeringState
## Check peering status for Storage VNet
az network vnet peering show --name peer-storage-aks --resource-group $resourceGroup --vnet-name $PEVNetName --query peeringState
Testing and Validation in Dedicated VNet
Like the first scenario, after setting up the Private Endpoint in a dedicated VNet and configuring VNet peering, it is crucial to test and validate the connectivity between the AKS cluster and the Storage Account. In a similar manner we deploy a pod within the AKS cluster to test connectivity to the storage account in the dedicated VNet. Follow the steps in earlier Testing section, with summary listing below.
# Test to validate NW connectivity to storage accounts from AKS Pod
PrivateLinkNameSharedVNet="privatelink.blob.core.windows.net"
storageAccountName="saprivatededicatedvnet"
nslookup $storageAccountName.$PrivateLinkNameSharedVNet
# from Pod CLI run below
az login
resourceGroup="privateep-dedicatedvnet-rg"
storageAccountName="saprivatededicatedvnet"
storageAccountKey=$(az storage account keys list --resource-group $resourceGroup --account-name $storageAccountName --query '[0].value' --output tsv)
echo $storageAccountKey
# List the containers in the storage account
az storage container list --account-name $storageAccountName --account-key $storageAccountKey
Deletion on Completion
# Delete custom resource group on completion to remove all resources created above (other than AKS)
az group delete --name $resourceGroup --yes --no-wait
References
- Create a private endpoint by using the Azure CLI
- Azure services DNS zone configuration
- Connect to a storage account using an Azure Private Endpoint
- Connect privately to an Azure container registry using Azure Private Link
- Create a private Azure Kubernetes Service (AKS) cluster
Conclusion
Azure Private Endpoints provide a secure and private method to connect Azure services like AKS and Storage Accounts. By leveraging Private Endpoints, organizations can ensure that their data remains within the Azure network, reducing the exposure to potential threats. Whether you choose to use a shared VNet or a dedicated VNet for your Private Endpoint, the steps outlined in this blog post will guide you in setting up and validating the connectivity between your AKS cluster and Azure Storage Account.
Disclaimer
The sample scripts are not supported by any Microsoft standard support program or service. The sample scripts are provided AS IS without a warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.