This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .
As documented in Use Azure tags in Azure Kubernetes Service (AKS), you can use Azure tags on an AKS cluster to associate its related resources to a given workload or tenant. For some resources, such as a managed data disk created via a persistent volume claim or an Azure Public IP created by a public Kubernetes service, you can also use Kubernetes manifests to set Azure tags. Azure tags are a helpful mechanism to track resource usage and charge back their costs to separate tenants or business units within an organization. This article explains how to set Azure tags for AKS clusters and related resources.
Azure Tags
Tags are metadata elements that you apply to your Azure resources. They're key-value pairs that help you identify resources based on settings relevant to your organization. If you want to track the deployment environment for your resources, add a key named Environment. To identify the resources deployed to production, give them a value of Production. Fully formed, the key-value pair becomes Environment = Production. You can apply tags to your Azure resources, resource groups, and subscriptions. For more information on Azure Tags, see Use tags to organize your Azure resources and management hierarchy.
You can use Azure Policy to create tagging rules and conventions for Azure resources, including AKS clusters or its resources. By creating a policy, you avoid the scenario of resources being deployed to your subscription that don't have the expected tags for your organization. Instead of manually applying tags or searching for resources that aren't compliant, you create a policy that automatically applies the needed tags during deployment. Tags can also now be applied to existing resources with the new Modify effect and a remediation task. For more information, see Assign policy definitions for tag compliance.
How tags work in AKS
Before diving into how using Azure tags in AKS, let's see what happens when you set and update Azure tags with AKS clusters and their related resources:
- Tags set on an AKS cluster apply to all resources related to the cluster, but not the node pools and related virtual machine scale sets (VMSS) in the node resource group. This operation overwrites the values of existing keys.
- Tags set on a node pool apply only to resources related to that node pool, mainly the virtual machine scale set (VMSS) used by the node pool. This operation overwrites the values of existing keys. Resources outside that node pool, including resources for the rest of the cluster and other node pools, are unaffected.
- Azure public IPs, Azure files, and managed disks can have tags set by Kubernetes through a YAML Kubernetes manifest. Azure tags set in this way will maintain the Kubernetes values, even if you update them later using another method. When Azure public IPs, Azure files, or managed disks are removed through Kubernetes, any tags set by Kubernetes are removed. Tags on those resources that aren't tracked by Kubernetes remain unaffected.
Prerequisites
- The Azure CLI version 2.0.59 or later, installed and configured. To find your version, run
az --version
. If you need to install it or update your version, see Install Azure CLI. - Kubernetes version 1.20 or later, installed.
Limitations
- Azure tags have keys that are case-insensitive for operations, such as when you're retrieving a tag by searching the key. In this case, a tag with the specified key will be updated or retrieved regardless of the casing. Tag values are case-sensitive.
- In AKS, if multiple tags are set with identical keys but different casings, the tags are used in alphabetical order. For example,
{"Key1": "val1", "kEy1": "val2", "key1": "val3"}
results inKey1
andval1
being set. - For shared resources, tags aren't able to determine the split in resource usage on their own.
Add tags to the cluster
When you create or update an AKS cluster with the --tags
parameter, the following are assigned the Azure tags that you've specified:
- The AKS cluster
- The route table that's associated with the cluster
- The public IP that's associated with the cluster
- The load balancer that's associated with the cluster
- The network security group that's associated with the cluster
- The virtual network that's associated with the cluster
- The AKS managed kubelet managed identity associated with the cluster
- The AKS managed addon managed identity associated with the cluster
- The private DNS zone associated with the private cluster
- The private endpoint associated with the private cluster
NOTE: Azure Private DNS only supports 15 tags. tag resources.
To create a cluster and assign Azure tags, you can run az aks create with the --tags
parameter, as shown in the following command. Running the command creates a myAKSCluster
in the myResourceGroup
with the tags dept=IT
and costcenter=9999
.
az aks create \
--resource-group myResourceGroup \
--name myAKSCluster \
--tags dept=IT costcenter=9999 \
--generate-ssh-keys
NOTE: To set tags on the initial node pool, the node resource group, the virtual machine scale set, and each virtual machine scale set instance that's associated with the initial node pool, also set the
--nodepool-tags
parameter.
az aks create \
--resource-group myResourceGroup \
--name myAKSCluster \
--tags dept=IT costcenter=9999 \
--nodepool-tags dept=IT costcenter=9999 \
--generate-ssh-keys
IMPORTANT: If you're using existing resources when you're creating a new cluster, such as an IP address or route table,
az aks create
overwrites the set of tags. If you delete that cluster later, any tags set by the cluster will be removed.
You can use the az aks show command to verify that the tags have been properly applied to the cluster and related resources. The cluster tags for myAKSCluster
are shown in the following example:
$ az aks show -g myResourceGroup -n myAKSCluster --query '[tags]'
{
"clusterTags": {
"costcenter": "9999",
"dept": "IT"
}
}
To update the tags on an existing cluster, you can run the az aks update command with the --tags
parameter. Running the command updates the myAKSCluster
with the tags team=alpha
and costcenter=1234
.
az aks update \
--resource-group myResourceGroup \
--name myAKSCluster \
--tags team=alpha costcenter=1234
You can use the az aks show command to verify that the tags have been applied to the cluster. For example:
$ az aks show -g myResourceGroup -n myAKSCluster --query '[tags]'
{
"clusterTags": {
"costcenter": "1234",
"team": "alpha"
}
}
IMPORTANT: Setting tags on a cluster by using az aks update overwrites the set of tags. For example, if your cluster has the tags
dept=IT
andcostcenter=9999
and you use az aks update with the tagsteam=alpha
andcostcenter=1234
, the new list of tags would beteam=alpha
andcostcenter=1234
.
In the following example, the businessUnit=FastTrack for Azure
and supportedBy=Paolo Salvatori
tags of the AKS cluster were created via Azure Policy, while the IaC=Bicep
and workload=tags
tags were created at creation time via Bicep.
Adding tags to node pools
You can apply one or more Azure tags to a new or existing node pool in your AKS cluster. Tags applied to a node pool are applied to each node within the node pool and are persisted through upgrades. Tags are also applied to new nodes that are added to a node pool during scale-out operations. Adding a tag can help with policy tracking or cost estimation tasks.
When you create or update a node pool with the --tags
parameter, the tags that you specify are assigned to the following resources:
- The node pool
- The node resource group
- The virtual machine scale set and each virtual machine scale set instance that's associated with the node pool
To create a node pool with an Azure tag, run az aks nodepool add with the --tags
parameter. Running the following command creates a contosoNodePool
node pool with the tags tenant=contoso
and costcenter=5555
in the myAKSCluster
.
az aks nodepool add \
--resource-group myResourceGroup \
--cluster-name myAKSCluster \
--name contosoNodePool \
--node-count 1 \
--tags tenant=contoso costcenter=5555 \
--no-wait
You can use the az aks show command to verify that the tags have been applied to the contosoNodePool
node pool.
$ az aks show -g myResourceGroup -n myAKSCluster --query 'agentPoolProfiles[].{nodepoolName:name,tags:tags}'
[
{
"nodepoolName": "nodepool1",
"tags": null
},
{
"nodepoolName": "contosoNodePool",
"tags": {
"tenant": "contoso",
"costcenter": "5555"
}
}
]
To update a node pool with an Azure tag, you can run the az aks nodepool update command with the --tags
parameter. Running the following command updates the contosoNodePool
node pool with the tags appversion=1.0.0
and costcenter=4444
in the myAKSCluster
, which already has the tags tenant=contoso
and costcenter=5555
.
az aks nodepool update \
--resource-group myResourceGroup \
--cluster-name myAKSCluster \
--name contosoNodePool \
--tags appversion=1.0.0 costcenter=4444 \
--no-wait
IMPORTANT: Setting tags on a node pool by using the az aks nodepool update command overwrites the set of tags. For example, if your node pool has the tags
tenant=contoso
andcostcenter=5555
, and you use the az aks nodepool update command with the tagsappversion=1.0.0
andcostcenter=4444
, the new list of tags would beappversion=1.0.0
andcostcenter=4444
.
You can use the az aks show command to verify that the tags have been appropriately updated on the node pool.
$ az aks show -g myResourceGroup -n myAKSCluster --query 'agentPoolProfiles[].{nodepoolName:name,tags:tags}'
[
{
"nodepoolName": "nodepool1",
"tags": null
},
{
"nodepoolName": "contosoNodePool",
"tags": {
"appversion": "1.0.0",
"costcenter": "4444"
}
}
]
In the following example, the osDiskType=ephemeral
and osType=Windows2022
tags of the node pool VMSS were created using the az aks nodepool create command with the --tags
parameter at provisioning time.
Add tags by using Kubernetes manifests
You can apply Azure tags to Azure public IPs, managed disks, and Azure files by using a Kubernetes YAML manifest.
Azure Public IPs
The service.beta.kubernetes.io/azure-pip-tags
annotation can be to determines what Azure tags should be applied to the Azure public IP used by a Kubernetes service. The supported format is a=b,c=d,...
. After updated, the old user-assigned tags would not be replaced by the new ones.
For example, the following YAML file creates a Kubernetes deployment that uses NGINX container image, and exposes it via a public LoadBalancer Kubernetes service. The service definition uses the costcenter=1234,tenant=Contoso
annotation to set Azure tags on the Azure public IP resource created by the cloud controller manager.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: nginx
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
resources:
requests:
memory: "128Mi"
cpu: "250m"
limits:
memory: "256Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /
port: 80
failureThreshold: 1
initialDelaySeconds: 1
periodSeconds: 30
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: 80
failureThreshold: 1
initialDelaySeconds: 1
periodSeconds: 30
timeoutSeconds: 5
startupProbe:
httpGet:
path: /
port: 80
failureThreshold: 1
initialDelaySeconds: 1
periodSeconds: 30
timeoutSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
annotations:
service.beta.kubernetes.io/azure-pip-tags: costcenter=1234
spec:
externalTrafficPolicy: Local
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
type: LoadBalancer
In the following example, the costcenter=1234
tag of the Azure Public IP in the node resource group of the AKS cluster was created by the service.beta.kubernetes.io/azure-pip-tag
annotation in the YAML manifest used to deploy the Kubernetes service.
IMPORTANT: Setting tags on files, disks, and public IPs by using Kubernetes updates the set of tags. For example, if your disk has the tags dept=IT and costcenter=5555, and you use Kubernetes to set the tags team=beta and costcenter=3333, the new list of tags would be dept=IT, team=beta, and costcenter=3333. Any updates that you make to tags through Kubernetes will retain the value that's set through Kubernetes. For example, if your disk has tags dept=IT and costcenter=5555 set by Kubernetes, and you use the portal to set the tags team=beta and costcenter=3333, the new list of tags would be dept=IT, team=beta, and costcenter=5555. If you then remove the disk through Kubernetes, the disk would have the tag team=beta.
Azure Disks and Files
You can apply Azure tags to Azure managed disks and Azure files by using a Kubernetes YAML manifest created via a persistent volume claim PVC. Today you cannot specify metadata and, in particular, tags in a PVC that are passed down to a CSI driver. Today, the only way to tag managed resources dynamically created by a PVC on a cloud platform is using the tags parameter of the Azure Disks CSI driver:
- Create a custom storage class and specify tags in the parameters:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: contoso-managed-csi-premium
provisioner: disk.csi.azure.com
parameters:
skuname: Premium_LRS
tags: costcenter=1234,tenant=Contoso
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
- Create a PVC based on the custom storage class:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: contoso-managed-csi-premium
spec:
accessModes:
- ReadWriteOnce
storageClassName: contoso-managed-csi-premium
resources:
requests:
storage: 50Gi
- Use the PVC in your deployment definition
In a multi-tenant environment, this approach can lead to two kinds of problems:
- The proliferation of custom storage classes, one for each tenant. If you share an AKS cluster across hundreds of tenants each using PVs, and you want to use this technique to tag managed disks, you can quickly end up with hundreds of custom storage classes
- Storage classes are not namespaced resources, so this can lead to issues when trying to segregate or restrict security using Kubernetes RBAC or Azure AD integrated RBAC
You can use the following workaround to avoid the proliferation of storage classes:
- You can create a custom storage class with the proper tags for a tenant
- Based on the custom storage class, create one or more PVCs, which dynamically create related managed resources on the current cloud platform.
- Delete the custom storage class.
In the following example, the costcenter=1234
and tenant=Contoso
tags of the Azure managed disk in the node resource group of the AKS cluster were created by the tags
parameter in the YAML manifest used to create the contoso-managed-csi-premium
storage class used by the persistent volume claim (PVC).