This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
This article and the companion Azure code sample show how to use Azure AD workload identity for Kubernetes in a .NET Standard application running on Azure Kubernetes Service. It leverages the public preview capability of Azure AD workload identity federation and a user-assigned managed identity.
Azure AD Workload Identity for Kubernetes
Today Azure Kubernetes Service (AKS) allows you to assign managed identities at the pod-level, which has been a preview feature. This pod-managed identity allows the hosted workload or application access to resources through Azure Active Directory (Azure AD). For example, a workload stores files in Azure Storage, and when it needs to access those files, the pod authenticates itself against the resource as an Azure-managed identity. This authentication method has been replaced with AzureAD workload identity, which integrates with the Kubernetes native capabilities to federate with any external identity provider. This approach is simpler to use and deploy and overcomes several limitations in Azure AD Pod Identity:
- Removes the scale and performance issues that existed for identity assignment
- Supports Kubernetes clusters hosted in any cloud or on-premises
- Supports both Linux and Windows workloads
- Removes the need for Custom Resource Definitions and pods that intercept Azure Instance Metadata Service (IMDS) traffic
- Avoids the complicated and error-prone installation steps, such as cluster role assignment from the previous iteration
Azure AD workload identity works especially well with the Azure Identity client library using the Azure SDK and the Microsoft Authentication Library (MSAL) if you're using Azure AD registered applications. Your workload can use any of these libraries to authenticate and access Azure cloud resources seamlessly.
How does it work?
As shown in the following diagram, the Kubernetes cluster becomes a security token issuer, issuing tokens to Kubernetes Service Accounts. These tokens can be configured to be trusted on Azure AD applications and user-defined managed identities. They can then be exchanged for an Azure AD access token using the Azure Identity SDKs or the Microsoft Authentication Library (MSAL).
For more information, see the following resources:
- Azure Workload Identity open-source project
- Use an Azure AD workload identity on Azure Kubernetes Service (AKS
- Deploy and configure workload identity on an Azure Kubernetes Service (AKS) cluster
- Modernize application authentication with workload identity sidecar
- Tutorial: Use a workload identity with an application on Azure Kubernetes Service (AKS)
- Workload identity federation
- Azure Managed Identities with Workload Identity Federation
- Azure AD workload identity federation with Kubernetes
- Azure Active Directory Workload Identity Federation with external OIDC Identy Providers
- Minimal Azure AD Workload identity federation
Prerequisites
- For online deployments, you need an Azure account. If you don't have one, create a free Azure account before you begin.
- Install Microsoft Visual Studio 2022 or later with .NET Standard 6.0. For more information, see Visual Studio Tools for Docker.
- Install Docker for Windows and configure it to use Linux containers.
Architecture
This sample provides a Bicep and an ARM template to deploy a public or a private AKS cluster with API Server VNET Integration with Azure CNI network plugin and Dynamic IP Allocation. In a production environment, we strongly recommend deploying a private AKS cluster with Uptime SLA. For more information, see private AKS cluster with a Public DNS address. Alternatively, you can deploy a public AKS cluster and secure access to the API server using authorized IP address ranges.
Both the Bicep and ARM template deploy the following Azure resources:
- Microsoft.ContainerService/managedClusters: A public or private AKS cluster composed of a:
system
node pool in a dedicated subnet. The default node pool hosts only critical system pods and services. The worker nodes have node taint which prevents application pods from beings scheduled on this node pool.user
node pool hosting user workloads and artifacts in a dedicated subnet.
- Microsoft.Network/virtualNetworks: a new virtual network with six subnets:
SystemSubnet
: this subnet is used for the agent nodes of thesystem
node pool.UserSubnet
: this subnet is used for the agent nodes of theuser
node pool.PodSubnet
: this subnet is used to allocate private IP addresses to pods dynamically.ApiServerSubnet
: API Server VNET Integration projects the API server endpoint directly into this delegated subnet in the virtual network where the AKS cluster is deployed.AzureBastionSubnet
: a subnet for the Azure Bastion Host.VmSubnet
: a subnet for a jump-box virtual machine used to connect to the (private) AKS cluster and for the private endpoints.
- Microsoft.ManagedIdentity/userAssignedIdentities: a user-defined managed identity used by the AKS cluster to create additional resources like load balancers and managed disks in Azure.
- Microsoft.Compute/virtualMachines: Bicep modules create a jump-box virtual machine to manage the private AKS cluster.
- Microsoft.Network/bastionHosts: a separate Azure Bastion is deployed in the AKS cluster virtual network to provide SSH connectivity to both agent nodes and virtual machines.
- Microsoft.Storage/storageAccounts: this storage account is used to store the boot diagnostics logs of both the service provider and service consumer virtual machines. Boot Diagnostics is a debugging feature that allows you to view console output and screenshots to diagnose virtual machine status.
- Microsoft.ContainerRegistry/registries: an Azure Container Registry (ACR) to build, store, and manage container images and artifacts in a private registry for all container deployments.
- Microsoft.KeyVault/vaults: an Azure Key Vault used to store secrets, certificates, and keys that can be mounted as files by pods using Azure Key Vault Provider for Secrets Store CSI Driver. For more information, see Use the Azure Key Vault Provider for Secrets Store CSI Driver in an AKS cluster and Provide an identity to access the Azure Key Vault Provider for Secrets Store CSI Driver. The sample application retrieves parameters stored as secrets in Azure Key Vault.
- Microsoft.DocumentDB/databaseAccounts: an Azure Cosmos DB account, database, and container used by the sample application to store data.
- Microsoft.ServiceBus/namespaces: an Azure Service Bus namespace and a queue where a message is sent whenever the backend application adds, deletes, or updates an item in the Azure Cosmos DB database.
- Microsoft.Network/privateEndpoints: an Azure Private Endpoints is created for each of the following resources:
- Azure Container Registry
- Azure Cosmos DB
- Azure Service Bus
- Azure Key Vault
- Azure Storage Account
- API Server when deploying a private AKS cluster.
- Microsoft.Network/privateDnsZones: an Azure Private DNS Zone is created for each of the following resources:
- Azure Container Registry
- Azure Cosmos DB
- Azure Service Bus
- Azure Key Vault
- Azure Storage Account
- API Server when deploying a private AKS cluster.
- Microsoft.Network/networkSecurityGroups: subnets hosting virtual machines and Azure Bastion Hosts are protected by Azure Network Security Groups that are used to filter inbound and outbound traffic.
- Microsoft.OperationalInsights/workspaces: a centralized Azure Log Analytics workspace is used to collect the diagnostics logs and metrics from all the Azure resources:
- Azure Kubernetes Service cluster
- Azure Cosmos DB
- Azure Service Bus
- Azure Key Vault
- Azure Network Security Groups
- Azure Container Registry
- Azure Storage Account
- Azure virtual machine
- Microsoft.Resources/deploymentScripts: a deployment script is used to run the
install-helm-charts.sh
Bash script which installs the following packages to the AKS cluster via Helm. For more information on deployment scripts, see Use deployment scripts in Bicep
NOTE
You can find thearchitecture.vsdx
file used for the diagram under thevisio
folder.
Visual Studio Solution
This sample provides a Visual Studio solution under the src
folder that contains the following projects:
TodoWeb
: this project is an ASP.NET Web application written in C# using .NET Standard 6.0. This project contains the code of the frontend application. The user interface is composed of a set of Razor pages that can be used to browse, create, delete, update and see the details of a collection of todo items stored in a Cosmos DB database. The frontend service is configured to send logs, events, traces, requests, dependencies and exceptions toApplication Insights
.TodoApi
: this project contains the code of an ASP.NET REST API invoked by the frontend application to access the data stored in the Cosmos DB database. Each time a CRUD operation is performed by any of the methods exposed bu theTodoController
, the backend service sends a notification message to aService Bus queue
. You can use my Service Bus Explorer to read messages from the queue. The frontend service is configured to send logs, events, traces, requests, dependencies and exceptions toApplication Insights
. The backend service adopts Swagger/OpenAPI to expose a machine-readable representation of its RESTful API.
Infrastructure Deployment
You can use the deploy.sh
Bash script under the bicep
folder to deploy the infrastructure using Bicep modules, or the deploy.sh
Bash script under the arm
folder to deploy the infrastructure using the ARM template. In both cases, make sure to change the name of the AKS cluster in the deploy.sh
Bash script and substitute the placeholders in the azuredeploy.parameters.json
file with meaningful values. Also, make sure to enable the following public preview features before deploying the ARM template:
- PodSecurityPolicyPreview
- RunCommandPreview
- EnablePodIdentityPreview
- EnablePrivateClusterPublicFQDN
- PodSubnetPreview"
- AutoUpgradePreview
- EnableOIDCIssuerPreview
- EnableWorkloadIdentityPreview
- EnableImageCleanerPreview
The deploy.sh
deployment script automatically registers the above preview features.
You can deploy the Bicep modules in the bicep
folder using the deploy.sh
Bash script in the same folder. Specify a value for the following parameters in the deploy.sh
script and main.parameters.json
parameters file before deploying the Bicep modules.
prefix
: specifies a prefix for the AKS cluster and other Azure resources.authenticationType
: specifies the type of authentication when accessing the Virtual Machine.sshPublicKey
is the recommended value. Allowed values:sshPublicKey
andpassword
.vmAdminUsername
: specifies the name of the administrator account of the virtual machine.vmAdminPasswordOrKey
: specifies the SSH Key or password for the virtual machine.aksClusterSshPublicKey
: specifies the SSH Key or password for AKS cluster agent nodes.aadProfileAdminGroupObjectIDs
: when deploying an AKS cluster with Azure AD and Azure RBAC integration, this array parameter contains the list of Azure AD group object IDs that will have the admin role of the cluster.keyVaultObjectIds
: Specifies the object ID of the service principals to configure in Key Vault access policies.
We suggest reading sensitive configuration data such as passwords or SSH keys from a pre-existing Azure Key Vault resource. For more information, see Use Azure Key Vault to pass secure parameter value during Bicep deployment.
Application Architecture
The following diagram shows the architecture of the application on an AKS cluster with the OIDC Issuer and Workload Identity enabled. The figure also shows how both the frontend and backend applications exchange the security tokens issued by the AKS cluster to their service account with a security token issued by Azure AD to the federated managed identity and how they use these tokens to access the following Azure AD protected resources:
- Azure Key Vault
- Azure CosmosDB
- Azure Storage Account
- Azure Service Bus
There are four steps to get the sample working end-to-end in a Kubernetes cluster:
- Configure the AKS cluster to issue tokens. An OIDC discovery document is published to allow validation of these tokens.
- Configure their deployments to use a Kubernetes service account federated with a user-assigned managed identity.
- Configure the user-assigned managed identity to trust the security tokens issued by the OIDC issuer of the AKS cluster.
- At runtime, AKS-issued tokens are exchanged for Azure AD tokens, and used to access Azure AD protected resources.
Azure Identity SDK
The sample Todolist
application makes use of the Azure Identity client library for .NET with the DefaultAzureCredential to authenticate via Azure AD Workload Identity and get a security token to access the following services:
- Azure Key Vault
- Azure CosmosDB
- Azure Storage Account
- Azure Service Bus
Here are some snippets from the application code that show how to access Azure resources using the Azure Identity client library for .NET and Azure AD workload identity for Kubernetes.
Azure Key Vault
Azure Cosmos DB
Azure Storage Account
Azure Service Bus
Application Configuration
The Bicep and ARM templates create all the necessary secrets used by both the frontend and backend applications as shown in the picture below:
Azure Key Vault secret names are limited to alphanumeric characters and dashes. Hierarchical values in ASP.NET configuration sections use -- (two dashes) as a delimiter, as colons aren't allowed in key vault secret names. Colons delimit a section from a subkey in ASP.NET Core configuration. The two-dash sequence is replaced with a colon when the secrets are loaded into the app's configuration. For more information, see Secret storage in the Production environment with Azure Key Vault.
The following table describes the application settings:
Name | Description |
---|---|
ApplicationInsights--InstrumentationKey | Specifies the Instrumentation Key of the Azure Application Insights resource used by both the frontend and backend services. |
DataProtection--BlobStorage--AccountName | Specifies the name of the Azure Storage Account used to store the boot diagnostics logs or the virtual machine and the files used by the sample ASP.NET frontend and backend applications for ASP.NET Data Protection. This setting is used when the DataProtection--BlobStorage--UseAzureCredential is set to true . |
DataProtection--BlobStorage--ConnectionString | Contains the connection string to the Azure Storage Account . This setting is used when the DataProtection--BlobStorage--UseAzureCredential is set to false . |
DataProtection--BlobStorage--UseAzureCredential | Specifies whether the frontend and backend services should use Azure Identity client library for .NET with the DefaultAzureCredential to authenticate via Azure AD Workload Identity and get a security token to access the Azure Storage Account . If the value of this setting is equal to false , the frontend and backend applications will use the connection string of the Azure Storage Account contained in the DataProtection--BlobStorage--ConnectionString secret. |
NotificationService--ServiceBus--ConnectionString | Contains the connection string of the Azure Service Bus namespace. This setting is used when the NotificationService--ServiceBus--UseAzureCredential is set to false . |
NotificationService--ServiceBus--Namespace | Contains the name of the Azure Service Bus namespace. This setting is used when the NotificationService--ServiceBus--UseAzureCredential is set to true . |
NotificationService--ServiceBus--QueueName | Contains the name of the queue in the Azure Service Bus namespace where the backend service sends a message any time a todo-item is created in the Azure Cosmos DB database by the backend service. |
NotificationService--ServiceBus--UseAzureCredential | Specifies whether the frontend and backend services should use Azure Identity client library for .NET with the DefaultAzureCredential to authenticate via Azure AD Workload Identity and get a security token to access the Azure Service Bus namespace . If the value of this setting is equal to false , the frontend and backend applications will use the connection string of the Azure Service Bus namespace contained in the NotificationService--ServiceBus--ConnectionString secret. |
RepositoryService--CosmosDb--CollectionName | Specifies the name of the Azure Cosmos DB container. |
RepositoryService--CosmosDb--DatabaseName | Specifies the name of the Azure Cosmos DB database. |
RepositoryService--CosmosDb--EndpointUri | Specifies the endpoint URI of the Azure Cosmos DB account. |
RepositoryService--CosmosDb--PrimaryKey | Contains the primary key of the Azure Cosmos DB account. This setting is used when the RepositoryService--CosmosDb--UseAzureCredential is set to false . |
RepositoryService--CosmosDb--UseAzureCredential | Specifies whether the frontend and backend services should use Azure Identity client library for .NET with the DefaultAzureCredential to authenticate via Azure AD Workload Identity and get a security token to access the Azure Cosmos DB account . If the value of this setting is equal to false , the frontend and backend applications will use the primary key of the Azure Cosmos DB account contained in the RepositoryService--CosmosDb--PrimaryKey secret. |
Application Local Debugging
To debug the application locally, you must install Microsoft Visual Studio 2022 or later with .NET Standard 6.0. For more information, see Visual Studio Tools for Docker. In addition, make sure to change the values in the docker-compose.override.yml
as follows:
- You can use the same or an alternative identity to access the secrets from Key Vault that contain the credentials to access the Azure Service Bus namespace, Azure Cosmos DB, and Azure Storage Account. In this case, make sure to specify the name of the Key Vault in the corresponding environment variable and assign the
List
andGet
permissions to the Azure AD application via access policies or Azure RBAC. - As an alternative, you can use the environment variables to specify the credentials to access the Azure Service Bus namespace, Azure Cosmos DB, and Azure Storage Account in the
docker-compose.override.yml
file (not recommended).
Application Deployment
The 00-variables.sh
contains the variables used by all the scripts necessary to deploy the sample. The first step consists in assigning a proper value to each variable before starting with the application deploying.
Run the scripts/01-build-container-images.sh
script to build the Linux container images for the frontend and backend applications using Docker.
Run the scripts/02-push-docker-images.sh
script to push the container images to your Azure Container Registry.
Run the scripts/03-enable-oidc.sh
script on an existing AKS cluster, to register and enable the OIDC Issuer feature. Running this script is unnecessary if you deployed the AKS cluster using the bicep modules in the bicep
folder.
Run the scripts/04-enable-workload-identity.sh
script on an existing AKS cluster, to register and enable the Workload Identity feature. Running this script is unnecessary if you deployed the AKS cluster using the bicep modules in the bicep
folder.
Run the scripts/05-install-azure-ad-workload-cli.sh
script to install the Azure AD Workload CLI (azwi). azwi
is a utility CLI that helps manage Azure AD Workload Identity and automate error-prone operations:
- Generate the JWKS document from a list of public keys
- Streamline the creation and deletion of the following resources:
- AAD applications
- Kubernetes service accounts -Federated identities
- Azure role assignments
The Bicep and ARM templates automatically create the secrets used by the frontend and backend services. Hence, running this script is unnecessary if you deployed the AKS cluster using the bicep modules in the bicep
folder. If you want to deploy the sample on your AKS cluster, you can use the scripts/06-create-key-vault-and-secrets.sh
script to create the secrets in your Azure Key Vault.
The next step consists in creating a user-assigned managed identity and granting permissions to:
- Get and list secrets in Azure Key Vault
- Read and write data in Azure Cosmos DB
- Read and write blobs in the Azure Storage Account
- Send and receive messages to queues in a Service Bus namespace
You can run the scripts/07-create-aad-application.sh
script to create the Azure AD application and grant permissions. Running this script is unnecessary if you deployed the AKS cluster using the bicep modules in the bicep
folder because the modules automatically create the user-assigned managed identity and role assignments.
You can use the scripts/08-create-service-account.sh
script to create a Kubernetes service account in the application namespace and annotate it with the client ID of the user-assigned managed identity. The last step of the script establishes federated identity credential between the managed identity and the service account issuer and subject. This service account is used by the Kubernetes deployment of both the frontend and backend services. For more details, see the scripts/todolist.yml
YAML manifest or the Helm chart under the chart
folder.
You can use the scripts/09-create-nginx-ingress-controller.sh
script to install the NGINX Ingress Controller on your AKS cluster via a Helm chart. NGINX Ingress controller works with both NGINX and NGINX Plus and supports the standard Ingress features - content-based routing and TLS/SSL termination. In this sample, the NGINX Ingress controller is used to publicly expose both the frontend and backend service. Normally, the backend service should not be directly exposed to the public internet, but for demo purpose, the REST API is publicly exposed. The Bicep and ARM templates in the bicep
folder use a deployment script
to bootstrap the cluster and install the following Helm packages:
Hence, running the scripts/09-create-nginx-ingress-controller.sh
script is unnecessary if you deployed the AKS cluster using the bicep modules in the bicep
folder.
If you use a public Azure DNS to host your DNS domain and manage your DNS records as described in the Host your domain in Azure DNS tutorial, you can use the scripts/10-configure-dns-records.sh
script to create A records for both the frontend and backend services. Feel free to change the name of the frontend and backend subdomains in the script below.
You can use the scripts/11-install-cert-manager.sh
script to install cert-manager on your AKS cluster via a Helm chart. cert-manager
adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using those certificates. It can issue certificates from a variety of supported sources, including Let’s Encrypt, HashiCorp Vault, and Venafi as well as private PKI. It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry. In this sample, cert-manager
is configured to issue certificates for the frontend and backend service from Let’s Encrypt. Running this script is unnecessary if you deployed the AKS cluster using the bicep modules in the bicep
folder.
You can use the scripts/12-create-default-backend.sh
script to deploy a default backend for the NGINX ingress controller
. This step is facultative and not necessary.
You can use the scripts/13-deploy-workload-using-helm.sh
script to deploy the application to your AKS cluster via a Helm chart to the target namespace that contains the service account used by the frontend and backend Kubernetes deployments. Alternatively, you can use the scripts/14-deploy-workload-using-kubectl.sh
script to deploy the sample application using kubectl and YAML manifests.
Testing
If you properly deployed and configured both the frontend and backend application in the same namespace on your AKS cluster, and properly exposed these services via the NGINX Ingress Controller and Azure DNS, you should be able to access both services, as shown in the following pictures: