This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .
Introduction
When you are building an AKS cluster for your team, one of the first questions you need to ask is: how are you going to manage access to the different groups or people? How to have something simple to manage but still secure?
When using the Azure Portal to create a new AKS instance, it gives you with the following options:
- Local accounts with Kubernetes RBAC
- Azure AD authentication with Kubernetes RBAC
- Azure AD authentication with Azure RBAC
These options are summarized in this doc and its referenced articles. If you don't have so much experience with Kubernetes and Azure yet, the official documentation can be overwhelming. This article aims to help you decide which option is best for your case and provide an "easier" way to understand the official documentation.
The scenario
Let's assume you are the cluster admin/owner, and this new AKS cluster will be used by many different developer teams to deliver their applications. Therefore, you are planning to:
- Create a new namespace in AKS for each of the developer teams
- Assign namespace permissions to each team. Then, each dev team is sub-divided into 2 groups:
- Namespace users group => people here will be able to deploy and edit applications within the namespace but not assign access to others
- Namespace admins group => people here will be able to do everything the previous group does but also assign/remove access to others within this namespace. This is useful so the main Cluster Admins do not need to keep managing access to every namespace in the cluster. You delegate that for each team.
This is a very common yet simple scenario when building an AKS cluster which will be shared with other teams. So, how this case looks like in practice on each RBAC option available in AKS?
To give real values to the scenario above, here the details we will use for the rest of the article:
- Namespace name: blog
- Azure AD group with cluster admin permissions: aks-admins
- Azure AD group with namespace admin permissions: aks-blog-admins
- Azure AD group with namespace user permissions: aks-blog-users
Local accounts with Kubernetes RBAC
With this option, there is no integration between Azure Active Directory and the AKS cluster. That means you can't have a specific group of users in AD mapped to a specific namespace inside your AKS cluster. You need to utilize one of the native Kubernetes ways like using client certificates, bearer tokens, etc. More on this here in the official documentation.
Or, you could also use the Az CLI command az aks get-credentials to fetch local kubeconfig credentials if you are part of one of the AKS built-in roles but that will give all users the same access (clusterAdmin or clusterUser) inside the cluster. There is no way to differentiate the users inside Kubernetes if Azure AD is not enabled when using this method.
I would only recommend creating clusters with this configuration if all of the users are not in Azure AD and have no way to be included/invited to, for some reason. User management in this scenario becomes very challenging.
Kubernetes API server supports integration with OpenID Connect providers exactly to make it easier to manage users outside of Kubernetes. Let's put more focus this article in the other two options with Azure AD integration enabled.
Azure AD authentication with Kubernetes RBAC
The first option with Azure AD integration makes AKS delegates authentication to Azure AD, not authorization. It means AKS trusts Azure AD in respect to "who is logging in". But the list of permissions (what actions the users are authorized to do inside the AKS cluster) must still be defined inside the cluster and not from the Azure AD roles and permissions system.
I am a person that learns mostly by a hands-on approach. So let's try to make things clearer from a practical standpoint. What steps need to be performed in an AKS cluster to accomplish what I described in the scenario above?
Prerequisites
- Az CLI
- Kubectl
- Basic understanding of Azure AD users and groups
- Make sure you have the cluster created or updated to use Azure AD and the admin group is correctly to set aks-admins. More on this here
- Make sure both groups aks-blog-admins and aks-blog-admins are allowed to fetch the cluster credentials using "az aks get-credentials" by going to Azure Portal => AKS => Access Control (IAM) and assign role "Azure Kubernetes Service Cluster User Role"
- There is a namespace called blog in the cluster. If you don't have it yet, just start by grabbing your kubectl credentials by using az aks get-credentials using one of the admin users of aks-admins and then use kubectl:
kubectl create namespace blog
Steps
1) Next step is to create a Kubernetes RoleBinding YAML file with the following content:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: blog-admin-binding
namespace: blog
subjects:
- kind: Group
name: d89e6a33-eaca-4be4-b1e5-9f5c74d6a35c
#Azure AD group name: aks-blog-admins
namespace: blog
roleRef:
kind: ClusterRole
name: admin
apiGroup: rbac.authorization.k8s.io
Apply the YAML manifest:
kubectl apply -f <<filename.yaml>>
Pay attention to line number 8: that is the group object id from Azure AD. It means any user in this cluster that belongs to that group will get the built-in Kubernetes admin role (line 13) for the namespace blog (line 10)
2) At this point, any users from aks-blog-admins group can now login in the cluster and have full access rights to the blog namespace. Now, they can create RoleBindings within that namespace to assign permissions to other users. Let's assign access now to a developer that will have full access to blog namespace except permissions to create RoleBindings:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: blog-user-binding
namespace: blog
subjects:
- kind: Group
name: 996844bd-95a3-48fc-93e9-7f1f1177ff83
#Azure AD group name: aks-blog-users
namespace: blog
roleRef:
kind: ClusterRole
name: edit
apiGroup: rbac.authorization.k8s.io
Apply the YAML manifest:
kubectl apply -f <<filename.yaml>>
Note that this new RoleBinding assigns the built-in role edit (line 13) instead of admin to the group aks-blog-users (line 8).
3) From now on, the authorization is correctly set up inside the AKS cluster. You should use the Azure AD groups to manage people (add and remove) from the groups for the given namespace
One thing to note on both YAML files is that we can't use the friendly Azure AD group name but always the group object id. That's why it is recommended that in your YAML files you add a comment line describing the group name. Be aware the commented lines will be removed by Kubernetes when applying the manifests in the cluster so you would need to look in the source control files.
Azure AD authentication with Azure RBAC
In simple words, Azure RBAC will take the Azure AD integration one step further and will take care of both authentication and authorization inside an AKS cluster. You don't need to create any YAML manifests to manage user access in the namespaces, for example. The idea here is to work similarly as the other Azure services by using Azure AD roles (IAM) only. Again, to make things clearer let's replicate the same scenario as we did previously for Kubernetes RBAC.
Prerequisites
- Az CLI
- Basic understanding of Azure AD users and groups
- Make sure you have the cluster created or updated to use Azure AD and Azure RBAC. Assign the IAM role "Azure Kubernetes Service RBAC Cluster Admin" to the group aks-admins. Check this doc for more information
- Make sure both groups aks-blog-admins and aks-blog-admins are allowed to fetch the cluster credentials using "az aks get-credentials" by going to Azure Portal => AKS => Access Control (IAM) and assign role "Azure Kubernetes Service Cluster User Role"
- Make sure there is a namespace called blog in the cluster. If you don't have it yet, just start by grabbing your kubectl credentials by using az aks get-credentials using one of the admin users of aks-admins and then use kubectl:
kubectl create namespace blog
Steps
1) Second step is to assign another IAM role called "Azure Kubernetes Service RBAC Cluster Admin" to aks-blog-admins. There is one "catch" though: you can't use the Portal and assign this role using the Access Control (IAM) blade from the AKS service because it will assign that role for all namespaces inside the cluster, not just for namespace blog. You need to use the Az CLI to assign that RBAC Cluster Admin scoped only to a specific namespace:
az role assignment create --role "Azure Kubernetes Service RBAC Admin" --assignee <<groud-id for aks-blog-admins>> --scope /subscriptions/<<subscription>>/resourcegroups/<<rg>>/providers/Microsoft.ContainerService/managedClusters/<<cluster-name>>/namespaces/blog
Note that the --scope property assigns that role only for the blog namespace. One downside for this approach is that you can't see this role assignment in the Portal either. You will need to utilize Az CLI to see the scopes you have assigned for namespaces:
az role assignment list --scope /subscriptions/<<subscription>>/resourcegroups/<<rg>>/providers/Microsoft.ContainerService/managedClusters/<<cluster-name>>/namespaces/blog
2) At this point, users from group aks-blog-admins can login using az aks get-credentials and can perform any operations in the namespace using tools like kubectl. All right - but how can can they manage access to other regular non-admin users for this namespace and not depend on Kubernetes YAML rolebindings for that?
3) Using an Owner user for your cluster, assign the IAM role "User Access Administrator" to the group aks-blog-admins, then they are able to assign IAM roles to other users for the cluster. Here we have the same problem as step 2: if you use the Portal, you will be giving the group rights to manage access to the entire cluster. You need to again use the Az CLI to assign that role just for blog namespace:
az role assignment create --role "User Access Administrator" --assignee <<groud-id for aks-blog-admins>> --scope /subscriptions/<<subscription>>/resourcegroups/<<rg>>/providers/Microsoft.ContainerService/managedClusters/<<cluster-name>>/namespaces/blog
4) Now, users under aks-blog-admins have permissions to assign Azure RBAC roles to other users for the scope of the blog namespace only. They can execute this Az CLI command to assign "Azure Kubernetes Service RBAC Writer" to group aks-blog-users:
az role assignment create --role "Azure Kubernetes Service RBAC Writer" --assignee <<groud-id for aks-blog-users>> --scope /subscriptions/<<subscription>>/resourcegroups/<<rg>>/providers/Microsoft.ContainerService/managedClusters/<<cluster-name>>/namespaces/blog
And that's it. After that, any Azure user under group aks-blog-users can get their cluster credentials using az aks get-credentials and perform writing operations in the namespace, but they can't give access to others because this group doesn't have User Access Administrator IAM role like the admin group. For description on what each Azure RBAC role allows inside a AKS cluster, check here
Conclusion and recommendations
The difference between the options here can be summarized as "how much" of Azure RBAC is used in AKS when it comes to authorization and authentication. There is no absolute right choice, it all depends on your needs. Here are some deciding factors that might support you choosing one option over the others:
Local accounts with Kubernetes RBAC:
- Only use this if the users of the AKS cluster have no possibility to be in Azure AD for some reason
- Managing users in "raw" Kubernetes becomes really complex with big teams
Azure AD authentication with Kubernetes RBAC:
- Choose this option if you want to use Azure RBAC just to decide who will be able to get AKS credentials but Kubernetes YAML manifests to describe what these users can do inside the cluster
- Your cluster becomes more portable because it contains all the role bindings definition in it, even if these bindings contain Azure-specific group IDs and users in their definitions
- Checking who have access to what inside the cluster is not so easy when working with AD groups because you need to work with group IDs in the YAML and not their display names; make sure to save your YAML definitions in a source control with proper line commenting to make that correlation easier (as described in the steps earlier)
Azure AD authentication with Azure RBAC:
- Choose this option if you want to use Azure RBAC just to decide who and what users can do inside the cluster. This is a YAML-free option to deal with user access in AKS
- To give/list permissions to specific namespaces, you need to use the Az CLI for the moment. There is no option in the Portal yet to manage that. The Access Control (IAM) blade for AKS assigns roles for the entire cluster
References
Concepts - Access and identity in Azure Kubernetes Services (AKS) - Azure Kubernetes Service
Manage Azure RBAC in Kubernetes From Azure - Azure Kubernetes Service
Use Azure AD and Kubernetes RBAC for clusters - Azure Kubernetes Service