This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
The official documentation on how to implement right level of security on your AKS cluster and it’s different components is here: Azure security baseline for Azure Kubernetes Service. This doc is exhaustive and explains various aspects of AKS security and related actions to be taken in detail. In this current article (or series of articles) we are going to dive a bit deeper on some areas of AKS security and will show how to achieve those security standards as explained in the official document. The areas we are going to cover are:
• CIS Benchmark
• AppArmor and seccomp
• Audit Logs
• Network policy
• Securing Egress and Ingress
CIS (Center for Internet Security) Benchmark is a set of configuration guidelines created and maintained by cybersecurity experts to safeguard against modern-day cyber threats. This is primarily a volunteering effort and contributed by SMEs from various products and technologies. At the time of authoring this blog post, there exist more than 100 CIS Benchmarks across 25+ vendor product families. For more information on CIS Benchmark please read CIS Benchmark FAQ. CIS Benchmark became somewhat de facto industry standard for cybersecurity. You can download CIS Benchmark for Kubernetes from here. If you go through CIS Benchmark for Kubernetes document, you will find there are instructions for both master nodes and worker nodes. But in case of AKS, we can’t access the master nodes and for our benefit we also have an exclusive CIS Benchmark document for AKS as well.
I recommend you download and read through the benchmark document on AKS. You will see there is only one instruction for master nodes and that is related to switching on the Audit Logs. We are going to cover Audit Logs anyway in this series. Let’s discuss what shall we do to cover the part related to worker nodes. But before we roll up our sleeves, I recommend you go through Security Hardening for AKS node Host OS. It also refers some CIS benchmarks already configured by Microsoft.
Setting up the Environment
As you can see in CIS Benchmark document for AKS, one of the basic criteria to achieve the benchmarks is to get into the worker nodes through SSH. Let’s do that first.
Step 1
Open Azure portal and go to your AKS Cluster and click on connect. Note the first 2 commands from right hand side.
Step 2
Open Azure Cloud Shell. Run those 2 commands sequentially to connect to your AKS cluster and run command “kubectl get nodes -o wide”. Note the internal IPs of the worker nodes. We will need those later.
Step 3
Get the Node Resource Group (resource group starts with MC_) and the VM Scale Set associated with it. Here, name of my resource group is “aksgithubcicdrg” and AKS cluster name is “aksgithubcicd”.
NODE_RESOURCE_GROUP=$(az aks show --resource-group aksgithubcicdrg --name aksgithubcicd --query nodeResourceGroup -o tsv)
SCALE_SET_NAME=$(az vmss list --resource-group $NODE_RESOURCE_GROUP --query [0].name -o tsv)
Step 4
Configure SSH to the worker nodes:
az vmss extension set --resource-group $NODE_RESOURCE_GROUP --vmss-name $SCALE_SET_NAME --name VMAccessForLinux --publisher Microsoft.OSTCExtensions --version 1.4 --protected-settings "{\"username\":\"azureuser\", \"ssh_key\":\"$(cat ~/.ssh/id_rsa.pub)\"}"
az vmss update-instances --instance-ids '*' --resource-group $NODE_RESOURCE_GROUP --name $SCALE_SET_NAME
Step 5
We have now SSH configured to worker nodes, but we cannot connect to those as they have internal IPs only. Hence, we will provision a pod to connect to them:
kubectl run aks-ssh --image=nginx --restart=Never
Now, exec to the pod and install OpenSSH client and exit from the pod:
kubectl exec -it aks-ssh -- bash
apt-get update && apt-get install openssh-client -y
exit
Step 6
Copy RSA key to the Pod and then use it to SSH to one of the Nodes from the pod using the internal IP:
kubectl cp ~/.ssh/id_rsa $(kubectl get pod -l run=aks-ssh -o jsonpath='{.items[0].metadata.name}'):/id_rsa
kubectl exec -it aks-ssh -- bash
chmod 0600 id_rsa
ssh -i id_rsa azureuser@10.0.0.4
Configure CIS Benchmark on Worker Nodes
Now as you can already access all the individual nodes, it’s time to implement CIS Benchmark on the worker nodes as stated in CIS Benchmark document for AKS. We can check all the suggestions given in the document one by one and take actions to remediate. But prior to that, let’s see what is already configured. For this we will take help something called Kube-Bench. It’s a tool that does the work for you – it can check whether all the standards as recommended in CIS Benchmark are configured or not.
Step 7
SSH to one of your worker nodes and install Docker:
sudo snap install docker
Check this GitHub article: https://github.com/aquasecurity/kube-bench/blob/main/docs/running.md. And run the first command given under heading “Running Inside a Container”.
sudo docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t aquasec/kube-bench:latest --version 1.18
Here is the summary of the checks for one of my nodes:
Step 8
As we can see one check is failed, let’s see which one.
Now, go back to CIS Benchmark document for AKS and check under section 4.1.5 and take the remediation action given in there.
In this way you can check other configurations as given in the document as well and take corresponding remediation actions. That’s pretty much it. Don’t forget to uninstall docker once you are done (sudo snap remove docker). We will talk about AppArmor and Seccomp in next part of this series.