This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .
A number of organizations adopting microservices architectures have looked to service meshes as one of the tools that help these organizations in enforcing enterprise-wide policies and rules for traffic, security, reliability, and monitoring. These are generally categorized into Security, Reliability and Observability. While a networking solution other than service mesh can provide some of these features, it won’t do it without disrupting the application code or existing infrastructure. Service Mesh provides an independent infrastructure layer that can be managed by a separate team and is least intrusive. It deploys a sidecar container to pods by design, which makes it very powerful for monitoring your services within a kubernetes cluster.
Open Service Mesh (OSM) is a lightweight, extensible service mesh that is utilized to secure communication of services running in a Kubernetes environment. To achieve interoperability between service meshes, a common interface standard SMI (Service Mesh Interface) has been defined by CNCF. OSM is Microsoft's implementation of the Service Mesh Interface in an actual service mesh. It is easy to install, maintain and operate. It is also effortless to troubleshoot and less complicated to configure with SMI.
The Open Service Mesh extension is a managed service mesh for Arc-enabled Kubernetes clusters that is lightweight and extensible. It can be configured with Service Mesh Interface APIs, works by injecting envoy proxy as a sidecar to each application instance, and brings a new Azure Portal experience for onboarding.
Some of the use cases you can solve with OSM are:
- Service to Service communication over mutual TLS authentication
- Access control for service-to-service communication
- Traffic management for canary deployments
- Observability for services
- Traffic management capabilities like auto retries, rate limiting, circuit breaking
- Multi-cluster communication over mTLS.
With the OSM extension being enabled through Azure Arc, OSM can be applied to your on-premises AKS hybrid environment to manage communication and secure your cloud-native workload.
In this article, the OSM functionalities were applied to the communication between the bookbuyer and bookstore sample applications. First, traffic access between the bookbuyer and bookstore services was disabled, but only got enabled by applying an SMI traffic policy. Additionally, the traffic-splitting capability was also implemented by creating a second bookstore service and then weighted traffic from the bookbuyer to bookstore-v1 and bookstore-v2 respectively.
Setting up OSM
The latest OSM version (v1.2.0) used for this article requires a kubernetes cluster running v1.22.9 or higher and a Kubernetes command-line tool. It also requires a workstation that is capable of executing bash commands/scripts and has the OSM code repo available locally.
For this work, the setup was done in an Arc-enabled Aks cluster with WSL-2 (windows subsystem for linux) enabled. To setup your own Aks hybrid cluster, see Use PowerShell to set up Kubernetes on Azure Stack HCI and Windows Server clusters - AKS-HCI | Microsoft Learn. This should have Az CLI running, but you can check using az version.
To get things going, bring up your cluster.
PS C:\Users\ekele> Get-AksHciCluster
Status : {ProvisioningState, Details}
ProvisioningState : Deployed
KubernetesVersion : v1.23.8
PackageVersion : v1.23.8
NodePools : linuxpool
WindowsNodeCount : 0
LinuxNodeCount : 3
ControlPlaneNodeCount : 3
ControlPlaneVmSize : Standard_A2_v2
AutoScalerEnabled : False
AutoScalerProfile :
LoadBalancer : {VMSize, Count, Sku}
Name : <ClusterName>
The next step is to enable OSM kubernetes extension on the AKS hybrid cluster after connecting the cluster to Azure Arc.
Note: In order to run some bash commands for validation purposes, the rest of the cmdlets used in this article were run in Git bash terminal on Visual Studio Code with Az cli installed. Also, ensure to download the OSM cli to be able to run osm commands. To download the latest OSM cli, see Releases · openservicemesh/osm (github.com)
ea@SA18n30-3 MINGW64 ~ (master)
$ az k8s-extension create --cluster-name <ClusterName> --resource-group <ResourceGroup> --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm
{
"aksAssignedIdentity": null,
"autoUpgradeMinorVersion": true,
"configurationProtectedSettings": {},
"configurationSettings": {},
"customLocationSettings": null,
"errorInfo": null,
"extensionType": "microsoft.openservicemesh",
"id": "/subscriptions/5807bea4-cd17-4b81-b4d8-d4e94caab82d/resourceGroups/<ResourceGroup>/providers/Microsoft.Kubernetes/connectedClusters/<ClusterName>/providers/Microsoft.KubernetesConfiguration/extensions/osm",
"identity": {
"principalId": "0c8ed1d6-358f-4f6a-8342-61d10e1a9571",
"tenantId": null,
"type": "SystemAssigned"
},
"name": "osm",
"packageUri": null,
"provisioningState": "Succeeded",
"releaseTrain": "Stable",
"resourceGroup": "<ResourceGroup>",
"scope": {
"cluster": {
"releaseNamespace": "arc-osm-system"
},
"type": "Microsoft.KubernetesConfiguration/extensions",
"version": "1.1.1-1"
}
Verify that the OSM k8s extension was installed correctly by running
ea@SA18n30-3 MINGW64 ~ (master)
$ az k8s-extension show --cluster-type connectedClusters --cluster-name <ClusterName>
--resource-group <ResourceGroup> --name osm
Get the credentials of your cluster to be sure you can run kubectl using
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl config current-context
<ClusterName>-admin@<ClusterName>
You can further verify that the OSM components are running on your cluster by checking the deployment, pods, and services in arc-osm-system namespace.
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl get deployment -n arc-osm-system
NAME READY UP-TO-DATE AVAILABLE AGE
osm-bootstrap 1/1 1 1 86m
osm-controller 1/1 1 1 86m
osm-injector 1/1 1 1 86m
osm-metrics-agent 1/1 1 1 86m
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl get pods -n arc-osm-system
NAME READY STATUS RESTARTS AGE
osm-bootstrap-58b8986b87-wfp6g 1/1 Running 0 88m
osm-controller-768df46759-n29ps 1/1 Running 0 88m
osm-injector-6ddd574779-wl52s 1/1 Running 0 88m
osm-metrics-agent-78c846d57b-v9vtd 4/4 Running 1 (87m ago) 88m
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl get services -n arc-osm-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
osm-bootstrap ClusterIP 10.99.80.35 <none> 9443/TCP,9095/TCP 88m
osm-controller ClusterIP 10.111.193.40 <none> 15128/TCP,9092/TCP,9091/TCP 88m
osm-injector ClusterIP 10.100.99.169 <none> 9090/TCP 88m
osm-validator ClusterIP 10.108.48.217 <none> 9093/TCP 88m
OSM controller configuration
OSM deploys a MeshConfig resource osm-mesh-config as a part of its control plane in arc-osm-system namespace. The MeshConfig provides the mesh owner or operator the ability to update some of the mesh configurations based on need. The default values of the MeshConfig can be viewed using the following command.
$ kubectl describe meshconfig osm-mesh-config -n arc-osm-system
Name: osm-mesh-config
Namespace: arc-osm-system
Labels: <none>
Annotations: <none>
API Version: config.openservicemesh.io/v1alpha2
Kind: MeshConfig
Metadata:
Creation Timestamp: 2022-10-17T22:56:58Z
Generation: 1
Managed Fields:
API Version: config.openservicemesh.io/v1alpha2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
{ …
Traffic:
Enable Egress: false
Enable Permissive Traffic Policy Mode: true
… }
It can be observed that the permissive traffic policy mode of the output is set at true. With permissive traffic policy mode enabled in your OSM mesh:
- The SMI traffic policy enforcement is bypassed.
- OSM automatically discovers services that are a part of the service mesh.
- OSM programs traffic policy rules on each Envoy proxy sidecar to be able to communicate with these services.
It is important to note that values in the MeshConfig osm-mesh-config are persisted across upgrades. Changes to osm-mesh-config can be made using the kubectl patch command. In a subsequent step for this sample deployment, the permissive traffic policy mode was changed to false through this command.
Deploy Bookstore Applications
After validating the checks, the sample applications manifest (Bookbuyer, Bookstore, Bookwarehouse) is deployed that will generate the services on which OSM will be applied to.
Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookbuyer.yaml
Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookstore.yaml
Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookwarehouse.yaml
Validate Application by port-forwarding the Kubernetes service
The components of the bookstore namespace can be checked to see the pods and services running using ‘kubectl get all -n bookstore’. Afterwards, the logs of the bookbuyer pod could be traced to ensure it is communicating with the bookstore service.
To query that bookbuyer is reaching bookstore service, run the following commands to trail the bookbuyer logs.
POD="$(kubectl get pods -n bookbuyer --show-labels --selector app=bookbuyer --no-headers | grep -v 'Terminating' | awk '{print $1}' | head -n1)"
Kubectl logs “${POD}” -n bookbuyer -c bookbuyer --tail=100 -f
{ … Fetching http://bookstore.bookstore:14001/books-bought
Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]
Identity: bookstore-v1
Booksbought: 5258
Server: n/a
Date: Tue, 18 Oct 2022 04:42:48 GMT
Status: 200 OK
MAESTRO! THIS TEST SUCCEEDED!
Fetching http://bookstore.bookstore:14001/buy-a-book/new
Request Headers: map[]
Identity: bookstore-v1
Booksbought: 5259
Server: n/a
Date: Tue, 18 Oct 2022 04:42:49 GMT
Status: 200 OK
MAESTRO! THIS TEST SUCCEEDED!
… }
The status should show that everything is okay and that it is able to fetch http bookstore-v1 data.
Onboard Bookstore Apps namespaces for OSM
In order to start using OSM capabilities, the application namespaces need to be onboarded to the service mesh so that OSM can manage these namespaces. Run the command
$ osm namespace add bookstore bookbuyer bookwarehouse
Namespace [bookstore] successfully added to mesh [osm]
Namespace [bookbuyer] successfully added to mesh [osm]
Namespace [bookwarehouse] successfully added to mesh [osm]
Enable proxy sidecar injection for Bookstore Apps
The sidecar containers can be enabled using the ‘kubectl rollout restart’ command. You can check the pod details before and after to know that the sidecar proxies were correctly deployed.
$ kubectl get pods -n bookbuyer
NAME READY STATUS RESTARTS AGE
bookbuyer-56d6bcc7f8-gvzn8 1/1 Running 0 86m
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl rollout restart deployment bookbuyer -n bookbuyer
deployment.apps/bookbuyer restarted
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl rollout restart deployment bookstore -n bookstore
deployment.apps/bookstore restarted
ea@SA18n30-3 MINGW64 ~ (master)
$ kubectl rollout restart deployment bookwarehouse -n bookwarehouse
deployment.apps/bookwarehouse restarted
By checking the pod details of the bookbuyer after restart, the ‘Ready’ state shows that there are 2 out of 2 pods running.
$ kubectl get pods -n bookbuyer
NAME READY STATUS RESTARTS AGE
bookbuyer-6ccd656dbf-7c28x 2/2 Running 0 115s
To get further information, the ‘kubectl describe pod’ command will show an envoy running alongside the bookbuyer container.
POD="$(kubectl get pods -n bookbuyer --show-labels --selector app=bookbuyer --no-headers | grep -v 'Terminating' | awk '{print $1}' | head -n1)"
Kubectl describe pod $POD -n bookbuyer
Normal Started 4m29s kubelet Started container bookbuyer
Normal Pulling 4m29s kubelet Pulling image "mcr.microsoft.com/oss/envoyproxy/envoy:v1.19.3"
Normal Pulled 4m26s kubelet Successfully pulled image "mcr.microsoft.com/oss/envoyproxy/envoy:v1.19.3" in 3.413382681s
Normal Created 4m26s kubelet Created container envoy
Normal Started 4m26s kubelet Started container envoy
Verify Bookstore App functioning with sidecar injection
Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f
…
Fetching http://bookstore.bookstore:14001/books-bought
Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]
Identity: bookstore-v1
Booksbought: 415
Server: envoy
Date: Tue, 18 Oct 2022 04:51:14 GMT
Status: 200 OK
MAESTRO! THIS TEST SUCCEEDED!
Fetching http://bookstore.bookstore:14001/buy-a-book/new
Request Headers: map[]
Identity: bookstore-v1
Booksbought: 416
Server: envoy
Date: Tue, 18 Oct 2022 04:51:15 GMT
Status: 200 OK
MAESTRO! THIS TEST SUCCEEDED!
…
Disable OSM Permissive Mode
If the permissive traffic policy mode is disabled, communication between the bookbuyer and bookstore will cease. This can be done using the following command
$ kubectl patch meshconfig osm-mesh-config -n arc-osm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}' --type=merge
meshconfig.config.openservicemesh.io/osm-mesh-config patched
Verify that the bookbuyer can no longer reach the bookstore
Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f
{ …
Fetching http://bookstore.bookstore:14001/books-bought
Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]
Error fetching http://bookstore.bookstore:14001/books-bought: Get "http://bookstore.bookstore:14001/books-bought": dial tcp 10.99.49.182:14001: connect: connection
refused
ERROR: response code for "http://bookstore.bookstore:14001/books-bought" is 0; expected 200
Fetching http://bookstore.bookstore:14001/buy-a-book/new
Request Headers: map[]
Error fetching http://bookstore.bookstore:14001/buy-a-book/new: Get "http://bookstore.bookstore:14001/buy-a-book/new": dial tcp 10.99.49.182:14001: connect: connection refused
ERROR: response code for "http://bookstore.bookstore:14001/buy-a-book/new" is 0;
expected 200
… }
Deploy SMI Traffic Access Policies
To restore traffic access between the bookbuyer and bookstore, run the SMI traffic policy that will enable communication between bookbuyer and bookstore.
$ Kubectl apply -f 01-deploy-smi-access-policy.yaml
traffictarget.access.smi-spec.io/bookbuyer-access-bookstore created
httproutegroup.specs.smi-spec.io/bookstore-service-routes created
traffictarget.access.smi-spec.io/bookstore-access-bookwarehouse created
httproutegroup.specs.smi-spec.io/bookwarehouse-service-routes created
If the bookbuyer logs are checked again, traffic between bookbuyer and bookstore are restored.
$ Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f
{ … Fetching http://bookstore.bookstore:14001/books-bought
Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]
Identity: bookstore-v1
Booksbought: 552
Server: envoy
Date: Tue, 18 Oct 2022 04:56:07 GMT
Status: 200 OK
MAESTRO! THIS TEST SUCCEEDED!
Fetching http://bookstore.bookstore:14001/buy-a-book/new
Request Headers: map[]
Identity: bookstore-v1
Booksbought: 553
Server: envoy
Date: Tue, 18 Oct 2022 04:56:07 GMT
Status: 200 OK
MAESTRO! THIS TEST SUCCEEDED!
…}
Deploy bookstore v2 service
To display the traffic split functionality of OSM, an additional bookstore v2 service was deployed to allow the bookbuyer reach either bookstore-v1 or bookstore-v2. The deployed bookstore-v2 service would have an SMI traffic policy created as well.
$ Kubectl apply -f 02-deploy-bookstore-v2.yaml
service/bookstore-v2 created
serviceaccount/bookstore-v2 created
deployment.apps/bookstore-v2 created
traffictarget.access.smi-spec.io/bookbuyer-access-bookstore-v2 created
Implement a SMI Traffic Split
The SMI traffic split is applied to enable the bookbuyer to communicate with the bookstore and the bookstore-v2 service at weighted measures. Here, the weight was shared at 25% for bookstore and 75% for bookstore-v2 service.
To see the content of the trafficsplit manifest run
$ cat 03-deploy-smi-trafficsplit.yaml
apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
name: bookstore-split
namespace: bookstore
spec:
service: bookstore.bookstore
backends:
- service: bookstore
weight: 25
- service: bookstore-v2
weight: 75
Apply the manifest by running
$ Kubectl apply -f 03-deploy-smi-trafficsplit.yaml
trafficsplit.split.smi-spec.io/bookstore-split created
Trail the identity of the bookbuyer logs and see that traffic is split with 75% going to bookstore-v2 and 25% going to bookstore-v1. i.e.
$ Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f | grep 'Identity:'
Identity: bookstore-v2
Identity: bookstore-v2
Identity: bookstore-v2
Identity: bookstore-v2
Identity: bookstore-v1
Identity: bookstore-v1
Identity: bookstore-v2
Identity: bookstore-v2
Identity: bookstore-v1
Identity: bookstore-v2
Identity: bookstore-v2
Identity: bookstore-v2
To Uninstall OSM from your AKS hybrid cluster
az k8s-extension delete --cluster-type connectedClusters --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name osm
Summary
In this article, we see the capabilities of OSM been applied to a sample bookstore application and its usefulness in securing your Kubernetes cluster in a variety of ways like Ingress traffic management via policies to permit/deny external access into the mesh, Sidecar proxy injection using the envoy network proxy to ensure services participate in the mesh and obey policies that have been defined, Service-to-service access control via Layer 7 (http) access policies to allow authorized clients and disallow unauthorized clients, and Traffic splitting for canary and blue-green deployments.
In a subsequent article, the observability benefits would be demonstrated to provide metrics, tracing, and logs. This would also enable visualizing application traffic flows, errors, latency and other metrics in tools like Grafana and Jaeger.
Further reading
Azure Arc-enabled Open Service Mesh - Azure Arc | Microsoft Learn