Deploy configurations on AksHci using GitOps with Flux v2

Posted by

This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .

Using Azure Kubernetes Service on Azure Stack HCI to deploy your workload can provide significant infrastructural advantages, such as flexible scaling, management of distributed components, and control over different versions of your application. However, as infrastructure and applications continue to rise with business need, increased control will bring about increased complexity. By enabling GitOps using Flux v2 in your AksHci cluster, you can manage these complexities by declaring the desired state of your Kubernetes clusters in files in Git repositories.

For those who are new to the concept, GitOps is an applications development and operations methodology. It is a strategy that enables you to declaratively describe your entire system, have the desired system state versioned in Git, allow approved changes to be automatically applied to the system, and presents a way of reconciling to ensure correctness. With GitOps, you can use your Git repository as the source of truth for cluster configuration and application deployment.

Typical Git repositories may contain the following files to include: 

  • Yaml formatted manifests that describe Kubernetes resources such as Namespaces, Secrets, Deployments and others. 
  • Helm charts for deploying applications. 
  • Kustomize files to describe environment specific changes. Note: Using Kustomize for helm to apply taints, coloration, and other application configuration should work fine, however, using Kustomize to configure the kubernetes cluster has limited support and some of the changes might be lost during update. E.g., Crictl config 

Azure provides configuration management capability using GitOps in Azure Kubernetes Service (AKS) managed clusters and Azure Arc-enabled Kubernetes connected clusters as a Microsoft.KubernetesConfiguration/extensions/microsoft.flux cluster extension resource. You can easily enable and use GitOps on AksHci clusters.

As at the time of writing this documentation, GitOps using Flux v2 has been announced to be generally available for cloud and hybrid environments Announcing General Availability for GitOps with Flux v2 in Azure - Microsoft Tech Community. The process to implement this is as follows:

Supported Regions: These include East US, Southeast Asia, West Europe and Australia east.

 

Implementation steps using Azure CLI 

1. Ensure you have an existing AksHci cluster or create one if you don’t. You can also create one through the steps in Create AksHci cluster. You can check for the status of your cluster by running Get-AksHciCluster.

PS C:\Windows\system32> Get-AksHciCluster

Status  : {ProvisioningState, Details}

ProvisioningState  : Deployed

KubernetesVersion  : v1.22.6

PackageVersion  : v1.22.6-kvapkg.2

NodePools  : linuxpool

WindowsNodeCount  : 0

LinuxNodeCount  : 3

ControlPlaneNodeCount : 3

ControlPlaneVmSize  : Standard_A2_v2

AutoScalerEnabled  : False

AutoScalerProfile  :

LoadBalancer  : {VMSize, Count, Sku}

Name  : mycluster1

 

2. Ensure that your cluster is Arc connected by running Enable-AksHciArcConnection  -name mycluster1, or follow steps to connect an existing Kubernetes cluster to Azure Arc Quickstart: Connect an existing Kubernetes cluster to Azure Arc - Azure Arc | Microsoft Docs.

 

3.  Ensure you have Azure CLI downloaded and check version to be certain you have Azure CLI version 2.15 or later. Run az version  

PS C:\Windows\system32> az version

{

  "azure-cli": "2.36.0",

  "azure-cli-core": "2.36.0",

  "azure-cli-telemetry": "1.0.6",

  "extensions": {

    "k8s-configuration": "1.5.1",

    "k8s-extension": "1.2.0"

  }

4. Login to azure to register your subscription by running - az login. 

PS C:\Windows\system32>  az login

A web browser has been opened at https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.

[

  {

    "cloudName": "AzureCloud",

    "homeTenantId": "ff547155-eb0a-4b14-afe1-129411efeb4c",

    "id": "5807bea4-cd17-4b81-b4d8-d4e94caab82d",

    "isDefault": true,

    "managedByTenants": [],

    "name": "Visual Studio Enterprise Subscription",

    "state": "Enabled",

    "tenantId": "ff547155-eb0a-4b14-afe1-129411efeb4c",

    "user": {

      "name": "ekelearthur@outlook.com",

      "type": "user"

    }

  }

]

5. Check to see that you are registered with the Azure Service providers. You can run az provider show --n Microsoft.Kubernetes -o table to see registration status, or run the following commands to register

  • az provider register --namespace Microsoft.Kubernetes.
  • az provider register --namespace Microsoft.ContainerService
  • az provider register --namespace Microsoft.KubernetesConfiguration

PS C:\Windows\system32> az provider show -n Microsoft.Kubernetes -o table

>> az provider show -n Microsoft.KubernetesConfiguration -o table

Namespace                                                RegistrationPolicy                       RegistrationState

--------------------                                      --------------------                      -------------------

Microsoft.Kubernetes                                RegistrationRequired                    Registered

Namespace                                                  RegistrationPolicy                     RegistrationState

---------------------------------                    --------------------                     -------------------

Microsoft.KubernetesConfiguration          RegistrationRequired                    Registered

 

6. You can see the list of Az CLI extensions installed and version with the command. Run az extension list -o table 

PS C:\Windows\system32> az extension list -o table

Experimental    ExtensionType    Name                        Path                                                                              Preview    Version

--------------  ---------------    -----------------         -----------------------------------------------------            ---------  ---------

False                   whl                k8s-configuration      C:\Users\ekele\.azure\cliextensions\k8s-configuration     False      1.5.1

False                   whl                k8s-extension            C:\Users\ekele\.azure\cliextensions\k8s-extension            False      1.2.0

 

7. You can now apply the Flux configuration to use GitOps in your Arc-enabled Kubernetes cluster through the k8s-configuration Azure CLI extension or Azure Portal. The sample repo used in this article is https://github/Azure/gitops-flux2-kustomize-helm-mt. If the Microsoft.Flux extension is not already installed in the cluster, it will be installed.

  • Run az k8s-configuration flux create -g cluster-config -c <clustername> -n <namespace> -t <managedClusters or connectedClusters> --scope cluster –u https://github/Azure/gitops-flux2-kustomize-helm-mt --branch main –kustomization --kustomization name=infra path=./infrastructure prune=true --kustomization name=apps path=./apps/staging prune=true dependsOn=["infra"]

'Microsoft.Flux' extension not found on the cluster, installing it now. This may take a few minutes...

'Microsoft.Flux' extension was successfully installed on the cluster

Creating the flux configuration 'cluster-config' in the cluster. This may take a few minutes...

},

  "name": "cluster-config",

  "namespace": "cluster-config",

  "provisioningState": "Succeeded",

  "repositoryPublicKey": "",

  "resourceGroup": "<>",

  "scope": "cluster",

  "sourceKind": "GitRepository",

  "sourceSyncedCommitId": "main/cc2e6a384fb...",

  "sourceUpdatedAt": "2022-08-23T20:29:15+00:00",

  "statusUpdatedAt": "2022-08-23T20:29:21.599000+00:00",

  "statuses": [

    {

      "appliedBy": null,

      "complianceState": "Compliant",

      "helmReleaseProperties": null,

      "kind": "GitRepository",

      "name": "cluster-config",

      "namespace": "cluster-config",

      "statusConditions": [

        {

          "lastTransitionTime": "2022-08-23T20:29:15+00:00",

          "message": "stored artifact for revision 'main/cc2e6a384fb...'",

          "reason": "Succeeded",

          "status": "True",

          "type": "Ready"

        },

 

The following namespaces were created:  

  • flux-system: Holds the Flux extension controllers.  
  • cluster-config: Holds the Flux configuration objects.  
  • nginx, podinfo, redis: Namespaces for workloads described in manifests in the Git repository.  

PS C:\Windows\system32> kubectl get namespaces

NAME                        STATUS   AGE

azure-arc                   Active   9h

cluster-config           Active   86m

default                       Active   17h

flux-system               Active   88m

kube-node-lease     Active   17h

kube-public              Active   17h

kube-system             Active   17h

nginx                          Active   86m

podinfo                      Active   83m

redis                           Active   86m

The Git repository tracks all changes made to files in your project and configuration information usually stored in a .git folder.  

The Git repository contains the following top directories:  

  • apps dir contains Helm releases with a custom configuration per cluster  
  • infrastructure dir contains common infra tools such as NGINX ingress controller and Helm repository definitions  
  • clusters dir contains the Flux configuration per cluster  

The apps configuration is structured into:  

  • apps/base/ dir contains namespaces and Helm release definitions  
  • In apps/base/podinfo/ dir we have a HelmRelease with common values for both clusters  
  • apps/production/ dir contains the production Helm release values  
  • In apps/production/ dir we have a Kustomize patch with the production specific values  
  • apps/staging/ dir contains the staging values  
  • In apps/staging/ dir we have a Kustomize patch with the staging specific values  

In infrastructure/sources/ dir we have the Helm repositories definitions  

The clusters dir contains the Flux configuration. 

  • In clusters/staging/ dir the Kustomization definitions are made             

The flux-system namespace contains flux extension objects such as Azure Flux controllers (fluxconfig-agent, fluxconfig-controller), and OSS Flux controllers (source-controller, kustomize-controller, helm-controller, notification-controller). The Flux agent and controller pods are in a running state. Run kubectl get pods -n flux-system 

  

PS C:\Windows\system32> kubectl get pods -n flux-system

NAME                                                                 READY   STATUS           RESTARTS   AGE

fluxconfig-agent-5d98b7c568-d9xsz              2/2      Running                   0          92m

fluxconfig-controller-cd6444dd9-k8p84        2/2      Running                   0          92m

helm-controller-75988fc95c-8q7rs                 1/1      Running                  0          92m

kustomize-controller-5d7d58477f-wng26     1/1     Running                   0          92m

notification-controller-88879667f-9st8k        1/1     Running                   0          92m

source-controller-7896dfc4b7-4c9nw             1/1    Running                   0          92m

 

The cluster-config namespace has the Flux configuration objects. Run kubectl get crds 

PS C:\Windows\system32> kubectl get crds

NAME                                                                                               CREATED AT

alerts.notification.toolkit.fluxcd.io                                             2022-08-23T20:27:12Z

arccertificates.clusterconfig.azure.com                                     2022-08-23T12:24:20Z

azureclusteridentityrequests.clusterconfig.azure.com              2022-08-23T12:24:20Z

azureextensionidentities.clusterconfig.azure.com                    2022-08-23T12:24:20Z

bgpconfigurations.crd.projectcalico.org                                   2022-08-23T04:29:59Z

bgppeers.crd.projectcalico.org                                                  2022-08-23T04:29:59Z

blockaffinities.crd.projectcalico.org                                           2022-08-23T04:29:59Z

buckets.source.toolkit.fluxcd.io                                                 2022-08-23T20:27:12Z

clusterinformations.crd.projectcalico.org                                 2022-08-23T04:29:59Z

connectedclusters.arc.azure.com                                              2022-08-23T12:24:20Z

csinodeinfos.csi.storage.k8s.io                                                  2022-08-23T04:29:58Z

customlocationsettings.clusterconfig.azure.com                     2022-08-23T12:24:20Z

extensionconfigs.clusterconfig.azure.com                                2022-08-23T12:24:20Z

felixconfigurations.crd.projectcalico.org                                   2022-08-23T04:29:59Z

fluxconfigs.clusterconfig.azure.com                                          2022-08-23T20:27:12Z

gitconfigs.clusterconfig.azure.com                                            2022-08-23T12:24:20Z

gitrepositories.source.toolkit.fluxcd.io                                      2022-08-23T20:27:12Z

globalnetworkpolicies.crd.projectcalico.org                             2022-08-23T04:29:59Z

globalnetworksets.crd.projectcalico.org                                   2022-08-23T04:29:59Z

helmcharts.source.toolkit.fluxcd.io                                            2022-08-23T20:27:12Z

helmreleases.helm.toolkit.fluxcd.io                                           2022-08-23T20:27:13Z

helmrepositories.source.toolkit.fluxcd.io                                  2022-08-23T20:27:12Z

hostendpoints.crd.projectcalico.org                                           2022-08-23T04:29:59Z

imagepolicies.image.toolkit.fluxcd.io                                         2022-08-23T20:27:12Z

imagerepositories.image.toolkit.fluxcd.io                                 2022-08-23T20:27:12Z

imageupdateautomations.image.toolkit.fluxcd.io                   2022-08-23T20:27:12Z

ipamblocks.crd.projectcalico.org                                                2022-08-23T04:30:00Z

ipamconfigs.crd.projectcalico.org                                               2022-08-23T04:30:00Z

ipamhandles.crd.projectcalico.org                                             2022-08-23T04:30:00Z

ippools.crd.projectcalico.org                                                       2022-08-23T04:30:00Z

kubecontrollersconfigurations.crd.projectcalico.org              2022-08-23T04:30:00Z

kustomizations.kustomize.toolkit.fluxcd.io                               2022-08-23T20:27:12Z

networkpolicies.crd.projectcalico.org                                        2022-08-23T04:30:00Z

networksets.crd.projectcalico.org                                              2022-08-23T04:30:00Z

providers.notification.toolkit.fluxcd.io                                       2022-08-23T20:27:12Z

receivers.notification.toolkit.fluxcd.io                                       2022-08-23T20:27:12Z

renewrevocations.certificates.microsoft.com                          2022-08-23T04:30:02Z

 

PS C:\Windows\system32> kubectl get fluxconfigs -A

NAMESPACE        NAME            SCOPE     URL                                                                                       PROVISION   AGE

cluster-config   cluster-config   cluster   https://github.com/Azure/gitops-flux2-kustomize-helm-mt   Succeeded   97m

 

PS C:\Windows\system32> kubectl get gitrepositories -A

NAMESPACE        NAME             URL                                                                                         AGE   READY   STATUS

cluster-config   cluster-config   https://github.com/Azure/gitops-flux2-kustomize-helm-mt   98m   True    stored artifact for revision 'main/cc2e6a384fb...'

 

PS C:\Windows\system32> kubectl get helmreleases -A

NAMESPACE             NAME      AGE   READY   STATUS

cluster-config           nginx        99m   True    Release reconciliation succeeded

cluster-config           podinfo   96m   True    Release reconciliation succeeded

cluster-config           redis         99m   True    Release reconciliation succeeded

 

PS C:\Windows\system32> kubectl get kustomizations -A

NAMESPACE          NAME                             AGE    READY   STATUS

cluster-config     cluster-config-apps         100m   True    Applied revision: main/cc2e6a384fbfa...

cluster-config     cluster-config-infra         100m   True    Applied revision: main/cc2e6a384fbfa...

 

Workloads would be deployed from manifests in the Git repository

 

PS C:\Windows\system32> kubectl get deploy -n nginx

NAME                                                                     READY   UP-TO-DATE   AVAILABLE         AGE

nginx-ingress-controller                                          1/1            1                        1                100m

nginx-ingress-controller-default-backend               1/1            1                        1                100m

 

PS C:\Windows\system32> kubectl get deploy -n podinfo

NAME      READY   UP-TO-DATE   AVAILABLE   AGE

podinfo     1/1                1                   1           98m

 

PS C:\Windows\system32> kubectl get all -n redis

NAME                        READY   STATUS    RESTARTS   AGE

pod/redis-master-0     1/1     Running       0              101m

pod/redis-replicas-0   1/1     Running        0              101m

pod/redis-replicas-1   1/1     Running        0              100m

pod/redis-replicas-2   1/1     Running        0               99m

 

NAME                                          TYPE        CLUSTER-IP      EXTERNAL-IP      PORT(S)    AGE

service/redis-headless             ClusterIP   None                   <none>                          6379/TCP   101m

service/redis-master                ClusterIP   10.111.146.68   <none>         6379/TCP   101m

service/redis-replicas               ClusterIP   10.98.161.85    <none>          6379/TCP   101m

 

NAME                                          READY      AGE

statefulset.apps/redis-master     1/1     101m

statefulset.apps/redis-replicas   3/3     101m

 

Supporting documents: For more information on deploying configurations on your AksHci cluster using GitOps with Flux v2, see Tutorial: Use GitOps with Flux v2 in Azure Arc-enabled Kubernetes or Azure Kubernetes Service (AKS) clusters - Azure Arc | Microsoft Docs 

 and Azure/gitops-flux2-kustomize-helm-mt: example repo for gitops with flux v2 cloned from the fluxcd project (https://github.com/fluxcd/flux2-kustomize-helm-example) and updated to work with multi-tenancy
 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.