This post has been republished via RSS; it originally appeared at: Microsoft Mobile Engineering - Medium.
This is the second part of our multi-part series. You can find the previous part below. In this article, we will cover the building blocks of Azure Pipelines and how we set up agents within the SwiftKey Android team.
Let us start with some key concepts of Azure Pipelines.
Key concepts
If you are familiar with how Azure Pipelines work, feel free to jump ahead to the SwiftKey Agents setup section below.
- A trigger tells a pipeline to run.
- A pipeline is made up of one or more stages. A pipeline can deploy to one or more environments.
- A stage is a way of organising jobs in a pipeline and each stage can have one or more jobs.
- Each job runs on one agent. A job can also be agentless.
- An agent runs a job that has one or more steps.
- A step can be a task or script and is the smallest building block of a pipeline.
- A task is a pre-packaged script that performs an action, such as invoking a REST API or publishing a build artifact.
- An artifact is a collection of files published by a run.
Azure Pipelines come with certain built-in tasks to enable basic build and deployment scenarios. The full catalogue of built-in tasks can be found here. Visual Studio Marketplace has a plethora of additional custom extensions from Microsoft or 3rd Parties. Installing a custom extension, expands the task catalogue in your own Azure DevOps (ADO) subscription and it requires a security approval from the subscription admin.
Azure Pipeline Execution
This is a high-level overview of the series of events that take place in executing a pipeline.
- Start a pipeline: Either a developer or one of the trigger events starts the pipeline on Azure DevOps (ADO).
- Agent lookup: ADO looks at the pipeline configuration to find all jobs that can run in parallel and finds a suitable agent for each job based on the demands specified in the job configuration.
- Submit job: Once an agent is allocated, ADO submits that job to the agent. Every job is executed on an agent except agentless jobs which run on the ADO server.
- Job execution: Agent executes each step defined in the job sequentially and report the results back to ADO.
- Logs and artifacts: ADO gathers all the execution logs from each step and artifacts in every job and makes them accessible through ADO.
Agents on Azure Pipelines
An agent on Azure Pipelines is the entity on which jobs are executed. An agent can be a physical machine, virtual machine or a docker container. Agents are grouped into agent pools. Broadly there are two types of agent pools: Microsoft-hosted agents and Self-hosted agents.
Microsoft-hosted agents
These are managed entirely by Microsoft, so software updates and maintenance are taken care of. These agents are re-imaged between each job so we can expect clean and predictable state for each job. They come pre-installed with commonly used tools and packages. At the time of writing, these agents have the following limitations:
- Custom packages need to be installed at runtime*
- Run on Azure DSv2-series VMs
- 10 GB of storage
- 360 mins execution time limit
*Microsoft-hosted agents based on Windows 2019 and Ubuntu support running jobs in docker containers. This could speed up jobs that depend on installing custom packages.
Self-hosted agents
ADO supports deploying an agent pool where the agents are entirely self-managed. These agents can be based on Windows, Linux, or Mac operating systems. Users have full control over how these agents are built and configured so, it is possible to:
- Pre-install any custom packages
- Use more-powerful machines
- Add agents to internal networks (useful if certain resources are available only over internal network)
- Host agents on-premises to attach physical devices such as Android phones
SwiftKey Agents setup
At SwiftKey we need agents with different capabilities as we discussed in the part 1 of this series. Our agents are broadly divided into three categories:
Microsoft-hosted agents
These agents are managed by ADO. We use them for build jobs which use only the pre-installed packages. Jobs that run on these agents include builds for open-source repositories, and builds that are not time critical.
Self-hosted agents in Cloud
These agents are fully managed by us and contain custom packages that are required for certain jobs. These agents are Docker containers deployed on Azure Cloud Infrastructure. We use these agents for jobs that have at least one of the following requirements:
- Need an emulator
- Need custom packages
- Need more memory or CPU
Jobs that typically run on these agents include Android nightly tests, and APK builds.
Self-hosted agents on-premises
These agents are also fully managed by us, containerised using the same image as the Self-hosted agents in Cloud. These agents are used for jobs that have at least one of the following requirements:
- Need a physical device
- Need to access resources available only on the internal network
Jobs that run on these agents are performance tests, smoke tests etc.
Config ownership
As shown in the figure above, the Android team now owns bigger part of our CI/CD infrastructure. This allows us to rapidly iterate on changes to the agents and their configurations. Our Cloud Infrastructure (CLI) team owns the infrastructure which deploys agents following the configuration defined by the Android team.
In the next article in this series, we will discuss different Azure Pipeline features and how we leveraged them in our CI / CD processes.
CI / CD at SwiftKey (Part 2) was originally published in Microsoft Mobile Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.