How-To Deploy your Synapse Workspace Artifacts to a Managed VNET Synapse Workspace

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Azure DevOps is the service of choice that you can use to automate the provisioning of your modern data platform and to allow the continuous integration (CI) and continuous delivery (CD) to continuously test, build , and deploy your code across multiple environments.

 

In the particular context of Azure Synapse Analytics service, you can think of CICD as a two staged process:

 

- A first stage, where you deal with resources such as compute pools and the workspace itself

- A second stage, where you deal with the so called artifacts (Synapse Notebooks, SQL Scripts, pipelines, etc)

 

In my previous article, I demonstrated how we could easily use the Synapse Workspace Deployment extension to accomplish this second stage of the process. I’m now coming back to this topic as I realized that many of our customers were reporting difficulties in completing this second stage of their Synapse CICD process because they were failing to deploy Synapse artifacts to a Managed VNET Synapse Workspace.

In this particular scenario, the deployment was failing because public network access to their target workspace has been disabled.

 

RuiCunha_1-1632004957903.png

 

The deployment was failing with the following error:

 

Deploy artifact failed: {"code":"PublicNetworkAccessDenied","message":"The public network interface on this Workspace is not accessible. To connect to this Workspace, use the Private Endpoint from inside your virtual network or enable public network access for this workspace."}

 

To address this specific problem, a solution would be to combine the usage of Private Endpoints and Azure DevOps Self-hosted agents. I’m providing below a two phased action plan with the detailed steps to achieve this :

 

In a first phase, it will be detailed the necessary configuration process when it comes to configure the DevOps side of the equation

 

The second phase will focus on the necessary networking configuration.

 

Stage 1 – Configure DevOps

 

First things first: you need to use our own compute infrastructure that will run the Azure DevOps Release Pipeline. Remember that your Synapse workspace is blocking access from public networks, so instead of using a Microsoft-hosted that will try to reach the workspace via public endpoint, you will need to use a Self-hosted agent and sit this agent on a VNET so it can communicate with the Workspace on a  private and secure way.

 

Let’s start by creating a Self-hosted Windows agent (you can refer to this public doc to get more details about this step). In this example, you'll need an Azure VM to install this agent:

 

  1. Connect to your Azure VM and open a new browser session. Navigate to https://dev.azure.com/{your_organization} and Sign in with the user account you plan to use in your Azure DevOps organization

  2. If you haven’t created your your personal access token (PAT) please follow the instructions provided here.

  3. Download and configure the Self-hosted agent following this instructions

 

RuiCunha_0-1632005343605.png

 

 

This screenshot below reflects an example of this configuration step:

 

RuiCunha_1-1632005343615.png

 

When you have finished the configuration process above, check your “Default” agent pool in DevOps and you should see your newly created Self-hosted agent Online:

 

 

RuiCunha_2-1632005343621.png


If you still see your agent Offline, make sure to run the agent from the VM, submitting the following command from the Powershell:

.\run.cmd

Once you have your Self-hosted agent up and running, it’s time to configure the Release Pipeline.

 

  1. Follow these steps below to configure your Release Pipeline to deploy Synapse Workspace artifacts from a source environment to a target environment:

 

From the left menu select “Pipelines-->Releases” and then select “New release pipeline”

 

RuiCunha_3-1632005343628.png

 

Select to start with an “Empty job” when selecting your template. Provide a new name to your Release Pipeline and also to the initial stage. Click on the “+ Add” to add a new  Artifact.

 

RuiCunha_4-1632005343633.png

 

Configure the Pipeline source artifact by providing your Project, Source Repository and Default branch (typically the workspace branch, where you have your workspace JSON files published).

 

 

RuiCunha_5-1632005343642.png

 

Select “Add” to confirm your settings and to create the source artifact. Now it’s time to configure the Release Stage. Select  the “1 job , 0 task” link on your stage.

 

RuiCunha_6-1632005343653.png

 

Provide a new display name to your agent. Once thing important is to change the “Agent selection” from “Azure Pipelines” to “Default” (this is the pool that is hosting your newly created Self-hosted agent)

 

 

RuiCunha_7-1632005343657.png

 

Click the “+” sign on the Agent job bar to add a new task. Then search for Synapse and select “Add” to add the "Synapse Workspace Deployment"  task to your agent.

 

 

RuiCunha_8-1632005343663.png

 

 

Configure the Synapse Workspace deployment task (for the sake of this demo I’m not overriding any parameters) and don’t forget to save your changes

 

 

RuiCunha_9-1632005343674.png

 

At the end of this first stage you should have have finished the following action items:

 

  1. The Self-Hosted up and running

  2. The Release Pipeline configured

 

Let’s move forward to the final stage of this article: putting all networking pieces together so the Azure DevOps service and Synapse service can communicate privately and securely.

 

Stage 2 – Network Setup

 

Navigate to your Synapse Workspace resource page and create a Private Endpoint to the Synapse Workspace development endpoint.

 

 

RuiCunha_10-1632005343686.png

 

 

RuiCunha_11-1632005343694.png

 

You must select “Dev” when choosing the “Target sub-resource”. This will refer to the Synapse Workspace Development endpoint.

 

 

RuiCunha_12-1632005343701.png

 

Select the Virtual Network where you have hosted the DevOps Self-hosted agent VM

 

 

RuiCunha_13-1632005343712.png

 

 

RuiCunha_14-1632005343724.png

 

 

After the “Review and create” step, hit the “Create” button to create the Synapse Workspace Private Endpoint.

If you check your Virtual Network Private Endpoints you’ll see the newly create private endpoint:

 

 

RuiCunha_15-1632005343730.png

 

 

Check your Private Endpoint DNS Configuration and look for the IP Address and FQDN

 

 

RuiCunha_16-1632005343736.png

 

If you "nslookup" the Synapse Workspace from your VM, confirm if values from the output match the Private DNS Config:

 

 

RuiCunha_17-1632005343739.png

 

Before you kick-off the DevOps Release Pipeline, you need also to confirm that the DevOps Connection Service Principal that is being used by Azure Synapse Deployment task has been granted the necessary Synapse RBAC permissions (if not, the task will fail with a 403 authorization error) 

 

 

RuiCunha_18-1632005343749.png

 

 

Finally, you can create your first release to start deploying your Synapse Workspace artifacts to the target Managed VNET Synapse Workspace:

 

RuiCunha_19-1632005343754.png

 

 

You should now see the Synapse Workspace Deployment task succeeded!

A final note to thank to my colleagues  and @FonsecaSergio for all the support

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.