This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .
Azure Synapse Analytics is a limitless analytics service that brings together data integration, enterprise data warehousing and big data analytics. It gives you the freedom to query data on your terms, using either serverless or dedicated resources—at scale. Azure Synapse brings these worlds together with a unified experience to ingest, explore, prepare, manage and serve data for immediate BI and machine learning needs.
Azure Resource Manager is the deployment and management service for Azure and the ARM templates allow you to create and deploy an entire Azure infrastructure declaratively. For example, you can deploy not only virtual machines, but also the network infrastructure, storage systems, and any other resources you may need.
ARM template is a JavaScript Object Notation (JSON) file that defines one or more resources to deploy to a resource group, subscription, management group, or tenant. The template can be used to deploy the resources consistently and repeatedly. (more)
When the Synapse workspace is created with the encryption that is referred to Azure KeyVault key, the workspace needs to be “Activated” before moving forward with other deployments such as Dedicated SQL pools, Apache Spark Pools.
There are multiple ways that the workspace can be activated, but there are multiple steps that needs to be completed before activating the workspace, (more).
Example:
- Confirm the Azure KeyVault is created with “purge protection”.
Purge protection is an optional Key Vault behavior and is not enabled by default. Purge protection can only be enabled once soft delete is enabled. It can be turned on via CLI or PowerShell. - Grant the Workspace managed Identity required permissions (Get, Wrap Key, Unwrap Key) on Azure KeyVault.
- Manually activate the workspace.
Even If the KeyVault is already deployed, there are multiple use cases related to the automations and activating the Synapse Workspaces such as,
- Create access policies to Synapse workspace managed identity during the deployments.
- Provide proper permissions (Get, Wrap Key, Unwrap Key) to workspace managed identity during the deployments.
- Activate the Synapse workspace without any manual action.
- Complete all above actions (#1, #2, #3) in a single run.
There are multiple cmdlets and commands provided in Azure PowerShell and Azure CLI to achieve this and activate the workspace successfully, but that requires to run multiple commands time to time.
Example:
In Azure PowerShell,
“Enable-AzSynapseWorkspace” activates the Synapse Workspace, but the access policies need to be configured before running this command.
In Azure CLI,
“az synapse workspace activate” activates the Synapse Workspace, but the access policies need to be configured before running this command.
The Azure Resource Manager Templates (ARM templates) can be used to perform all of these actions as one single task easily and activate the workspace during the deployment process.
The template file contains the deployment operations, and the Parameter file contains the values that required for the deployments.
The template file also can contain the different variables that can be an output of the previous operation. (Example: Managed Identity Object).
In this example, we assume that the Azure KeyVault is already deployed under the same resource group and below Template and Parameter files can be used to deploy the Synapse workspace & activate at the same time.
Example:
(Following example template file can be used to deploy a Synapse workspace and activate using a key in AzureKeyVault)
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.1",
"parameters": {
"name": {
"type": "String"
},
"location": {
"type": "String"
},
"defaultDataLakeStorageAccountName": {
"type": "String"
},
"defaultDataLakeStorageFilesystemName": {
"type": "String"
},
"sqlAdministratorLogin": {
"type": "String"
},
"sqlAdministratorLoginPassword": {
"defaultValue": "",
"type": "SecureString"
},
"setWorkspaceIdentityRbacOnStorageAccount": {
"type": "Bool"
},
"createManagedPrivateEndpoint": {
"type": "Bool"
},
"azureADOnlyAuthentication": {
"type": "Bool"
},
"allowAllConnections": {
"defaultValue": true,
"type": "Bool"
},
"managedVirtualNetwork": {
"allowedValues": [
"default",
""
],
"type": "String"
},
"tagValues": {
"defaultValue": {},
"type": "Object"
},
"useSystemAssignedIdentity": {
"type": "Bool"
},
"storageSubscriptionID": {
"defaultValue": "[subscription().subscriptionId]",
"type": "String"
},
"storageLocation": {
"defaultValue": "[resourceGroup().location]",
"type": "String"
},
"storageRoleUniqueId": {
"defaultValue": "[newGuid()]",
"type": "String"
},
"isNewStorageAccount": {
"defaultValue": false,
"type": "Bool"
},
"isNewFileSystemOnly": {
"defaultValue": false,
"type": "Bool"
},
"storageAccessTier": {
"type": "String"
},
"storageAccountType": {
"type": "String"
},
"storageSupportsHttpsTrafficOnly": {
"type": "Bool"
},
"storageKind": {
"type": "String"
},
"minimumTlsVersion": {
"type": "String"
},
"storageIsHnsEnabled": {
"type": "Bool"
},
"userObjectId": {
"defaultValue": "",
"type": "String"
},
"setSbdcRbacOnStorageAccount": {
"defaultValue": false,
"type": "Bool"
},
"setWorkspaceMsiByPassOnStorageAccount": {
"defaultValue": false,
"type": "Bool"
},
"workspaceStorageAccountProperties": {
"defaultValue": {},
"type": "Object"
},
"keysPermissions": {
"type": "array"
},
"keyVaultName": {
"type": "string"
},
"keyVaultKeyName": {
"type": "string"
},
"workspaceKeyName": {
"type": "string"
}
},
"variables": {
"storageBlobDataContributorRoleID": "ba92f5b4-2d11-453d-a403-e96b0029c9fe",
"defaultDataLakeStorageAccountUrl": "[concat('https://', parameters('defaultDataLakeStorageAccountName'), '.dfs.core.windows.net')]",
"keyVaultURI": "[concat('https://', parameters('keyVaultName'), '.vault.azure.net', '/keys/', parameters('keyVaultKeyName'))]",
"defaultAdlsGen2AccountResourceId": "[concat('/subscriptions/', parameters('storageSubscriptionID'), '/resourceGroups/',resourceGroup().name,'/providers/Microsoft.Storage/storageAccounts/',parameters('defaultDataLakeStorageAccountName'))]",
"managedResourceGroupName": "[concat(resourceGroup().name, '-mrg')]"
},
"resources": [
{
"type": "Microsoft.Synapse/workspaces",
"apiVersion": "2021-06-01",
"name": "[parameters('name')]",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Storage/storageAccounts/', parameters('defaultDataLakeStorageAccountName'))]",
"[concat('Microsoft.Resources/deployments/', parameters('defaultDataLakeStorageFilesystemName'))]"
],
"tags": "[parameters('tagValues')]",
"identity": {
"type": "SystemAssigned"
},
"properties": {
"defaultDataLakeStorage": {
"accountUrl": "[variables('defaultDataLakeStorageAccountUrl')]",
"filesystem": "[parameters('defaultDataLakeStorageFilesystemName')]",
"resourceId": "[variables('defaultAdlsGen2AccountResourceId')]",
"createManagedPrivateEndpoint": "[parameters('createManagedPrivateEndpoint')]"
},
"managedVirtualNetwork": "[parameters('managedVirtualNetwork')]",
"managedResourceGroupName": "[variables('managedResourceGroupName')]",
"azureADOnlyAuthentication": "[parameters('azureADOnlyAuthentication')]",
"sqlAdministratorLogin": "[parameters('sqlAdministratorLogin')]",
"sqlAdministratorLoginPassword": "[parameters('sqlAdministratorLoginPassword')]",
"encryption": {
"cmk": {
"key": {
"name": "[parameters('workspaceKeyName')]",
"keyVaultUrl": "[variables('keyVaultURI')]"
},
"keyIdentity": {
"useSystemAssignedIdentity": "[parameters('useSystemAssignedIdentity')]"
}
}
}
},
"resources": [
{
"type": "firewallrules",
"apiVersion": "2021-06-01",
"name": "allowAll",
"location": "[parameters('location')]",
"dependsOn": [
"[concat('Microsoft.Synapse/workspaces/', parameters('name'))]"
],
"properties": {
"startIpAddress": "0.0.0.0",
"endIpAddress": "255.255.255.255"
},
"condition": "[parameters('allowAllConnections')]"
}
]
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2021-04-01",
"name": "StorageRoleDeploymentResource",
"dependsOn": [
"[concat('Microsoft.Synapse/workspaces/', parameters('name'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts/providers/roleAssignments",
"apiVersion": "2022-01-01-preview",
"name": "[concat(parameters('defaultDataLakeStorageAccountName'), '/Microsoft.Authorization/', guid(concat(resourceGroup().id, '/', variables('storageBlobDataContributorRoleID'), '/', parameters('name'), '/', parameters('storageRoleUniqueId'))))]",
"location": "[parameters('storageLocation')]",
"properties": {
"roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('storageBlobDataContributorRoleID'))]",
"principalId": "[reference(concat('Microsoft.Synapse/workspaces/', parameters('name')), '2021-06-01', 'Full').identity.principalId]",
"principalType": "ServicePrincipal"
}
},
{
"condition": "[parameters('setSbdcRbacOnStorageAccount')]",
"type": "Microsoft.Storage/storageAccounts/providers/roleAssignments",
"apiVersion": "2022-01-01-preview",
"name": "[concat(parameters('defaultDataLakeStorageAccountName'), '/Microsoft.Authorization/', guid(concat(resourceGroup().id, '/', variables('storageBlobDataContributorRoleID'), '/', parameters('userObjectId'), '/', parameters('storageRoleUniqueId'))))]",
"properties": {
"roleDefinitionId": "[resourceId('Microsoft.Authorization/roleDefinitions', variables('storageBlobDataContributorRoleID'))]",
"principalId": "[parameters('userObjectId')]",
"principalType": "User"
}
}
]
}
},
"subscriptionId": "[parameters('storageSubscriptionID')]",
"resourceGroup": "[resourceGroup().name]",
"condition": "[parameters('setWorkspaceIdentityRbacOnStorageAccount')]"
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2021-04-01",
"name": "UpdateStorageAccountNetworkingAcls",
"dependsOn": [
"[concat('Microsoft.Synapse/workspaces/', parameters('name'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-01-01",
"location": "[parameters('storageLocation')]",
"name": "[parameters('defaultDataLakeStorageAccountName')]",
"properties": "[parameters('workspaceStorageAccountProperties')]"
}
]
}
},
"subscriptionId": "[parameters('storageSubscriptionID')]",
"resourceGroup": "[resourceGroup().name]",
"condition": "[parameters('setWorkspaceMsiByPassOnStorageAccount')]"
},
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2021-01-01",
"name": "[parameters('defaultDataLakeStorageAccountName')]",
"location": "[parameters('storageLocation')]",
"tags": {},
"sku": {
"name": "[parameters('storageAccountType')]"
},
"kind": "[parameters('storageKind')]",
"properties": {
"accessTier": "[parameters('storageAccessTier')]",
"supportsHttpsTrafficOnly": "[parameters('storageSupportsHttpsTrafficOnly')]",
"isHnsEnabled": "[parameters('storageIsHnsEnabled')]",
"minimumTlsVersion": "[parameters('minimumTlsVersion')]"
},
"resources": [
{
"type": "blobServices/containers",
"apiVersion": "2021-01-01",
"name": "[concat('default/', parameters('defaultDataLakeStorageFilesystemName'))]",
"dependsOn": [
"[concat('Microsoft.Storage/storageAccounts/', parameters('defaultDataLakeStorageAccountName'))]"
],
"properties": {
"publicAccess": "None"
},
"condition": "[parameters('isNewStorageAccount')]"
}
],
"condition": "[parameters('isNewStorageAccount')]"
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2021-04-01",
"name": "[parameters('defaultDataLakeStorageFilesystemName')]",
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"name": "[concat(parameters('defaultDataLakeStorageAccountName'), '/default/', parameters('defaultDataLakeStorageFilesystemName'))]",
"apiVersion": "2021-01-01",
"properties": {
"publicAccess": "None"
}
}
]
}
},
"subscriptionId": "[parameters('storageSubscriptionID')]",
"resourceGroup": "[resourceGroup().name]",
"condition": "[parameters('isNewFileSystemOnly')]"
},
{
"type": "Microsoft.KeyVault/vaults/accessPolicies",
"name": "[concat(parameters('keyVaultName'), '/add')]",
"apiVersion": "2021-10-01",
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[reference(concat('Microsoft.Synapse/workspaces/', parameters('name')), '2019-06-01-preview', 'Full').identity.principalId]",
"permissions": {
"keys": "[parameters('keysPermissions')]"
}
}
]
}
},
{
"type": "Microsoft.Synapse/workspaces/keys",
"apiVersion": "2021-06-01",
"name": "[concat(parameters('name'), '/',parameters('workspaceKeyName'))]",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/accessPolicies', parameters('keyVaultName'), 'add')]",
"[concat('Microsoft.Synapse/workspaces/', parameters('name'))]"
],
"properties": {
"isActiveCMK": true,
"keyVaultUrl": "[variables('keyVaultURI')]"
}
}
],
"outputs": {}
}
There are multiple operations to be performed in this template such as creating storage, creating firewall rules and workspace, but since the access policies needs to be created before the activation, that is very important action to activate the workspace, the access policies are creating using below code snip.
{
"type": "Microsoft.KeyVault/vaults/accessPolicies",
"name": "[concat(parameters('keyVaultName'), '/add')]",
"apiVersion": "2021-10-01",
"properties": {
"accessPolicies": [
{
"tenantId": "[subscription().tenantId]",
"objectId": "[reference(concat('Microsoft.Synapse/workspaces/', parameters('name')), '2019-06-01-preview', 'Full').identity.principalId]",
"permissions": {
"keys": "[parameters('keysPermissions')]"
}
}
]
}
}
The activation is mainly dependent on the policies created in Azure KeyVault.
There are 2 properties on below – (isActiveCMK / keyVaultUrl) that needs to have proper parameters to activate the workspace.
{
"type": "Microsoft.Synapse/workspaces/keys",
"apiVersion": "2021-06-01",
"name": "[concat(parameters('name'), '/',parameters('workspaceKeyName'))]",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/accessPolicies', parameters('keyVaultName'), 'add')]",
"[concat('Microsoft.Synapse/workspaces/', parameters('name'))]"
],
"properties": {
"isActiveCMK": true,
"keyVaultUrl": "[variables('keyVaultURI')]"
}
}
During the deployment, the template file does not contain any hard coded values, the values are referred from the parameter file, or the values can be the outputs of deployments in ARM.
Example:
(Following example parameter file can be used to deploy a Synapse workspace and activate using a key in AzureKeyVault)
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"value": "[Your Workspace Name]"
},
"location": {
"value": "[Region]"
},
"defaultDataLakeStorageAccountName": {
"value": "[Storage Account Name]"
},
"defaultDataLakeStorageFilesystemName": {
"value": "[Stroage Account File System]"
},
"sqlAdministratorLogin": {
"value": "[SQL Administrator Login]"
},
"sqlAdministratorLoginPassword": {
"value": "[SQL Administrator Password]"
},
"keyVaultName": {
"value": "[Azure KeyVault Name]"
},
"setWorkspaceIdentityRbacOnStorageAccount": {
"value": true
},
"createManagedPrivateEndpoint": {
"value": false
},
"azureADOnlyAuthentication": {
"value": false
},
"allowAllConnections": {
"value": true
},
"managedVirtualNetwork": {
"value": ""
},
"tagValues": {
"value": {}
},
"keyVaultKeyName": {
"value":"[Azure KeyVault Key name]"
},
"workspaceKeyName": {
"value":"[Azure KeyVault Key name]"
},
"useSystemAssignedIdentity": {
"value": true
},
"storageSubscriptionID": {
"value": "[Subscription ID]"
},
"storageLocation": {
"value": "[Storage Location]"
},
"storageRoleUniqueId": {
"value": "13c7d920-0450-4a62-baf8-37f4a367101c"
},
"isNewStorageAccount": {
"value": true
},
"isNewFileSystemOnly": {
"value": false
},
"storageAccessTier": {
"value": "Hot"
},
"storageAccountType": {
"value": "Standard_RAGRS"
},
"storageSupportsHttpsTrafficOnly": {
"value": true
},
"storageKind": {
"value": "StorageV2"
},
"minimumTlsVersion": {
"value": "TLS1_2"
},
"storageIsHnsEnabled": {
"value": true
},
"userObjectId": {
"value": "[User Object ID]"
},
"setSbdcRbacOnStorageAccount": {
"value": true
},
"setWorkspaceMsiByPassOnStorageAccount": {
"value": false
},
"workspaceStorageAccountProperties": {
"value": {}
},
"keysPermissions": {
"value": [
"get",
"list",
"wrapKey",
"unwrapKey"
]
}
}
}
The parameter file contains multiple parameters that is used during the deployment such as,
- Workspace name
- Location
- SQL admin login
- SQL password
- KeyVault name
As same as there are many deployment parameters mentioned. These parameters needs to be changed / modified based on your requirement.
You can initiate the deployments using the parameter file and the template file from Azure PowerShell command “New-AzResourceGroupDeployment”
Example:
(Following example is to execute a ARM deployment using “New-AzResouceGroupDeployment” PowerShell cmdlet)
New-AzResourceGroupDeployment -Name "[Name for the deployment]" -TemplateFile [Template File Location]' -TemplateParameterFile '[Parameter File Location]' -ResourceGroupName "[ResourceGroup Name]"
Monitor the Deployments:
The PowerShell cmdlet provides the output whether the deployment is success or not, but you can always monitor the deployment from the Azure portal by navigating in to the resource group.
The ongoing deployments and the history of the deployments can be checked using the “Deployments".
You can select the relevant deployment name from the list.
Deployment status:
Let's recapitulate what we discussed above,
When the Synapse workspace is created with the encryption that is referred to Azure KeyVault key, the workspace needs to be “Activated” before moving forward with other deployments such as Dedicated SQL pools, Apache Spark Pools.
The administrators required to perform this action during the deployments as they use different techniques for automations.
Azure PowerShell module and Azure CLI for Synapse has different set of commands to perform this action and also this activation can be done from Azure Portal manually.
Not only from above, but this action can also be performed using ARM templates thought a single deployment without multiple actions.