Getting started with REST APIs for Azure Synapse Analytics – Apache Spark Pool

Posted by

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.



Author: Abid Nazir Guroo is a Program Manager in Azure Synapse Customer Success Engineering (CSE) team.



Azure Synapse Analytics Representational State Transfer (REST) APIs are secure HTTP service endpoints that support creating and managing Azure Synapse resources using Azure Resource Manager and Azure Synapse web endpoints. This article provides instructions on how to setup and use Synapse REST endpoints and describe the Apache Spark Pool operations supported by REST APIs.



In order to perform any operation using Azure REST APIs you need to authenticate the request using an azure active directory authentication token. This token can be generated by various interactive and non-interactive methods. In this tutorial we will use a azure active directory service principal for generating a token with below steps:

  1. Register an application with Azure AD and create a service principal. You can create a new service principal using Azure Portal by following the steps outlined in Create an Azure AD app and service principal
  2. Create a secret for the registered service principal and save it securely for usage. Instructions for creating a secret are documented at - Create a secret for an application
  3. Assign an appropriate role to the registered service principal on the Synapse Workspace based on the operations you want to perform using the REST APIs. You can find detailed information on all available Azure Synapse Analytics roles and corresponding permissions at - Azure Synapse RBAC roles



You can use various command line utilities like CURL or Powershell, programming languages or GUI clients to interact with REST API endpoints. In this article we will leverage a widely used GUI application - Postman for its simplicity and ease of setup. Postman can be downloaded with this link.


Example: How to get an Azure AD Access Token

The following parameters are needed to successfully get an Azure AD access token:




Tenant ID

The Azure Active Directory (tenant) ID where the application is registered.

Client ID

The Application (client) ID for the application registered in Azure AD.

Client secret

The Value of the client secret for the application registered in Azure AD.


On the Postman UI

  1. Create a new HTTP request (File > New > HTTP Request or using the new tab (+) icon).
  2. In the HTTP verb drop-down list, select POST.
  3. For Enter request URL, enter<tenant_id>/oauth2/token, where <tenant_id> is your Active Directory (tenant) ID.
  4. On the headers tab enter a new key Content-Type with value application/x-www-form-urlencoded.
  5. On the Body tab, select x-www-form-urlencoded and enter below key value pairs.grant_type:client_credentials client_id:<client_id> client_secret:<Client_secret> resource:​




  6. Click send. You will receive the bearer token in the json response to the request. Below is an example of the response.



{ "token_type": "Bearer", "expires_in": "", "ext_expires_in": "", "expires_on": "", "not_before": "", "resource": "", "access_token": "<access token to be used for invoking REST APIs>" }




Note: The "resource" parameter value should be "" or """ for control plane operations and "https://{workspace name}" for data plane operations.


Azure Synapse Analytics Apache Spark REST APIs

Apache Spark Pools in Azure Synapse Analytics supports the following REST API operations that can be invoked using a HTTP endpoint:





API Docs

Create Or Update

Create a new Apache Spark Pool or modify the properties of an existing pool.




Delete a Apache Spark Pool pool.




Get the properties of a Apache Spark Pool.



List By Workspace

List all provisioned Apache Spark Pools in a workspace.




Update the properties of an existing Apache Spark pool.




Detailed documentation of all the REST API operations supported overall by Azure Synapse Analytics can be found with this link


Example: How to invoke an Apache Spark REST endpoint to create or update a spark pool

The Create Or Update REST API can be used to create a new Apache Spark Pool or change configurations of an existing pool, including the ability to upgrade/downgrade the Spark runtime version of a pool. For example, an existing Apache Spark pool with spark runtime version 3.1 can be upgraded to Spark version 3.2 without the need of deletion of the existing pool.


On the Postman UI

  1. Create a new HTTP request (File > New > HTTP Request or using the new tab(+) icon).
  2. In the HTTP verb drop-down list, select PUT.
  3. For Enter request URL, enter{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}?api-version=2021-06-01-preview where {subscriptionId} is the subscription Id of where the Azure Synapse workspace has been provisioned, {resourceGroupName} is the resource group of the workspace, {workspaceName} is name of the Synapse workspace and {bigDataPoolName} is the name of the Apache spark pool.
  4. On the headers tab enter a new key Content-Type with value application/json.
  5. Click on the Authorization tab, select Bearer Token in the Type dropdown menu and enter the AAD Token generated in the previous example.
  6. On the Body tab, select raw and select json on the dropdown menu and enter target Spark Pool JSON definition in the body. 






{ "location": "West US 2", "properties": { "sparkVersion": "3.2", "nodeCount": 6, "nodeSize": "Medium", "nodeSizeFamily": "MemoryOptimized", "autoScale": { "enabled": false, "minNodeCount": 6, "maxNodeCount": 6 }, "autoPause": { "enabled": true, "delayInMinutes": 30 } } }




Detailed list of configuration properties supported by Apache Spark Pools are documented in this link.

7. Click the send button. A successful creation or modification REST API operation will return a detailed provisioning JSON response like the below sample:



{ "properties": { "creationDate": "2022-10-28T21:03:07.2066667Z", "sparkVersion": "3.2", "nodeCount": 6, "nodeSize": "Medium", "nodeSizeFamily": "MemoryOptimized", "autoScale": { "enabled": false, "minNodeCount": 6, "maxNodeCount": 6 }, "autoPause": { "enabled": true, "delayInMinutes": 30 }, "isComputeIsolationEnabled": false, "sessionLevelPackagesEnabled": false, "cacheSize": 50, "dynamicExecutorAllocation": { "enabled": false }, "lastSucceededTimestamp": "2022-10-28T22:42:19.23Z", "isAutotuneEnabled": false, "provisioningState": "Provisioning" }, "id": "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}", "name": "{bigDataPoolName}", "type": "Microsoft.Synapse/workspaces/bigDataPools", "location": "westus2" }




Note: You can include an optional Boolean(true/false) parameter in the request URL to stop all currently running sessions/job on the target Spark pool.


The new request URL with forced termination of sessions:{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Synapse/workspaces/{workspaceName}/bigDataPools/{bigDataPoolName}?api-version=2021-06-01-preview&force={true}



Azure Synapse Analytics REST APIs REST APIs can be used for operation of various Synapse services. The REST APIs can be used to upgrade/downgrade Spark runtime version, auto-scale configuration, library management and more.


Our team publishes blog(s) regularly and you can find all these blogs here: 


For deeper level understanding of Synapse implementation best practices, please refer our Success By Design (SBD) site: 


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.