This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
Native streaming capabilities for ingestion and near real-time analytics with Azure Synapse Analytics (formerly SQL Data Warehouse) have been available since the launch at Microsoft Ignite. Onboarding and managing your streaming workloads for SQL analytics has never been easier.
Previously setting up and managing streaming workloads was a complex and cumbersome process for Azure Synapse. Users have to setup and configure separate services such as Azure Stream Analytics and Databricks where each service has their own billing and security models including separate provisioning experiences. In addition, the integration with Azure Synapse for these two services lacks critical streaming capabilities which are adoption blockers for many customers:
- Azure Stream Analytics uses the SqlBulkCopy API which had causes data ingestion bottlenecks at the control node of an Azure Synapse SQL pool. This approach is not able to meet customer workloads with high-throughput requirements and can sometimes lead to reliability issues.
- Although the Databricks connector uses PolyBase for high-throughput ingestion, the connector requires CONTROL permissions on the database which grants the load user access to all schemas and tables in the data warehouse - such high elevated permissions for streaming ingestion to a subset of tables poses security concerns for customers.
With the latest streaming capabilities, all these limitations are removed to make onboarding and managing streaming workloads as simple as possible:
- The COPY command is now leveraged under the covers for high-throughput data ingestion directly to compute nodes
- Users no longer require CONTROL permissions on their database set up their streaming jobs - only ADMINISTER DATABASE BULK OPERATIONS and INSERT permissions on the specified table is required
- Provisioning and managing streaming jobs can be done directly in the portal of Azure Synapse without opening multiple portal blades, notebooks, and editors
- Automatic schema detection on the streaming input source enables a simple and code free set-up process including automatic table creation
Stay tuned for additional streaming capabilities and enhancements to our user experience! Help us by joining our preview and signing up directly in the Azure Synapse portal:
