Stream Analytics updates – Ignite Fall 2021: New outputs, new security options and much more!

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Today we are releasing a new update of Azure Stream Analytics that introduces numerous new features to add new options for security, reliability, as well as introducing new outputs and improvement to our SQL language. Here is the list of significant changes. As a Platform-as-a-Service the product is continuously updated so you always get the latest version and can access all these new features automatically.

Output to Azure Data Explorer (Preview)

Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure Data Explorer is very complementary to Azure Stream Analytics as Azure Stream Analytics reacts to the data in real-time and can trigger alerts or transform the data as it comes and can send data to Azure Data Explorer where users can analyze extremely large volumes of data.

Today we are announcing that Azure Stream Analytics can directly output data to Azure Data Explorer, simplifying architecture where customers need both hot and warm path analytics on streaming data. This feature will be added to all Stream Analytics regions progressively. 

Output to Azure PostgreSQL (Preview)

We are also introducing output to Azure PostgreSQL, giving customers a new option to output streaming data to relational database.  With this new output, Azure Stream Analytics jobs write to Azure PostgreSQL Single Server, Flexible Server (Preview) and Hyperscale (Citus), supporting high throughput insert of streaming data to existing table.

SQL Query Language improvements

As we continue to bring the Stream Analytics Query Language closer to T-SQL, we are extending existing functions, and adding new ones, that will simplify queries for the most common tasks and bring stream and batch closer together.

In particular we added 12 String manipulation functions, Unicode/NCHAR conversion support, extended support of MIN/MAX aggregate to VARCHAR(MAX), and new bitwise operators.

More details and examples are available in this blog post: Ignite 2021 - Stream Analytics Query Language Improvements.

Extended support for Managed Identity

Managed Identity is a key security feature that eliminates the need for developers to manage credentials, and help to make your Stream Analytics jobs more secure, and remove the need to rotate your keys or connection strings. Today we are now extending support for Managed Identity with the following announcements:

• Managed Identity for Event Hubs input and output is now generally availability (GA). With the use of trusted services, this allows secure end-to-end communications between Event Hubs and Stream Analytics.
• Managed Identity for Blob/ADLS Gen2 Stream input is now generally available.
• Managed Identity for Synapse SQL Pools output is now generally available.
• Managed Identity for customer storage account is now generally available.
• We are also added managed Identity for Reference Data:
  • Blob reference data, generally available
  • SQL reference data, in preview

With these new announcements, Stream Analytics supports Managed Identity for the following inputs and outputs. More will be added in the future.

Policy for Data Exfiltration Prevention

In addition to the previous features, we are introducing a new policy for data exfiltration prevention, enabling users to make sure the highest level of security is applied to their clusters and jobs.

New Regions and support for Azure Availability Zones

In the last 6 months, we continued to extend the global footprint of Azure Stream Analytics, by adding 10 new regions: Australia Central, China East 2, China North 2, Germany West Central, Norway East, South India, South Africa North, Switzerland North, United Arab Emirates North and West US3. With this announcement, Stream Analytics is now available in 38 Azure regions worldwide.

Additionally Stream Analytics does now offer support for Availability Zones with Dedicated Cluster. Any new Dedicated Cluster will automatically benefit from Availability Zones, and in case of disaster in a zone will continue to run seamlessly by failing over to the other zones without the need of any user action. Availability Zones provide customers with the ability to withstand datacenter failures through redundancy and logical isolation of services. This will significantly reduce the risk of outage for your streaming pipelines.

New job diagram for troubleshooting

New job diagram in Azure portal and in VS code can help you visualize your job pipeline. It shows inputs, outputs, and query steps. You can use the job diagram to examine the metrics for whole job or each step. This will help you to understand how the job is progressing and to isolate the source of a problem when you troubleshoot issues. For example, How much data is coming in vs how much data is backlogged.

Job Diagram in VS Code

Customers can now use job diagram to view their pipeline for cloud jobs. You can also view logs to understand the errors.

Job diagram is VS Code

Job diagram is VS Code

Job Diagram in Azure portal

Customers now have improved User Experience for their job diagram with job topology, metrics and job summary in a single view. We also completely changed the backend of this feature to improve job diagram availability and responsiveness.

Job diagram is Azure Portal

Other improvements

This update includes hundreds of performance and functional improvements. While we are not releasing full release notes, but we wanted to call out the following ones:

• We are also now using AAD regional endpoints, which enables more resiliency in case of AAD global outage.
• Classifying ref data delta errors as user actionable: When a job's reference data is configured for delta refresh, but the schema is missing the required "operation" column, or that column contains an invalid value, this is currently classified as a system error, but this is actually a user error. This change reclassifies them from "job fatal" to "user actionable job fatal".

To get started with Azure Stream Analytics, you can use one of our quick starts on our documentation page .