This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
We are excited to share the latest features and improvements in Azure Synapse Data Explorer aka Kusto that promise to make your data analysis experience more seamless and productive than ever before.
We are excited to announce the following new features and capabilities:
1. Synapse Real Time Analytics in Microsoft Fabric (Preview)
Earlier today, we launched Microsoft Fabric - an all-in-one analytics solution for enterprises that covers everything from data movement to data science, real-time analytics, and business intelligence. Microsoft Fabric brings together new and existing components from Power BI, Azure Synapse, and Azure Data Explorer into a single integrated environment.
The platform is built on a foundation of Software as a Service (SaaS), which takes simplicity and integration to a whole new level. Kusto is available as part of the Real time Analytics experience in Fabric, providing all the amazing query capabilities, performance and scale that customers are used to with Azure Synapse Data Explorer in a SaaSified experience. Sign up for the free trial.
For more information read the Real-time analytics in Fabric announcement blog and Fabric RTA docs.
2. Free clusters (GA)
Ever since we launched Kusto externally as “Azure Data Explorer”, we looked for ways to make Kusto and KQL accessible to everyone – regardless of whether they have an Azure subscription. Last year, we announced the start for free Kusto offering in Preview which allows anyone who is interested in big data analytics to claim their own free personal cluster. Free clusters Kusto offering is now officially GA from Preview! Learn more.
3. Azure Event Hubs - send data to Kusto directly from the Azure Event Hubs page (Coming soon)
Azure Event Hubs is a highly scalable data streaming platform that can handle millions of events per second. Event Hubs represents the "front door" for an event pipeline, often called an event ingestor in solution architectures.
Coming soon, with just a few clicks, you'll be able to bring your data from the Event Hubs resource page in the Azure portal, directly into Kusto, without the need for any additional setup or configuration.
The Azure Data Explorer interface makes it easy to preview your event hub data, rapidly ingest, transform, and query it. Once the data is in Azure Data Explorer you can also visualize your insights with charts and graphs, making it easy to share your findings with others
4. Azure Functions input & output bindings (Preview)
You can now integrate your Azure functions with Kusto via bindings - a declarative way of connecting external resources to Azure functions. With Kusto bindings, you can seamlessly read and write data from your Azure functions with minimal code in declarative fashion leveraging the Azure functions framework. There are two types of bindings available -
- Input binding: read data from Kusto.
- Output binding: write data to Kusto.
This new capability makes it simpler to build data processing pipelines that involve Kusto and enables you to easily integrate Kusto with other Azure services using Azure Functions. Learn more.
5. Azure Log Alerts support for Azure Data Explorer (Preview)
Azure Monitor Alerts allow you to monitor your Azure and application telemetry to quickly identify issues affecting your service. The Azure Monitor alerts is introducing now support for running queries on Azure Data Explorer (ADX) tables, and even joining data between ADX and data in Log Analytics and Application Insights.
As part of this newly added support, log alert rules now support managed identities for Azure resources – allowing you to see and control the exact permissions of your log alert rule.
To write queries to Log Search Alerts (LSA) you need to use the ADX(‘<cluster url>’) pattern. Learn more
6. Cosmos DB Synapse Link to ADX (GA)
Enabling near real time analytics on Cosmos DB data in a managed setting, i.e. Azure Data Explorer data connection is going to be Generally Available (GA) soon.
When using streaming ingestion, latency between Cosmos DB and ADX can be as low as sub seconds. You can now have Power BI direct query on your Cosmos DB data, rendering analytical queries in a second. You can control how the data is mapped from JSON documents to tables and transform it to suit your use case.
This brings the best of both worlds: fast / low latency transactional workload with Azure Cosmos DB and fast / ad hoc analytics with Azure Data Explorer. Learn more.
7. Serilog Connector (GA)
Serilog is a popular logging framework for .NET applications. Serilog allows developers to control which log statements are output with arbitrary granularity based on the logger's name, level, and message pattern. The Serilog sink, also known as an appender, for Azure Data Explorer streams your log data to Azure Data Explorer, where you can analyze and visualize your logs in real time.
8. NLog Connector (GA)
NLog is a popular logging framework for .NET applications. It provides a flexible and extensible way to log messages to various targets, such as files, databases, and event loggers.
NLog allows developers to configure log messages to be filtered, formatted, and routed based on their severity and context. The Kusto NLog Sink is a library that provides a way to send log messages from an NLog logger to a Kusto cluster. It is built on top of the Kusto .NET Client Library, which provides a set of APIs for interacting with Kusto.
9. Delta external table read support (Preview)
You can now create external tables over Delta Lake. ADX external tables operate on data stored and managed outside of the Azure Data Explorer cluster.
Delta Lake is a popular open-source storage layer. With this new feature, you can query Delta tables with ADX for scenarios where you want to query directly from the delta tables.
The table creation syntax supports auto inferring schema from your delta tables, but you can also provide it explicitly. Learn more.
10. External table read support for Cosmos db, MySQL and Postgres (Coming soon)
Until now, querying cosmos db, mySQL and postgres was possible using the following plugins: (cosmosdb_sql_request, mysql_request and postgresql_request).
The only SQL database that was supported by external tables was SQL Server.
By the end of June, we’ll introduce the ability to create external tables over these types of databases as well, which will allow writing queries in a more convenient way.
11. Improved performance for SQL requests (GA)
Requests to SQL will become more optimized, both when using external tables, and when using the sql_request_plugin . Where possible, Kusto will push down predicates and functions into the SQL query.
For example, the following query used to pull all the data from the Students external table, and then Kusto Engine applied the filter on FirstName, and counted the results:
`external_table(“Students”) | where FirstName == “Bob” | count`
With push-down predicates, instead of pulling all the data from SQL, Kusto Engine will send a query to SQL that looks roughly like this:
`SELECT COUNT(*) FROM Students WHERE FirstName = 'Bob'`
This will reduce the load from the SQL Server, and also from Kusto Engine, resulting in much more performant queries that consume less resources on both sides.
12. New cosine similarity function for vector similarity search (GA)
If you are dealing with Large Language Models (LLMs), OpenAI and embeddings, you can use ADX for vector similarity search. We have published a new user-defined function series_cosine_similarity_fl to perform vector similarity searches on top of the vectors stored in ADX. Learn more.
13. Kusto Emulator on Linux (GA)
You can now run the Kusto Emulator as a Linux container.
Unlike the Windows container, which was released almost a year ago, the Linux Container is smaller, loads faster and runs on both Windows and Linux. You can still do local development and automated testing, but now in Linux environments and with a smaller footprint!
14. Python plugin improvements (GA):
The Python plugin allows you to run a user-defined function (UDF) using a Python script.
We have updated the Python image to 3.10.8 and latest packages and improved the sandboxing technology to use Hyper-V containers which are more secure and performs better on latest SKUs
15. Followers - function level sharing (GA)
The follower database feature allows you to attach a database located in a different cluster to your Azure Data Explorer cluster.
Until now, you could only share specified tables and external tables with other clusters but had no control over specific functions. But now you can share specific functions, which provides you full control over which entities to share.
You can use the functionsToInclude, and functionsToExclude parameters to control this behavior. Learn more.
16. Confidential Compute support for Azure Data Explorer (Preview)
The Azure Data Explorer Confidential solution is based on the AMD EPYC SEV-SNP technologies. This AMD technology adds an advanced level of security for virtualized environments, by encrypting all data found in memory, and offering a hierarchical paging model that allows the admin of a VM only to access the memory of the VM. This creates an isolated memory region within each VM that is inaccessible to other VM administrators and the hypervisor admin.
The ADX Confidential computing solution is based on the AMD ECasv5 SKU family. To create a confidential Azure Data Explorer cluster, select an ECASV5 SKU during cluster creation. You can use either ARM of the Azure Portal to deploy or migrate to a Confidential Computing cluster.
17. New Azure Policies (GA)
Five new built-in Azure policies are now available to further help you improve your security posture with ADX.
- Public network access on Azure Data Explorer should be disabled
- Configure Azure Data Explorer to disable public network access
- Azure Data Explorer cluster should use private link
- Azure Data Explorer should use a SKU that supports private links
- Configure Azure Data Explorer clusters with private endpoints
18. Azure for Operators (AFO)
The Azure for Operators AIOps program is helping Operators modernize some of the most complex networks in the world using the same technologies that power Microsoft Azure.
Two new services were announced at Mobile World Congress, Azure Operator Insights and Azure Operator Service Manager. Azure Operator Insights enables the collection and analysis of massive quantities of network data gathered from complex multi-part or multi-vendor network functions. It delivers insights for operator-specific workloads to help operators understand the health of their networks and the quality of their subscribers' experiences. Azure Operator Insights leverages Azure Data Explorer's unparalleled ability to analyze complex datasets to deliver near real-time insights to Operators. Operators looking to reduce operational costs are pairing Azure Operator Insights with Azure Operator Service Manager to automate modifications to their networks based on these same insights.
19. Well-Architected Review for ADX (GA)
Well-Architected Framework (WAF) allows customers to assess their workload according to the following pillars:
- Reliability
- Cost management
- Operational excellence
- Security
- Performance efficiency.
Azure Data Explorer is now included as one of the available WAF analytics services. After assessing your workload, you'll receive pragmatic recommendations based on your specific needs. Over time, you can improve your score by following your personalized recommendations. Learn more.
20. Kusto Detective Agency Season 2
And finally, we are excited to bring Season 2 of your beloved Kusto Detective Agency (KDA)- a gamified way of learning Kusto Query Language (KQL)
Thousands of data enthusiasts have participated in Season 1 so far, proudly flashing their badges on social media platforms. With Season 2, we are taking the KDA experience to the next level with more cases, more fun and even more exciting prizes to be won.
Recruiting now at: https://detective.kusto.io/
We would love to hear your feedback and overall experience with these new capabilities.