This installment is part of a broader series to keep you up to date with the latest features in Azure Sentinel. The installments will be bite-sized to enable you to easily digest the new content.
This blog is a collaboration between myself and my colleague, Sreedhar Ande.
What truly sets Azure Sentinel apart from other SIEM tools or other security solutions in the market is the extensive use of machine learning to fuel built-in analytics and custom machine learning models. These capabilities are the culmination of decades of research and experience protecting Microsoft services at massive scale by Microsoft security experts. As you might already be aware, Microsoft Ignite 2020 announcements highlighted some of the most recent innovations in this space.
We are delighted to announce that 32 new Fusion detections and Build Your Own Machine Learning framework are now available in public preview! Below has a recap of what these features are and how they work.
Fusion Detections
What is Fusion technology in Azure Sentinel?
Using machine learning, Fusion detections combine low- and medium-severity alerts from Microsoft and 3rd-party security products into high-severity incidents. By design, these incidents are low-volume, high-fidelity, and high-severity. Here is an example of how a Fusion incident looks like in Azure Sentinel portal.
The main goals of Fusion detections can be summarized into two points.
- Fusion detects threats that fly under radar: Azure Sentinel can automatically detect multistage attacks by identifying combinations of anomalous behaviors and suspicious activities that are observed at various stages of the kill-chain. On the basis of these discoveries, Azure Sentinel generates incidents that would otherwise be difficult to catch.
- Alert fatigue reduction: Fusion incorporates graph-based machine learning and a probabilistic kill chain to reduce alert fatigue by 90 percent.
For more details of how Fusion technology works behind the scene, please check out this excellent article by our colleague, Ram Shankar Siva Kumar.
What are the new Fusion detections?
Our Fusion team recently released 32 new Fusion detections in public preview, reaching a total of 90 Fusion incident types which are turned on by default in Azure Sentinel. These additional detections fall into eight scenario types.
MDATP + Palo Alto Network firewall:
- Suspicious remote WMI execution followed by anomalous traffic flagged by Palo Alto Networks firewall
- Suspected use of attack framework followed by anomalous traffic flagged by Palo Alto Networks firewall
AAD IP + MCAS:
- Suspicious inbox manipulation rules set following suspicious Azure AD sign-in (5 distinct detections)
- Multiple VM creation activities following suspicious Azure Active Directory sign-in (5 distinct detections)
- Multiple VM delete activities following suspicious Azure AD sign-in (5 distinct detections)
- Suspicious email deletion activity following suspicious Azure AD sign-in (5 distinct detections)
- Multiple Power BI report sharing activities following suspicious Azure AD sign-in (5 distinct detections)
- Suspicious Power BI report sharing following suspicious Azure AD sign-in (5 distinct detections)
How to enable and use Fusion detections
Under Analytics blade in Azure Sentinel portal, in your Active Rules view, a built-in rule of Fusion rule type named “Advanced Multistage Attack Detection” is enabled by default for all Sentinel workspaces. You have the option to disable the rule any time. There is no extra cost to use this detection rule on top of the normal data ingestion and storage cost. All you need for the rule to work is to have your data connectors configured and data ingested correctly. To see what data connector sources are required for each Fusion incident type, please refer to the documentation.
To get step-by-step instructions about Fusion in Azure Sentinel, please refer to our documentation, which has been revamped with updated detection descriptions, now includes MITRE ATT&CK Tactics and Techniques, and is now organized by threat classifications for easier navigation.
Build Your Own Machine Learning (BYO-ML)
Many security organizations understand the value of machine learning for security, though not many of them have the luxury of professionals who have expertise in both security and ML. We designed the framework Build-Your-Own ML (BYO-ML) for security organizations and professionals to grow with us in their ML journey. Organizations new to ML, or without the necessary expertise, can get significant protection value out of Azure Sentinel’s built-in ML capabilities.
ML detection models can adapt to individual environments and to changes in user behavior, to reduce false positives and identify threats that would not be found with a traditional approach. Azure Sentinel makes it easier for data scientists in these organizations to unlock these insights with a BYO-ML framework.
What is the Build Your Own Machine Learning (BYO-ML) platform?
For organizations that have ML resources and would like to build customized ML models for their unique business needs, we offer the BYO-ML platform. The platform makes use of the Azure Databricks/Apache Spark environment and Jupyter Notebooks to produce the ML environment. It provides the following components:
- A BYO-ML package, which includes libraries to help you access data and push the results back to Log Analytics (LA), so you can integrate the results with your detection, investigation, and hunting.
- ML algorithm templates for you to customize to fit specific security problems in your organization.
- Sample notebooks to train the model and schedule the model scoring.
Besides all this, you can bring your own ML models, and/or your own Spark environment, to integrate with Azure Sentinel.
For more details on BYO-ML platform, please check out this excellent blog by our colleague, Andi Comisioneru. For supported use cases, please refer to the documentation.
How to use Build Your Own Machine Learning (BYO-ML) platform
To build custom ML models on your data, you have two options.
- For smaller amounts of data, like alerts and anomalies, you can use Azure ML to run models hosted in the Azure Sentinel Notebooks (new menu option currently in Preview). Azure Machine Learning offers Intellisense for improved ease of use, support for existing Jupyter and JupyterLab experiences, as well as point-in-time notebook snapshots and a notebook file explorer for easy notebook collaboration. Dedicated compute and multiple pricing options provide increased flexibility and control. Take advantage of built-in security analytics via MSTICPy and Jupyter notebook templates help you get started.
Note: To use the Notebooks, you must first create an Azure Machine Learning (ML) workspace. For step-by-step instructions on how to create an Azure Machine Learning (ML) workspace, please refer to the documentation.
- For development and operationalization of models built on larger data, like analyzing feeds of raw data, you will need to make this data accessible to the ML model in Azure Databricks.
Apache Spark™ provides a unified environment for building big data pipelines. Azure Databricks builds on this environment, providing a zero-management cloud platform, holistically addressing the platform needed for data analysts to develop their custom ML based security analysis.
You can either bring your raw data directly to the Azure Databricks ML environment, via EventHub or Azure Blobs or you can use the capabilities provided with Azure Sentinel, to export the data from Azure Sentinel Log Analytics tables. Regardless of the export methods used for raw data, you can use the libraries provided by BYO-ML framework to import the scoring of the ML model back into Sentinel Log Analytics tables for further processing and creating incidents.
You can either set up a new Azure Databricks environment or use one already set up for other use. To set up a new Databricks environment, please refer to the quickstarts document (note that MMLSpark used by our algorithm requires Spark 2.4.5).
On Azure Sentinel roadmap, we plan to support Azure Synapse in addition to Azure Databricks as a BYO-ML development environment.
Get started today!
We encourage you to explore these machine learning innovations in the Azure Sentinel to detect and protect your organization from threats.
Try it out, and let us know what you think!
You can also contribute new Notebooks in Azure Sentinel. Get started now by joining the Azure Sentinel BYOML GitHub community.