Azure Sentinel Notebooks Ninja Part 3: Overview of the Pre-built Notebooks – the Grand List

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

This installment is part of a broader learning series to help you become a Jupyter Notebook ninja in Azure Sentinel. The installments will be bite-sized to enable you to easily digest the new content. 

 

Through Part 1 and Part 2 of this Azure Sentinel Notebook Ninja series, we’ve discussed the concepts and activities to best become acclimated with Jupyter notebooks for Azure Sentinel. The next step in our process is understanding the value of having ready-made notebooks ready for use as part of the solution.

 

When a customer stands-up Azure Sentinel for the first time, there are a number of additional pieces of ready-to-use collateral that are provided out-of-the-box including Analytics Rules, Hunting queries, Connectors, Solutions, Workbooks – and – you guessed it – Notebooks.

 

The notebooks are mostly one of three types:

 

  • Exploration notebooks. These are meant to be used as they are or with your own customizations to explore specific hunting and investigation scenarios. Examples of this type include the Entity explorer series. (“Entity” refers to items such as hosts, IP addresses, accounts, URLs, etc.)
  • Simple How-To notebooks like the Get Started notebook.
  • Sample notebooks. These are longer and are meant to be instructional examples following a real or simulated hunt or investigation.

rodtrent_0-1632174794690.png

 

The following Grand List represents the notebooks that are provided when setting up Azure Sentinel.

 

Azure Sentinel Notebooks – The Grand List

 

Notebook Name

Type

Description

Special Requirements

 

 

 

 

A Getting Started Guide For Azure Sentinel ML Notebooks

Getting Started

This notebook guides you through the basic steps of using notebooks for security analysis. It covers all the basic steps you need to understand to start using the notebooks provided with Azure Sentinel.

Data type(s): SigninLogs

 

External service(s): VirusTotal, MaxMind, MSTICPy

 

Package(s): MSTICPy

A Getting Started Guide For PowerShell AML Notebooks

Getting Started, PowerShell

This notebook takes you through the basics needed to get started with PowerShell notebooks that leverage Azure Sentinel data and APIs.

Data type(s): SecurityEvent

 

 

A Tour of Cybersec notebook features

Getting Started

This notebook takes you through some of the features of Azure Sentinel notebooks and MSTICPy. It's a good second notebook to explore after "A Getting Started Guid for Azure ML notebooks". It covers data queries, visualization, data analysis, enrichment with threat intelligence and pivot functions. It can be run against you Azure Sentinel workspace or standalone, with sample data.

Data type(s): SecurityEvent, SecurityAlert

 

If no data types are available, sample data will be used.

 

Package(s): MSTICPy

Configuring your Notebook Environment

Configuration

This notebook takes you through detailed setup of your settings for Azure Sentinel Notebooks and the MSTICPy library. It covers: Setting up your Python environment for notebooks (not required for AML notebooks), Creating and editing your msticpyconfig.yaml file, Understanding and managing you config.json file.

Package(s): MSTICPy

Credential Scan on Azure Blob Storage

Hunting

This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Blob Storage using Azure SDK for Python.

Package(s): MSTICPy

Credential Scan on Azure Data Explorer

Hunting

This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Data Explorer using Azure SDK for Python and KQL.

Package(s): MSTICPy

Credential Scan on Azure Log Analytics

Hunting

This notebook provides step-by-step instructions and sample code to detect credential leak into Azure Log Analytics using Azure SDK for Python and KQL.

Data type(s): SecurityEvent

 

Package(s): MSTICPy

Entity Explorer – Account

Hunting

Use queries and visualizations to help you assess the security state of an AAD/O365 account or an account on a local host. It includes examining Azure Active Directory (AAD) and Office365 activity for an account and identifying any related anomalous behavior. Allows you to correlate the IP addresses in related events with threat intelligence sources.​

Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat, AAD, OfficeActivity

 

Package(s):  kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2

 

Entity Explorer - Domain and URL

Hunting

Use queries and visualizations to help you assess the security state of a DNS host or domain name. Examine DNS requests to look for malicious DNS usage, such as DNS tunneling, as well as anomalous DNS name lookups. Query threat intelligence and domain ownership using WhoIS information for a specific DNS name, to get a better understanding of the domain name’s reputation.

Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat, DNS, Syslog

 

Package(s):  kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2

Entity Explorer - IP Address

Hunting

Brings together a series of queries and visualizations to help you assess the security state of an IP address. It works with both internal addresses and public addresses.

Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat

 

External service(s): VirusTotal, Alienvault OTX, IBM Xforce

Entity Explorer - Linux Host

Hunting

This notebook brings together a series of tools and techniques to enable threat hunting within the context of a singular Linux host. The notebook utilizes a range of data sources to achieve this but in order to support the widest possible range of scenarios this Notebook prioritizes using common Syslog data.

Data type(s): Syslog, Auditd_CL, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat

 

Package(s):  kqlmagic, msticpy, pandas, pandas_bokeh, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, ipwhois, folium, maxminddb_geolite2

Entity Explorer - Windows Host

Hunting

Brings together a series of queries and visualizations to help you determine the security state of the Windows host or virtual machine that you are investigating. It looks for related alerts, allows you to examine logon sessions and processes, check process command lines for IoCs and explores network traffic between the host and external endpoints.

Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Heartbeat

 

Package(s): kqlmagic, msticpy, pandas, numpy, matplotlib, bokeh, networkx, ipywidgets, ipython, scikit_learn, dnspython, ipwhois, folium, maxminddb_geolite2

Guided Hunting - Anomalous Office365 Exchange Sessions

Hunting

Brings together a series of data science techniques to help you hunt for anomalous sessions in your Office Exchange logs. It queries the OfficeActivity table, creates sessions from the PowerShell cmdlets (e.g. Set-Mailbox), trains a model and then visualizes the sessions.

Data type(s): OfficeActivity

 

Package(s):  msticpy, pandas, kqlmagic

Guided Hunting - Base64-Encoded Linux Commands 

Hunting

This notebook is a collection of tools for detecting malicious behavior when commands are Base64-encoded. It allows you to specify a workspace and time frame and will score and rank Base64 commands within those bounds. It utilizes multiple data sources, primarily focusing on Azure Sentinel Syslog data augmented by telemetry from the MSTIC research branch of the AUOMS audit collection tool.

Data type(s): Syslog, Auditd_CL, SecurityAlert, AzureNetworkAnalytics_CL

 

Package(s):  kqlmagic, msticpy, pandas, numpy, matplotlib, networkx, seaborn, datetime, ipywidgets, ipython, dnspython, folium, maxminddb_geolite2, BeautifulSoup

Guided Hunting – Covid-19 Themed Threats

Hunting

Brings together a number of techniques and approaches to allow an analyst to hunt for indicators of Covid-19 themed threats within an organization's data. This includes hunting templates for network, cloud and host logs in order to find potentially malicious Covid-19 themed activity.

Data type(s): SecurityEvent, OfficeActivity, CommonSecurityLog

 

Package(s):  MSTICPy

Guided Investigation - Anomaly Lookup

Investigation

Gain insights into the possible root cause of an alert by searching for related anomalies on the corresponding entities around the alert’s time. This notebook will provide valuable leads for an alert’s investigation, listing all suspicious increase in event counts or their properties around the time of the alert, and linking to the corresponding raw records in Log Analytics for the investigator to focus on and interpret.

Data type(s): SecurityEvent, SecurityAlert, AzureNetworkAnalytics_CL, Auditd_CL, OfficeActivity, CommonSecurityLog

 

Package(s):  MSTICPy

Guided Investigation - Process Alerts

Investigation

This notebook is intended for triage and investigation of security alerts. It is specifically targeted at alerts triggered by suspicious process activity on Windows hosts. Some of the sections will work on other types of alerts but this is not guaranteed.

Data type(s): SecurityEvent

 

External service(s): OTX, VirusTotal, XForce

Guided Investigation - SolarWinds Post

Investigation

This notebook assists defenders in hunting for SolarWinds post compromise Tactics , Tools and Procedures (TTPs) across different environments both on-premises and cloud data sources.

Data type(s): SecurityEvent, AuditLogs, SigninLogs, OfficeActivity

 

External service(s): OTX, VirusTotal, XForce

Guided Investigation – Alert Triage

Investigation

Rapidly triage alerts raised by a range of sources (Azure Sentinel, MDE, MCAS, ASC, etc.) by enriching alert data with Threat Intelligence and OSINT. This notebook is intended to be used by analysts as part of a standard operating procedure. The notebook uses UI widgets and simple workflow to ensure an easy introduction to notebooks for all levels of analyst levels. This Notebook uses MSTICpy to connect to Threat Intelligence sources

Data type(s): SecurityAlert

 

Package(s):  MSTICPy

Guided Web Shell Investigation - MDE Sentinel Enrichments

Investigation

This notebook investigates Microsoft Defender for Endpoint (MDE) web shell alerts. The notebook will guide you through steps to collect MDE alerts for web shell activity and link them to server access logs to identify potential attackers.

Data type(s): SecurityAlert, W3CIISLog

 

Package(s):  MSTICPy

Hands on - Data Discovery using Azure REST API

Getting Started

This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Sentinel data discovery by using Azure REST API.

Data type(s): SecurityEvent

Hands on - Surfing Your Data using Azure SDK for Python

Getting Started

This notebook will provide step-by-step instructions and sample code to guide you through Azure authentication, Sentinel log data discovery by using Azure SDK for Python and Kusto Query Language (KQL).

Data type(s): SecurityEvent

Machine Learning in Notebooks Examples

Getting Started

This notebook template guides you through using time series analysis to detect anomalous network activity, clustering to highlight unusual logon sessions, and using Markov Chain to identify anomalous sequences in events.

Data type(s): SecurityEvent

 

Package(s):  MSTICPy

 

These notebooks and more also exist in the official Azure Sentinel GitHub repository.  This repository contains notebooks contributed by Microsoft and the community to assist hunting and investigation tasks in Azure Sentinel. There are a number of notebooks in the GitHub repository that are valuable additions, but are not supplied automatically when you stand-up Azure Sentinel. You should regularly review this repository for more, or use the RSS feed to be notified of additions and changes.

 

Notebooks, like all other components and features for Azure Sentinel, are under constant review and undergoing constant improvement. Improvements and changes come from feedback and suggestions from our customers. You can use the DL (asinotebooks@service.microsoft.com) to send your questions, issues, and feedback and our various product teams will monitor and respond.

 

As these updates are made available, we’ll update the Grand List.

 

We are super-excited to be bringing this series (and the training) to you! For those that have already registered, the first session is scheduled for September 30, 2021. If you want to be included in additional training sessions, register using the form.

 

Look for more great knowledge on Azure Sentinel Notebooks as we prepare to deliver Part 4 of this series: How to create your own notebooks from scratch and how to customize the existing ones.

 

More reading/tutorial resources:

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.