Automating the deployment of Sysmon for Linux 🐧 and Azure Sentinel in a lab environment 🧪

Blog-page.png


Thanks to Kevin Sheldrake, Russell McDonald, Jessen Kurien and Ofer Shezaf for making this blog possible.


 


Today, we celebrate 25 years of Sysinternals, a set of utilities to analyze, troubleshoot and optimize Windows systems and applications. Also, as part of this special anniversary, we are releasing Sysmon for Linux, an open-source system monitor tool developed to collect security events from Linux environments using eBPF (Extended Berkeley Packet Filter) and sending them to Syslog for easy consumption. Sysmon for Linux is built on a library also released today named sysinternalsEBPF which is built on libbpf including a library of eBPF inline functions used as helpers. 


 


In this post, I will show you how to automatically deploy a research lab environment with an Azure Sentinel instance and a few Linux virtual machines with Sysmon for Linux already installed and configured to take it for a drive and explore its coverage.


 


As always, before getting into the technical parts of the main topics, it is important to understand some of the fundamental concepts behind Sysmon for Linux.  


 


What is eBPF?


According to the eBPF Foundation, eBPF is a technology that allows programs to run in a sandbox in an operating system kernel. In other words, eBPF enables programmers to write code which gets executed in kernel space in a more secure and restricted way in order to add additional capabilities to the operating system at runtime. 


 


Some of the use cases for eBPF are: 



  • Security: Combining visibility and better level of control to secure systems. 

  • Tracing and profiling: Powerful and unique insights to troubleshoot system performance. 

  • Networking: A natural fit for all packet processing requirements of networking solutions. 

  • Observability and monitoring: Collection and in-kernel aggregation of custom metrics.


 


Why eBPF for Sysmon for Linux? 


From an event-tracing perspective, eBPF allows us to write event-driven programs and have pre-defined hooks into operations such as system calls, network connections, file write/read, etc. We can then collect those events and use them to understand adversary behavior during research or an investigation. As mentioned before, Sysmon for Linux uses its own library “sysinternalsEBPF” to handle the security events monitoring process.


 


Cyb3rWard0g_0-1634229152075.png


 


You can find more information about the implementation of the new sysinternals EBPF library in the following resources: 



 


Installing Sysmon for Linux 


All the information presented here about the installation is available in its own GitHub repository:



 


Register Microsoft Key and Feed 


Sysmon for Linux requires the following packages during installation: 



  • sysinternalsebpf (.DEB or .RPM) 

  • sysmonforlinux (.DEB or .RPM)


For example, for Ubuntu you can run the following (More examples in the INSTALL documents above):


 


wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

 


 


Install Packages 


Depending on the Linux distribution and package manager, you can use the apt dependency resolver (Debian based distros) or the RPM package manager (Fedora based distros). 


 


Following the Ubuntu example, you can run the following commands to install sysinternalsEBPF and Sysmon in that order


 


sudo apt-get update
sudo apt-get install sysinternalsebpf
sudo apt-get install sysmonforlinux

 


 


Next, you should be able to run the sysmon command:


 


sysmon –h

 


 


Cyb3rWard0g_1-1634229152079.png


 


Run Sysmon as a Service 


Finally, we can use the sysmon binary to install and run Sysmon as a service with a specific Sysmon config (like how one installs Sysmon for Windows).


 


sudo sysmon -accepteula -i sysmonconfig.xml

 


 


Cyb3rWard0g_2-1634229152004.png


 


Explore Syslog Events  


You can explore Sysmon events from the Syslog log. Later in this post, I will show you how to use other tools to show Sysmon events in a more user-friendly view 😉


 


tail –f /var/log/Syslog

 


 


Cyb3rWard0g_3-1634229152008.png


 


The Sysmon for Linux Configuration 


At the time of writing, the Sysmon schema version is 4.81, check here for latest version. An example of a config to collect all events can be found in the following link (Not recommended to use in a production environment due to the large number of events generated):


 


SysmonForLinux-CollectAll-Config.xml (github.com) 


 
Use the following command to update the Sysmon config:


 


sudo sysmon –c newconfig.xml

 


 


Configuration Options 
















Option 



Description 



Sample 



FieldSizes 



specify how long you want fields to be so you can avoid the Syslog overrun/broken XML problem 



<FieldSizes>CommandLine:50,Image:50</FieldSizes> 



 


Available Events 






































Event ID 



Description 



1 



Logs when a new process is created. 



3 



Logs TCP/UDP connections on the machine 



5 



Logs when a process terminates. 



9 



Logs when a process conducts reading operations, from the drive. 



11 



Logs when a file is created or overwritten. 



16 



Logs when the local Sysmon configuration is updated. 



23 



Logs when a file is deleted by a process. 



 


All the events in Sysmon for Linux are already documented as data dictionaries in the open-source project OSSEM. You can access that information in the following link:


 


 https://github.com/OTRF/OSSEM-DD/tree/main/linux/sysmon 
 


The MSTIC Sysmon for Linux Configuration 


The MSTIC R&D team is sharing a few configuration files as part of the release of this project and will be maintaining them as we use them for research and development of detections.  


 


How do we automate the installation process? 


The installation of Sysmon for Linux can be automated with the following bash script 


 


https://raw.githubusercontent.com/OTRF/Blacksmith/master/resources/scripts/bash/Install-Sysmon-For-Linux.sh  
 


What about a full lab environment? Enter Azure Sentinel To-go!  


Azure Sentinel2Go is an open-source project developed to expedite the deployment of an Azure Sentinel lab along with other Azure resources to expedite research and the development of detections. 
 
https://github.com/OTRF/Azure-Sentinel2Go


 


Azure Sentinel + Sysmon for Linux Environment 


We have updated our previous Linux environment and we can now deploy everything needed for a small research lab with Sysmon for Linux configured and an Azure monitor agent sending logs to Azure Sentinel:


 


Azure-Sentinel2Go/grocery-list/Linux at master · OTRF/Azure-Sentinel2Go (github.com)


 


We were able to use Azure Resource Manager (ARM) templates and a bash script to automate the whole setup. These are all the resources used for each component of the research lab:  




 


Deploying the Lab Environment  



 


Cyb3rWard0g_4-1634229152010.png


 



  • Fill out the following parameters:

    • Subscription (selected by default)

    • Resource group

    • Region (selected by default) 

    • Admin Username

    • Admin Password

    • Remote Access Mode (AllowPublicIP selected by default. You can also use Azure Bastion Host. You would just need to set the Allowed IP Addresses parameter to *)

    • Allowed IP Addresses (If you use the default access mode AllowPublicIP, use your home or office public IP address to only allow access from secure places. 



  • Click the Review > Create buttons to start the deployment 

  • You can go to your resource group and explore all the resources being deployed 


  


Cyb3rWard0g_5-1634229152012.png


 



  • Wait around 5-10 minutes! You should be good to go! 


 


Cyb3rWard0g_6-1634229152015.png


 



  • You can go to the resource group and see all the resources  


 


Cyb3rWard0g_7-1634229152019.png


 


Validate Deployment 


It is very important to validate if everything was deployed properly before exploring events from Sysmon.


 


Sysmon running as a service 


SSH to all your VMs and run the following commands


 


systemctl status sysmon

 


 


Cyb3rWard0g_8-1634229152099.png


 


Explore Syslog Events  


You can explore Sysmon events from the Syslog log.


 


tail –f /var/log/Syslog

 


 


Cyb3rWard0g_9-1634229152043.png


 


Explore Sysmon Events via sysmonLogView 


Sysmon also comes with a binary named sysmonLogView to explore sysmon events in a friendly format.


 


Cyb3rWard0g_10-1634229152094.png


 


Cyb3rWard0g_11-1634229152083.png


 


Run the following commands to explore Sysmon event id 1 (ProcessCreate) events locally:


 


sudo tail -f /var/log/syslog | sudo /opt/sysmon/sysmonLogView -e 1

 


 


linux-sysmon-tail-sysmonlogview.png


 


Azure Sentinel 


Check if you are getting Syslog events via the Azure Sentinel interface: 



 


sysmon-azure-sentinel.png


 


Next, click on logs and run the following Kusto query to see if all your endpoints are generating events and are being collected by the Azure Log Analytics agent:


 


Syslog
| summarize count() by Computer

 


 


sysmon-azure-sentinel-query.png


 


Querying Sysmon for Linux 


You can query Sysmon for Linux logs by using the Syslog table with the following Kusto query:


 


Syslog
| extend EventID = parse_xml(SyslogMessage).Event.System.EventID
| extend EventData = parse_xml(SyslogMessage).Event.EventData.Data
| mv-expand bagexpansion=array EventData
| evaluate bag_unpack(EventData)
| extend Key=tostring([‘@Name’]), Value=[‘#text’]
| evaluate pivot(
Key, any(Value), TimeGenerated, TenantId, SourceSystem,
EventID, Computer, Facility, SeverityLevel, HostIP, MG, Type, _ResourceId
)
| summarize count() by tostring(EventID)

 


 


Additionally, as part of the ASIM (Azure Sentinel Information Model) project, we have created parsers for Sysmon for Linux. The parsers get imported automatically by the template we use to deploy the lab environment. Therefore, you can simply use the parsers available under Functions > Workspace functions:


 


VimProcessCreateLinuxSysmon
| limit 10

 


 


linux-sysmon-azure-sentinel.png


 


That’s it! You are now ready to use Sysmon for Linux in a lab environment for research and development of detections 😉  


 


Sysmon for Linux Resources 



 


References 


Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.