This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
Introduction
In today's data-driven world, it's essential to collect, analyze, and gain insights from various logs and data sources. Azure Data Explorer (aka Kusto) is a cloud-based data analytics platform designed for analyzing and visualizing large volumes of data in real time and is particularly well-suited for log time series and telemetry data. Its real-time capabilities make it a valuable resource for organizations looking to gain insights from their data such as forecasting, anomaly detection, and prediction among various other capabilities. More on leveraging the power of Kusto can be found here.
Splunk Universal Forwarder, commonly referred to as "Splunk UF," is a lightweight version of the Splunk Enterprise software. It is specifically designed for the sole purpose of collecting and forwarding log data and machine data from various sources to a central Splunk Enterprise server or a Splunk Cloud deployment. Splunk Universal Forwarder serves as an agent that simplifies the process of data collection and forwarding, making it an essential component in a Splunk deployment. More details on Splunk Universal Forwarder can be found here
In this article, we will talk about how we forward logs using Splunk Universal Forwarder to Azure Data Explorer (Kusto) using Kusto Splunk Univeral Forwarder Connector.
System Architecture
To forward logs or Splunk logs to Kusto, we need the Splunk Universal Forwarder to be installed in the same machine where the logs are located. The Splunk Universal Forwarder will be configured to send messages via TCP 9997 port (default port) to another system that will be hosting the Kusto Splunk Universal Forwarder Connector. This connector will be listening to the messages arriving on the TCP 9997 port and will forward the logs to Kusto. The following diagram represents the setup.
Prerequisites
Before we begin, make sure you have the following prerequisites in place:
-
Splunk Universal Forwarder: Install Splunk Universal Forwarder on the machine where your logs originate. For the Installation guide please refer here.
-
Azure Data Explorer (Kusto): Set up an Azure Data Explorer cluster and database to receive the logs. More on this can be found here.
-
Azure Active Directory Application: Create an Azure AD application and service principal to enable communication between Splunk and Kusto.
- Docker: Docker needs to be installed in the system which will be required to configure and run the Kusto Splunk Universal Forwarder connector.
Step 1: Configure Splunk Universal Forwarder
- While setting up Splunk Universal Forwarder the receiving indexer needs to be configured to point to the system hosting Kusto Splunk Universal Forwarder Connector. Destination Indexer can be left blank. For setting up the Windows Universal Forwarder please refer to the following screenshot. Further details on the receiving indexer configuration can be found here.
- Configure inputs.conf: Navigate to the folder where Splunk Universal Forwarder is installed and then to /etc/system/local/ folder. Inputs.conf needs to be modified/created to enable the universal forwarder to read logs from. Example configuration can be found below. more on configuration of Inputs.conf can be found here.
[default] index = default disabled = false [monitor://C:\Program Files\Splunk\var\log\splunk\modinput_eventgen.log*] sourcetype = modinput_eventgen - Configure outputs.conf: Navigate to the folder where Splunk Universal Forwarder is installed and then to /etc/system/local/ folder. Outputs.conf file needs to be created/modified to enable universal forwarder to determine the location to forward the logs to, in our case the hostname and port of system hosting Kusto Splunk Universal Forwarder connector. An example configuration can be found below. More details on the outputs.conf can be found here
[tcpout] defaultGroup = default-autolb-group sendCookedData=false [tcpout:default-autolb-group] server = 127.0.0.1:9997 [tcpout-server://127.0.0.1:9997] - Restart the Splunk Universal Forwarder after the changes.
Step 2: Configure Kusto Splunk Universal Forwarder Connector
- Download/Clone the connector from the github repo.
- Navigate to the base directory of the connector.
cd .\SplunkADXForwarder\ - Edit the config.yml by entering the required kusto credentials and database to enable the connector to send the data to kusto.
ingest_url : <ingest URL> client_id : <azure-client-id> client_secret : <azure-client-secret> authority : <authority> database_name : <kusto-database-name> table_name : <kusto-table-name> table_mapping_name : data_format : csv - Docker build
docker build -t splunk-forwarder-listener - Docker run
docker run -p 9997:9997 splunk-forwarder-listener
Step 3: Verify Data In Azure Data Explorer (Kusto)
-
Once the docker starts running, data is sent to your Azure Data Explorer table. You can verify that the data is ingested by running a query in the web UI query editor.
Conclusion
In conclusion, we have demonstrated how to ingest data using Splunk Universal Forwarder to route data to Azure Data Explorer, a powerful cloud-based data analytics platform. By using the Kusto Splunk Universal Forwarder Connector, we can easily forward logs from various sources to Kusto and perform real-time analysis and visualization. This enables us to gain insights from our data and leverage the capabilities of Kusto for various scenarios such as forecasting, anomaly detection, and prediction. We hope this article helps you get started with Splunk and Kusto integration and explore the benefits of both technologies.
Important
This connector can be used in Real-Time Analytics in Microsoft Fabric. Use the instructions in this article with the following exceptions:
- If required, create databases using the instructions in Create a KQL database.
- If required, create tables using the instructions in Create an empty table.
- Get query or ingestion URIs using the instructions in Copy URI.
- Run queries in a KQL queryset.