This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
This post is co-authored by Abhishek Saharn (Software Engineer, Microsoft)
NLog is a popular logging framework used by developers to log messages from .NET applications. One of the challenges of logging is how to store and analyze the large volume of data generated by the logs. Azure Data Explorer (ADX) is a fast and highly scalable data exploration service that can be used to store and analyze large volumes of data. In this blog post, we will show you how to use the Azure Data Explorer custom target connector with NLog to store your log data in an ADX cluster.
But first, Why logging is important?
Logging is a crucial component in software development, providing a record of events that occur during the operation of an application. This record can be used for a variety of purposes, including debugging, auditing, performance monitoring, security, troubleshooting, analytics, historical analysis, and continuous improvement.
When developers encounter errors or bugs in their code, logging can help identify the root cause by providing a detailed history of the events leading up to the issue. For compliance and auditing purposes, logging can track user activity and system events, providing an essential record of all transactions and changes. Additionally, provides developers with the opportunity for continuous improvement by identifying areas for optimization and refinement.
How Azure Data Explorer helps here?
One of the biggest challenges with logging is the sheer volume of data that is generated. For example, a single server in a typical enterprise environment can generate more than 10 GB of log data per day. Multiply that by the number of servers and applications in an organization, and you're looking at a truly staggering amount of data to manage and analyze.
To address this challenge, Azure Data Explorer(ADX) comes to help. ADX is a fast and highly scalable data analytics platform that is specifically designed for ingesting, analyzing, and visualizing large volumes of data, including log data.
By leveraging Azure Data Explorer's powerful query capabilities and built-in data visualization tools, organizations can quickly and easily analyze log data to gain insights into application performance, troubleshoot issues, and identify trends. It is feature-rich in time series analytics, log analytics, full-text search, advanced analytics (e.g., pattern recognition, forecasting, anomaly detection), visualization, scheduling, orchestration, automation, and many more native capabilities.
What is Azure Data Explorer NLog Connector?
The Azure Data Explorer connector is a custom target for NLog that allows you to send your log messages to an ADX cluster. The plugin is built on top of the Azure-Kusto-Data library and provides a simple and efficient way to sink your logs to ADX.
Now we will discuss how to get started with configuring NLog and Azure Data Explorer, by using the NLog-kusto sample application provided in the ADX-Nlog repository. You will learn how to write log messages and view them in Azure Data Explorer(ADX) cluster.
Prerequisites
- An ADX cluster and database. If you do not have one, Create Azure Data Explorer Cluster and DB
- An Active Directory Application with Database Ingestor role. If you do not have one, Create Azure Active Directory App Registration and grant it permissions to DB (Save the application ID and key for later).
- Create a table in Azure Data Explorer to store logs. The following command can be used to create a table with the name "ADXNLogSample".
- Clone the NLog-ADX Target git repo
- Set the following environment variables in the sample application:
- INGEST_ENDPOINT : Ingest URL of ADX cluster created. Eg: https://ingest-<clustername>.<region>.kusto.windows.net
- DATABASE : The name of the database to which data should be ingested into.
- APP_ID : Application Client ID required for authentication.
- APP_KEY : Application key required for authentication.
- AZURE_TENANT_ID : Tenant Id of the Azure Active Directory.
Step 1: Install the ADX Target for NLog
The ADX Target for NLog is available as a NuGet package. To install it, open the Package Manager Console and enter the following command:
Step 1.1: Configure the ADX Target for NLog
Alternatively, If you do not want to use environment variables, you can configure the NLog.config file. Open the configuration file and replace the existing target configuration one with the following:
Step 2: Build the application and run it
- Open a Powershell window, navigate to NLog ADX Target base folder and run the following command
- Once build got completed, Navigate to src/Nlog.Azure.Kusto.Samples/ run the following command to run the sample application
- The Program.cs contains predefined logs which will start getting ingested to ADX.
- The ingested log data can be verified by querying the created log table(ADXNLogSample in our case) by using the following KQL command.
Conclusion
In this blog post, we have seen how NLog and Azure Data Explorer can work together to provide a powerful and scalable solution for logging and analyzing .NET applications. By using the Azure Data Explorer custom target connector with NLog, you can easily send your log messages to an ADX cluster and take advantage of its fast and flexible query capabilities and data visualization tools. You can also leverage ADX’s advanced analytics features such as time series analysis, anomaly detection, forecasting, and pattern recognition to gain deeper insights into your application performance and behavior.
If you are interested in trying out this solution, you can follow the steps mentioned this post to set up your own ADX cluster and database, create a table to store your logs, install the ADX target for NLog, and run the sample application. You can also explore the ADX-Nlog repository on GitHub for more details and examples.
We hope you find this solution useful, and we welcome your feedback and suggestions.