How to leverage Azure Monitor to meet functional/non-functional requirements-No.3 Compute/Inside OS

Posted by

This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .

This article is a part of series articles for Azure Monitor. Please refer to How to leverage Azure Monitor to meet functional and non-functional requirements - No.1 overview first before reading this post. This post dives deeply for Compute/Inside category among monitoring categories as highlighted blue. 

Article No

monitoring category

monitoring target

Note

2

Compute

Reboot

monitor reboot frequency

 

 

CPU

monitor CPU usage

 

 

Memory

monitor memory usage

3

Compute/Inside OS

Log file

monitor event log and syslog

 

 

Process

monitor available process

4

Storage/Disk

Disk

monitor disk usage

 

 

folder/file

monitor folder usage and file size

5

Endpoint/IPv4 address

response/service

monitor specific address and port

 

Web site

Scenario

monitor web scenario

6

Network

Connectivity

monitor vNiC and VNET peering

 

 

Firewall

monitor Azure Firewall rule usage

7

Backup

Backup

monitor backup status

 

Azure Resources

Resource health

monitor resource availability

 

3.1.1 Log file monitoring: EventLog and Syslog

Configure eventlog of Windows Server or Syslog of Linux on Legacy agent management menu on your Log Analytics workspace. A screenshot below shows Linux Syslog configuration and added facilities.

Please note the Log Analytics agents won’t be supported as of August 31, 2024. Plan to migrate to Azure Monitor Agent prior to this date.

daisami_0-1661839974971.png

Syslog data will be stored to LogManagement on Log Analytics workspace after a short wait. Log data on /var/log/rsyslog will be stored to SyslogMessage column as each line if you configure syslog facility.

daisami_1-1661840615677.png

Here is Kusto to retrieve logs which SyslogMessage value contains "journal" string.

daisami_0-1661844070486.png

 

Syslog
| where Facility == "syslog"
| where SyslogMessage contains "journal"
| order by TimeGenerated

 

In this way, you can extract specific logs from Syslog table which stores /var/log/rsyslog data. Create a script which executes logger -p local0.critical "Critical message hoge" and configure crontab for the script to execute periodically, then we can find specific messages among stored logs and call an alert rule. We can also disable the alert rule and pause it in this case.

 

3.1.2 Log file monitoring: custom log

There are various types of logs which are generated by variety of applications. Azure Monitor allows you to collect such logs with Custom Logs feature. Refer to Collect text logs with the Log Analytics agent in Azure Monitor for the detail.

In this post, we will setup Custom Logs for a Java application using logback. Here is part of logback-spring.xml for this use case. This Java application will generate a log file on /var/log/demo2/applog.log

 

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <springProperty name="LOG_DIR_PATH" source="log.dir.path" defaultValue="/var/log/demo2/"></springProperty>
    <appender name="ROLLING" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${LOG_DIR_PATH}/applog.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>${LOG_DIR}/demo2-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <charset>UTF-8</charset>
            <pattern>%date %-4relative [%thread] %-5level %logger{35} - %msg %n</pattern>
        </encoder>
    </appender>

 

Setup Custom Logs configuration from Custom log menu on your Log Analytics workspace as follows.  

daisami_0-1661851523609.png

Run your Java application and let logback generates its log, then you can retrieve your logs with Kusto query.

daisami_1-1661853031202.png

RawData column stores full text of collected entry and you can parse the text data by leveraging Kusto query if you need. Refer to Parse text data in Azure Monitor logs for the detail.

Note: The log must have specific format for each timestamp to use Azure Monitor. Refer to Collect text logs with the Log Analytics agent in Azure Monitor for the detail.

 

Finally, here is check result of Log file monitoring. 

Type

category

Goal and outcome

Result

1

monitoring

Azure Monitor can satisfy functional requirements

OK

2

 

Azure Monitor can setup short granularity for detections

1 min

3

 

Azure Monitor can setup thresholds detections

OK

4

 

Azure Monitor can setup retry detections

OK

5

 

Azure Monitor can suspend and resume for checking threshold

OK

6

 

Azure Monitor can send a mail for detection results

OK 

7

statistics

Azure Monitor can retrieve workspace logs with specific duration

OK

8

 

Azure Monitor can visualize statistic data

OK

9

automation

Azure Monitor can have primary action based on alert rules

OK

10

 

Azure Monitor can send validation results

OK

 

3.2.1 Process monitoring: running processes inside OS

Open Legacy agent management menu on your Log Analytics workspace. A screenshot below shows Linux Syslog configuration and added local0 facility. This is an example for local0, thus add other facilities if you need.

daisami_0-1661853579481.png

Create and setup simple script which output logs when the process number is less than expected number. Enable crontab to periodically run the script. As an example, this script outputs "apache2 process down! please check" message to /var/log/syslog when httpd process number is less than 3.

 

#!/bin/sh
HTTPCOUNT=`ps -ef | grep -i httpd | grep -vc grep`
if [ ${HTTPCOUNT} -lt 3 ]
then
	logger -p local0.crit "apache2 process down! please check"
fi

 

We can see that the syslog data is stored on Log Analytics workspace. This message says that httpd processes is less than 3, thus we will configure an alert rule to send an email notification when this line is output. We can also disable the alert rule and pause it in this case.

 

Syslog
| where Computer == "gcp-instance-01"
| where Facility == "local0"
| where SyslogMessage contains "apache2"

 

daisami_0-1661854511862.png

Optional - 3.2.2 Process monitoring: Leverage VM Insights

We can identify with VM Insights tables that a process is running even though we can't check the number of the process. This example checks whether httpd process runs or not with VMProcess and InsightsMetrics tables. Log data is sent to Log Analytics workspace per minute.

 

let ApacheProcessId = toscalar (
    VMProcess
    | where Computer startswith "gcp-instance-01"
    | where ExecutablePath contains "httpd"
    | top 1 by TimeGenerated
    | project Process
);
InsightsMetrics
| where Name == "Heartbeat" and Origin == "vm.azm.ms/map"
| where Computer startswith "gcp-instance-01"
| where isnotempty(ApacheProcessId) and Tags contains ApacheProcessId
| where TimeGenerated > ago(5m)

 

daisami_1-1661855270227.png

Optional - 3.2.3 Process monitoring: Port monitoring with VM Insights

VMBoundPort table allows us to check status of a port. 

daisami_2-1661855386235.png

We can check whether the port is open or not, but we can't identify its hang up status. We can leverage Connection Monitor of Network Watcher in the case. Refer to Create a monitor in Connection Monitor by using the Azure portal for the detail.

 

The VM Insights Map allows you to check status of each process and its ports, and outbound network communication. This feature can adopt to outside Microsoft Azure for example this screenshot is a VM on GCP.

daisami_0-1661929003587.png

However, note that main purpose of this feature is not for sending notification when a specific process is down, but visualize outbound network communication from the VM. Overview of VM insights 

 

Finally, here is check result of Process monitoring. 

Type

category

Goal and outcome

Result

1

monitoring

Azure Monitor can satisfy functional requirements

OK*1

2

 

Azure Monitor can setup short granularity for detections

1 min

3

 

Azure Monitor can setup thresholds detections

OK

4

 

Azure Monitor can setup retry detections

OK

5

 

Azure Monitor can suspend and resume for checking threshold

OK

6

 

Azure Monitor can send a mail for detection results

OK 

7

statistics

Azure Monitor can retrieve workspace logs with specific duration

OK

8

 

Azure Monitor can visualize statistic data

OK

9

automation

Azure Monitor can have primary action based on alert rules

OK

10

 

Azure Monitor can send validation results

OK

Spoiler
*1 Process monitoring is sometimes required to check a process per seconds. You might be needed to consider other solution including 3rd party tools.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.