Linux Ubuntu VMs in non-responsive state for Service Map Agent- 02/25 – Resolved

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Final Update: Tuesday, 25 February 2020 05:09 UTC
We've confirmed that all systems are back to normal with no customer impact as of 02/25, 03:32 UTC. Our logs show the incident started on 02/20, 17:00 UTC and that during the 4 days 10 hours and 32 minutes that it took to resolve the issue customers using Service Map may have experienced issues causing Virtual Machines with Dependency Agent (Service Map) installed to be non-responsive resulting from CPUs running at 100% when Virtual Machines are auto-updated with the latest release of Azure Images for Ubuntu versions 16.04 and 18.04.
 
The updated VM Extension has been deployed to all regions and can safely be re-enabled.
  • If you removed the VM extension, as recommended earlier, you can add it back either manually or by visiting the Azure Monitor for VMs Get Started page. On this page you can identify the Virtual Machines that are missing the Dependency Agent extension and you can install the extension on them with a few clicks.  Note that VM Scale Sets are currently not supported on the Get Started page. You can re-enable the extension for VMSS by going to the Insights menu item from the respective VMSS blade.
  • If you still have Linux Virtual Machines or VM scale sets that are running the Dependency Agent extension, you will need to take an action to upgrade them to the latest extension version. You can do this manually by removing and adding the extension, or you can use the help scripts we have put together:
  • For a VM download the PowerShell script from here. To update all Linux VMs in a given subscription execute in Windows PowerShell: .\Manage-Extension-VM.ps1 -subscriptionId  <subId> -TargetOs Linux -Update
  •  For VM scale sets download the script from here. To update a given VMSS execute: .\Upgrade-VMSS-Extension.ps1 -SubscriptionId <subId> -ResourceGroupName <rg> -Name <vmssName> -Update\
  • Note: These scripts may take significant amount of time to execute depending on the number of resources in your subscription/scale set.
  • If you are managing the dependency agent installation manually without using a VM extension, follow these steps:
                    1. Download the latest Dependency Agent installer for Linux from here.
                    2. Check the integrity of the installer
                               sha256sum dependencyagentlinux
                               b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux
                   3. Check the version of the installer
                                sh dependencyagentlinux --version
                                Should report: Dependency Agent installer, version 9.10.2.9060      
                    4. Execute the installer
                        a. For an interactive installation use: sudo sh dependencyagentlinux
                        b. For silent/unattended installation use: sudo sh dependencyagentlinux -s
 
  • Incident Timeline: 4 Days 10 Hours & 32 minutes - 02/20, 17:00 UTC through 02/25, 03:32 UTC
We understand that customers rely on Service Map as a critical service and apologize for any impact this incident caused.

-Mohini
 
Update: Monday, 24 February 2020 18:31 UTC
 
An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:

       - If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
       - If you have already uninstalled the extension, there are no actions at this time
       - If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.

            1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux        
            Execute the following in a terminal window:
            wget https://aka.ms/dependencyagentlinux

            2. Check the integrity of the installer    
            sha256sum dependencyagentlinux
            b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

            3. Check the version of the installer
            sh dependencyagentlinux --version
            Should report: Dependency Agent installer, version 9.10.2.9060

            4. Execute the installer
                a. For an interactive installation use
                    sudo sh dependencyagentlinux

                b. For silent/unattended installation use
                   sudo sh dependencyagentlinux -s

If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.

  • Next Update: Before 02/25 07:00 UTC
-Leela

Update: Sunday, 23 February 2020 19:09 UTC
 
An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:

       - If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
       - If you have already uninstalled the extension, there are no actions at this time
       - If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.

            1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux        
            Execute the following in a terminal window:
            wget https://aka.ms/dependencyagentlinux

            2. Check the integrity of the installer    
            sha256sum dependencyagentlinux
            b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

            3. Check the version of the installer
            sh dependencyagentlinux --version
            Should report: Dependency Agent installer, version 9.10.2.9060

            4. Execute the installer
                a. For an interactive installation use
                    sudo sh dependencyagentlinux

                b. For silent/unattended installation use
                   sudo sh dependencyagentlinux -s

If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.

  • Next Update: Before 02/24 19:30 UTC
-Leela

Update: Saturday, 22 February 2020 21:00 UTC

An updated version of Dependency Agent for Linux that addresses the instability issues with the latest Ubuntu Azure Kernels is now available (9.10.2.9060).The team is currently working on updating the VM extension in all Azure regions. This process will take several days. While the updated extension is being deployed, our recommendations are as follows:

       - If you are still using the Dependency Agent for Linux VM extension, please uninstall the extension from any vulnerable systems to avoid unexpected downtime if your systems update to the latest Ubuntu Azure Kernels before they obtain the updated VM extension.
       - If you have already uninstalled the extension, there are no actions at this time
       - If you are not using the extension and have installed the Dependency Agent directly on your systems, follow the instructions below to upgrade them to the latest released version.

            1. Download the latest Dependency Agent installer for Linux from: https://aka.ms/dependencyagentlinux        
            Execute the following in a terminal window:
            wget https://aka.ms/dependencyagentlinux

            2. Check the integrity of the installer    
            sha256sum dependencyagentlinux
            b7725b6b205cf8c336d9aad87956336c816412740e9d6499bcacb6f862ae3896  dependencyagentlinux

            3. Check the version of the installer
            sh dependencyagentlinux --version
            Should report: Dependency Agent installer, version 9.10.2.9060

            4. Execute the installer
                a. For an interactive installation use
                    sudo sh dependencyagentlinux

                b. For silent/unattended installation use
                   sudo sh dependencyagentlinux -s

If the Dependency agent fails to start, check the logs for detailed error information. The log directory is /var/opt/microsoft/dependency-agent/log.

  • Next Update: Before 02/23 09:00 UTC
-Leela

Update: Saturday, 22 February 2020 17:00 UTC
We continue to investigate the issue. We currently have no estimate for resolution. Please continue using the same work around stated below.
  • Work Around

    From VM:
    Log into VM, if unable to login, restart the VM and login.
    Run the following command as ROOT:
    dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
    Restart the VM
    From Azure Portal:
    Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
    Restart the VM.
    Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
    Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
  • Next Update: Before 02/23 05:00 UTC
-Leela

Update: Saturday, 22 February 2020 00:32 UTC

We continue to investigate the issue. We currently have no estimate for resolution. Please continue using the same work around stated below.
  • Work Around

    From VM:
    Log into VM, if unable to login, restart the VM and login.
    Run the following command as ROOT:
    dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
    Restart the VM
    From Azure Portal:
    Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
    Restart the VM.
    Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
    Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
  • Next Update: Before 02/22 07:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Leela

Update: Friday, 21 February 2020 10:04 UTC
 
Starting at 17:00 UTC on 20 Feb 2020, customers using Service Map may experience issues causing Virtual Machines with Dependency Agent (Service Map) installed to be non-responsive resulting from CPUs running at 100% when Virtual Machines are auto-updated with the latest release of Azure Images for Ubuntu versions 16.04 and 18.04.

Current Workarounds:

From VM:
Log into VM, if unable to login, restart the VM and login.
Run the following command as ROOT:
dpkg --purge microsoft-dependency-agent-dkms dependency-agent dependency-agent-service
Restart the VM
From Azure Portal:
Uninstall dependency agent extension (Microsoft.Azure.Monitoring.DependencyAgent.DependencyAgentLinux)
Restart the VM.
Should you encounter issues following these instructions or require further assistance, please raise a support ticket.
Engineers are actively exploring mitigation options. The next update will be provided upon mitigation, or as events warrant.
-Jayadev

Initial Update: Friday, 21 February 2020 08:56 UTC

We are aware of issues within Service Map and are actively investigating. Some customers may experience issues with High CPU or non responsive behavior of VM's  which are on boarded to Service Map in multiple regions.
  • Work Around: None 
  • Next Update: Before 02/21 13:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anusha

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.