Experiencing Alerting failure for Metric Alerts – 12/18 – Resolved

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Final Update: Wednesday, 18 December 2019 15:41 UTC

We've confirmed that all systems are back to normal with no customer impact as of 12/18, 15:15 UTC. Our logs show the incident started on 12/13, 13:50 UTC and that during the 5 days 1 hours & 25 minutes that it took to resolve the issue ~45 customers using Classic Alert Rules and Autoscale in US Sovereign Cloud may have been experiencing missing email notifications for alerts that were fired or autoscale events.
  • Root Cause: The failure was due to an issue with one of our dependent service.
  • Incident Timeline: 5 Days 1 Hours & 25 minutes - 12/13, 13:50 UTC through 12/18, 15:15 UTC
We understand that customers rely on Metric Alerts as a critical service and apologize for any impact this incident caused.

-Anmol

Update: Wednesday, 18 December 2019 13:30 UTC

We continue to investigate issues within Metric Alerts. Some customers in US Sovereign Cloud using Classic Alert Rules and Autoscale may continue to experience missing email notifications for alerts that were fired or autoscale events. We are working to establish the start time for the issue, initial findings indicate that the problem began at 12/13 10:00 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 12/18 17:30 UTC
-Anmol

Initial Update: Wednesday, 18 December 2019 10:45 UTC

We are aware of issues within Metric Alerts and are actively investigating. Some customers using Classic Alert Rules and Autoscale in US Sovereign Cloud may have been experiencing missing email notifications for alerts that were fired or autoscale events.
  • Work Around: None
  • Next Update: Before 12/18 13:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Anmol

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.