Experiencing Webtest activation Notificaiton failure issue in Azure Portal – 06/06 – Resolved

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Final Update: Thursday, 06 June 2019 18:41 UTC

We've confirmed that all systems are back to normal with no customer impact as of June 6 2019, 18:32 UTC. Our logs show the incident started on June 6,2019 08:05 UTC and during impact timeline that it took to resolve the issue 100% of customers who subscribed to Webtest Alert Activation notification would have missed receiving their activation notification emails.
  • Root Cause: The failure was due to downstream notification service.
  • Lessons Learned: We are looking to the telemetry for improvements on our ability to detect such issues sooner.
  • Incident Timeline: 8 Hours & 27 minutes - June 6,2019 08:05 UTC through June 6 2019, 18:32 UTC

We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.


-chandar

Update: Thursday, 06 June 2019 17:42 UTC

Root cause has been isolated to a template in one of the downstream notification service which was impacting Webtest alert mail activated notifications. We are deploying a hotfix to address this issue .  Gen2 alerts are working as expected.  We estimate 2 hours before all impacts are addressed.
  • Work Around: None
  • Next Update: Before 06/06 20:00 UTC
-chandar

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.