Data Latency and Data Loss issue in App Insights ingestion (many regions) – 02/21 – Resolved

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Final Update: Saturday, 22 February 2020 10:59 UTC

We've confirmed that all systems are back to normal with no customer impact as of 2/22, 10:50 UTC. Our logs show the incident started on 2/21, 14:50 UTC and that during the 20 hours that it took to resolve the issue some customer may have experienced intermittent data latency, data gaps and incorrect alert activation.
  • Root Cause: The failure was due to issue with one of our backend services.
  • Incident Timeline: 20 Hours - 2/21, 14:50 UTC through 2/22, 10:50 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Rama

Update: Saturday, 22 February 2020 05:19 UTC

We continue to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience intermittent data latency, data gaps and incorrect alert activation. We are working to establish the start time for the issue, initial findings indicate that the problem began at 02/21 ~14:50 UTC. We currently have no estimate for resolution.
  • Work Around: None
  • Next Update: Before 02/22 11:30 UTC
-Rama

Update: Saturday, 22 February 2020 00:38 UTC

We still continue to investigate issues within Application Insights. Root cause is identified as there is an issue with the dependency service and they are working on it. Some customers may continue to experience intermittent data latency and data gaps and incorrect alert activation. We currently have no estimate for resolution.
  • Next Update: Before 02/22 05:00 UTC

We are working hard to resolve this issue and apologize for any inconvenience.

-Leela

Update: Friday, 21 February 2020 20:36 UTC

We continue to investigate issues within Application Insights. Root cause is not fully understood at this time. Some customers continue to experience intermittent data latency and data gaps and incorrect alert activation. We are working to establish the start time for the issue, initial findings indicate that the problem began at  02/21 ~14:53 UTC. We currently have no estimate for resolution.
  • Next Update: Before 02/22 01:00 UTC
-Leela

Initial Update: Friday, 21 February 2020 16:30 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers in West US2, West US, East US, South Central US may experience intermittent data latency and data gaps and incorrect alert activation.
  • Next Update: Before 02/21 19:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Leela

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.