Experienced Latency and Data Gaps in East US2 region – 10/24 – Resolved

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Final Update: Saturday, 24 October 2020 14:43 UTC

We've confirmed that all systems are back to normal with no customer impact as of 10/24, 14:15 UTC. Our logs show the incident started on 10/23, 19:01 UTC and that during the 19 hour 14 minutes that it took to resolve the issue some customers may have experienced data latency and data gaps in East US2 region.
  • Root Cause: The failure was due to bad configuration in our dependent services.
  • Incident Timeline: 19 Hours - 10/23, 19:01 UTC through 10/24, 14:15 UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.

-Sandeep

Update: Saturday, 24 October 2020 08:04 UTC

Root cause has been isolated to a backend infrastructure component that was accidentally deleted. We have recovered the component and waiting for complete propagation of the restored configuration. Customers may continue to experience data gaps and latency in East US2 region.
  • Work Around: None
  • Next Update: Before 10/24 20:30 UTC
-Sandeep

Update: Saturday, 24 October 2020 04:24 UTC

Root cause has been isolated to a backend infrastructure component that was accidentally deleted. We have recovered the component and waiting for complete propagation of the restored configuration. 
  • Work Around: None
  • Next Update: Before 10/24 08:30 UTC
-Sandeep

Update: Saturday, 24 October 2020 03:18 UTC

Root cause has been isolated to a backend infrastructure component that was accidentally deleted.  We have recovered the component and waiting for complete propagation of the restored configuration.  

  • Work Around:
  • Next Update: Before 10/24 06:30 UTC
-Chandar

Update: Saturday, 24 October 2020 02:55 UTC

We continue to investigate issues that impacts Workspace-enabled Application Insights resource only. Root cause is not fully understood at this time. Customers might experience data gaps and latency. We are working to establish the start time for the issue, initial findings indicate that the problem began at <10/23 ~19:05 UTC>.
  • Work Around:
  • Next Update: Before 10/24 05:00 UTC
-Chandar

Initial Update: Saturday, 24 October 2020 02:51 UTC

We are aware of issues within Application Insights and are actively investigating. Some customers may experience Latency and Data Loss.
  • Work Around:
  • Next Update: Before 10/24 05:00 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Chandar

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.