Outlook and Office365 Intermittent Slowness and Disconnects

Incident Report for Dickinson

Postmortem

Title: Some users experienced impact to multiple Microsoft 365 services and features

User impact: Users experienced impact to multiple Microsoft 365 services and features.

More info:
Impacted services and features include, but were not limited to:
Exchange Online
- Users were unable to access Outlook on the web (as communicated under EX571516)

Microsoft Teams
- Users may have had difficulties scheduling meetings and/or live events
- Users may have had trouble loading people profile cards
- Users may have had issues loading file lists
- Users may have been unable to create new teams & channels
- Users may have been unable to install apps
- Users may have experienced issues performing searches
- Users may have seen delays in admin policy changes taking effect
- Users might have seen errors when using messaging extensions
- Users may have been unable to view up to date Presence information
- Teams Graph APIs may have been impacted
- Assignments tab in Teams may have not loaded
- Teams Admin Center functionalities were not performing as expected

SharePoint Online and OneDrive for Business
- Users may have been unable to use Search functionality Microsoft Power Automate in Microsoft 365
- Users were unable to trigger flows using Office 365 connectors in Microsoft Power Automate Power Apps in Microsoft 365
- Users were unable to run apps in Power Apps that utilize Office 365 connectors
- Flows using the Office 365 Groups, Groups Mail, Outlook, Users, and Videos connectors may have been impacted

Final status: We've completed the restarts to address the residual impact, and after an extended period of monitoring, we've confirmed that the issue is resolved. We continue to investigate why the update caused impact and further details will be provided within the Post-Incident Report.

Scope of impact: Impact was specific to users routed through the affected infrastructure.
Start time: Monday, June 5, 2023, 10:10 AM EDT
End time: Monday, June 5, 2023, 1:13 PM EDT

Root cause: A recent update, intended to add additional resilliencies against downstream request remaining in a hanging state, inadvertently caused impact.

Posted Jun 05, 2023 - 14:04 EDT

Resolved

From Microsoft: Telemetry indicates that service availability is at approximately 99.9% across the service. We're continuing to isolate a small number of components which remain in an unhealthy state and are performing targeted restarts to remediate the residual impact.

Posted Jun 05, 2023 - 13:46 EDT

Monitoring

From Microsoft: We've reverted the update and telemetry shows service improvement. We're continuing to monitor the service to ensure recovery and performing actions to address any residual impact.

Posted Jun 05, 2023 - 12:39 EDT

Update

Update from Microsoft: We've prevented the potentially problematic update from propagating further across the service and are reviewing options to revert the change quickly and safely in those portions of our infrastructure where it has been applied.

Posted Jun 05, 2023 - 12:07 EDT

Identified

Update Provided from Microsoft:

Title: Some users are unable to access Outlook on the web.

User impact: Users are unable to access Outlook on the web.

More info: Additionally, other Microsoft 365 services are also experiencing impact:
Microsoft Teams
- Users may have difficulties scheduling meetings and/or live events
- Users may have trouble loading people profile cards
- Users may have issues loading file lists
- Users may not be able to create new teams & channels
- Users may not be able to install apps
- Users may have trouble issuing searches
- Users may see delays in admin policy changes taking effect
- Users might see errors when using messaging extensions.
- Teams Graph APIs may be impacted

Current status: We're continuing to review monitoring health systems targeted at networking, Client Access Front End (CAFE) routing, and database storage to determine the underlying cause of the issue.

We've identified an update which was applied around the time that impact started. We're reverting the update to potentially mitigate impact whilst we continue to investigate the underlying cause of the issue.

Posted Jun 05, 2023 - 11:36 EDT

Update

Microsoft is continuing to investigate this issue by "reviewing their networking systems and recent updates in an effort to identify the underlying root cause of the issue." Outlook on the Web and Outlook Desktop Application on both Windows and Mac computers are currently affected.

Posted Jun 05, 2023 - 11:22 EDT

Investigating

Microsoft is currently investigating an issue causing Outlook and Office365 system intermittent slowness and disconnects. We will provide updates here as we receive them. Thank you for your patience.

Posted Jun 05, 2023 - 10:58 EDT

This incident affected: Office 365 / Microsoft 365 (Email (Outlook), SharePoint).