Afternoon updates from Microsoft on this one:
April 20, 2023 2:38 PM · Quick update
We've begun the process of adding further processing throughput within the affected environment and we've confirmed that it is approximately 50 percent complete. We expect this process to complete in full in the next hour and we'll continue to monitor to identify whether it provides relief. This quick update is designed to give the latest information on this issue.
April 20, 2023 1:58 PM
Title: Some users may be unable to view or access Microsoft 365 apps or services User impact: Users may be unable to view or access Microsoft 365 apps or services. More info: Impacted services may include, but are not limited to: - Microsoft 365 web apps - Users may be unable to access Microsoft 365 web apps, such as Excel for the web. Additionally, the search bar may not appear in any Office for the web service. - Microsoft Teams - Admins may be unable to access the Microsoft Teams admin center. - SharePoint Online - Users may be unable to view the settings gear, search bar and waffle. - Microsoft Planner - Users may be unable to access Microsoft 365 web apps through Microsoft Planner. - Microsoft Yammer - The search bar is missing from the User Interface. - Outlook on the web - Users may experience slowness or latency when accessing or using the service. - Project for the web - Users may be unable to view the waffle menu, settings, and help content. - Microsoft Power Apps - Users may be unable to access Power Apps from the Office.com app launcher. While the Microsoft 365 apps may not render, users can still access the applications directly through the URL. Some examples of these include: Microsoft 365 admin center - admin.microsoft.com Microsoft Teams - teams.microsoft.com Microsoft Power Apps - make.powerapps.com Outlook for the web - outlook.office.com Word for the web - microsoft365.com/launch/word Excel for the web - microsoft365.com/launch/excel Current status: We’ve completed the full reversion of the service update that occurred on Monday, April 17, 2023, and we’re monitoring the environment to see if this resolves the issue. Additionally, we continue the preparations for adding further processing throughput within the environment in an attempt to provide relief. Scope of impact: Impact is specific to some users who are served through the affected infrastructure. The majority of reported impact is coming from users located in North and South America.
April 20, 2023 1:23 PM · Quick update
In addition to the ongoing update reversion and root cause investigation, we're also in the process of adding further processing throughput within the environment in an attempt to provide relief. This quick update is designed to give the latest information on this issue.
April 20, 2023 12:53 PM · Quick update
After reverting some of the specific configurations, we have not seen significant improvements in CPU utilization levels. We're performing a rollback of the entire update, which occurred on Monday, April 17, 2023, to rule it out of our investigation. We expect this action to complete within the next few hours. In parallel, we continue to analyze diagnostic logs and telemetry to isolate the source of the high CPU utilization. This quick update is designed to give the latest information on this issue.
April 20, 2023 12:09 PM
Title: Some users may be unable to view or access Microsoft 365 apps or services User impact: Users may be unable to view or access Microsoft 365 apps or services. More info: Impacted services may include, but are not limited to: - Microsoft 365 Online apps - Users may be unable to access Microsoft 365 web apps, such as Excel Online. Additionally, the search bar may not appear in any Office Online service. - Microsoft Teams - Admins may be unable to access the Microsoft Teams admin center. - SharePoint Online - Users may be unable to view the settings gear, search bar and waffle. - Microsoft Planner - Users may be unable to access Microsoft 365 web apps through Microsoft Planner. - Yammer - The search bar is missing from the User Interface. - Outlook on the web - Users may experience slowness or latency when accessing or using the service. - Microsoft Project for the Web - Users may be unable to view the waffle menu, settings, and help content. Whilst the Microsoft 365 apps may not render, users can still access the applications directly through the URL. Some examples of these include: Microsoft 365 Admin Center - admin.microsoft.com Outlook - outlook.office.com Microsoft Teams - teams.microsoft.com Word Online - microsoft365.com/launch/word Excel Online - microsoft365.com/launch/excel Current status: We're focusing our investigation on high CPU utilization that is occurring on the components which facilitate the back-end API calls for the navigation headers and features. We’re changing some specific configurations which were part of a service update that occurred on Monday, April 17, 2023, to see if this provides relief whilst we continue to investigate the source of the high CPU utilization. Scope of impact: Impact is specific to some users who are served through the affected infrastructure. The majority of reported impact is coming from users located in North and South America. Start time: Wednesday, April 19, 2023, 8:38 PM (4/20/2023, 12:38 AM UTC) Preliminary root cause: A section of caching infrastructure is performing below acceptable performance thresholds, causing some service calls to bypass the cache and go directly to Azure Active Directory infrastructure resulting in high resource utilization and subsequent impact.