On Thursday 12 November 2026, Outlook, Teams, Copilot and a dozen other Microsoft 365 services failed worldwide for almost a day. It was not an attack and not a traffic spike. It was a bug shipped in July that sat quietly for three months until three unrelated conditions lined up. Microsoft’s preliminary post-incident report is unusually candid, and it is one of the best case studies in cascading failure we have had in years.

Read this first. Facts as of 17 November 2026, mainly from Microsoft’s preliminary post-incident report (incident EX1550543 / MO1550821, dated 15 November). The final report is due on 20 November and may change details. I tag claims as [Official], [Reported] and [Inferred].

The numbers

Metric Value
Start 12 Nov 2026, 16:20 UTC (11:20 AM ET) [Official]
End of customer impact 13 Nov 2026, 16:10 UTC, about 24 hours [Official]
Servers unresponsive at worst More than 80% of the network gateway fleet [Official]
Time for the defect to spread fleet-wide Less than two hours [Official]
Capacity already out before the incident Nearly 14%, from an unrelated issue [Official]
Age of the bug Deployed from July 2026, over three months without impact [Official]
Downdetector About 120,000–130,000 concurrent Outlook reports in the US [Reported]
Countries affected At least 74 [Reported]
Services listed Exchange, Copilot, Teams, SharePoint, OneDrive, Defender, Purview, Fabric, Intune, Planner, Power Apps, To Do, Power BI, Universal Print [Official]

What happened, technically

Many Microsoft 365 services sit behind a shared networking gateway, a front-end fleet built from thousands of “capacity units” grouped into regions. Microsoft runs two independent, heterogeneous gateway systems so services can fail over between them. [Official] Microsoft says three factors together took that gateway down.

Factor 1: a latent bug from July

A fix prepared specifically for the Microsoft 365 gateway servers introduced lock contention inside an internal Windows process. Under certain load, it exhausted CPU and memory until the machine stopped responding. It went through safe deployment from July and ran worldwide for more than three months without a problem. Load tests missed it because the contention gets worse on newer hardware with more cores, while the labs mostly used weaker machines with fewer cores and a different traffic mix. [Official]

Factor 2: a routine failover added load

About 15 hours earlier, at 01:07 UTC, automation moved Teams from the alternative gateway onto the primary one that also serves Exchange Online. That is normal resilience behaviour, but it raised load and changed traffic patterns. Isolated server failures began during the morning and were handled automatically. [Official]

Factor 3: capacity already missing

Nearly 14% of gateway capacity was out of service from an unrelated issue the day before. On its own that was fine: there was headroom. [Official]

The cascade

More load plus less capacity hit the bug during US peak hours. Requests built up and servers stopped responding. The traffic management system did what it always does: it moved users to healthy endpoints. That extra load triggered the same bug on the next servers, and the next. In under two hours more than 80% of the fleet was unresponsive. Because of a separate issue, the bulkheads between regions did not stop it, so a regional problem became global. [Official]

Services that could leave the faulty gateway, such as Teams, OneDrive, SharePoint and Office apps, failed out and recovered earlier; Teams chats and meetings mostly worked. Exchange Online could not leave the gateway “for scale reasons”, so mail stayed down. Everything that leans on Exchange (Copilot reading mail, Teams calendars, Defender scanning messages) degraded with it. [Official]

Why recovery took almost a day

  • Restarted servers were crushed again. Each server that came back was immediately flooded by waiting connections and froze again; at 23:55 UTC Microsoft confirmed that recovered servers went unresponsive “upon receiving traffic”. [Official]
  • Automation was not built for this state. Automatic recovery was not designed to bring a fleet back from broad unresponsiveness, and Microsoft eventually disabled some automated traffic redirection at 01:53 UTC on 13 November. [Official]
  • Rollback didn’t work for an old change. Because the defect was months old, normal rollback, flighting and feature flags could not quickly revert it. [Official]
  • The real culprit took time to find. Engineers first chased health checks and a network configuration change. Only around 00:30–01:00 UTC did they pin it on connection negotiation, where competing processes overwhelmed the process that sets up connections. [Official]
  • The fix. At 07:20 UTC they applied an update that returned the code path to the previously known good state, then rolled it out in large batches (50%+ of a region at a time), prioritising regions entering business hours. Europe was healthy by 11:35 UTC, webmail globally by 14:17 UTC. [Official]

We have seen this before

  • Azure WAN, 25 January 2023. During a planned capacity addition, a command to update a router IP address behaved differently across router vendors and triggered widespread routing recalculation across Microsoft’s global WAN. [Reported]
  • Azure Front Door, 29 October 2025. A faulty configuration change in the Front Door data plane made nodes across the global fleet fail to load, cascading into Microsoft 365, Outlook, Copilot, the Azure Portal, Xbox and customer sites such as Alaska Airlines and Starbucks. [Reported]
  • Industry-wide. One tracker of major AWS, Azure, Google Cloud and Cloudflare outages since 2020 finds that a new configuration, a deployment, or automation acting on bad input caused 55% of them. [Reported]
  • The same week. The FAA ground stop the next day and the Chase outage on Monday were also triggered by changes, not attackers. [Inferred]

What is new here is the time bomb: the change was three months old and had passed every gate. [Inferred]

Technical lessons learned

1. Safe deployment is not the same as safe

Canaries and gradual rollouts catch bugs that show up quickly. A defect that needs a specific mix of load, hardware and traffic can pass every stage and sit for months. Keep watching resource behaviour (lock wait time, CPU per request, memory) long after a rollout ends, and compare fleets running old and new code.

2. Test on the hardware you actually run

The labs used weaker, fewer-core machines; production ran newer, many-core hardware where contention is worse. Load tests should cover the top of your fleet, not the average, and match the production traffic mix.

3. Failover is load, so budget for it

The Teams failover was normal, and so was 14% capacity being out. Together they removed the headroom. Capacity planning must assume “one failover plus one degradation at the same time”, and should alert when headroom drops under that line.

4. Health checks plus automatic redirection can amplify failure

Every server the system removed pushed more load onto the rest. Cap how much of a fleet automation may eject at once, and when too many nodes fail together, stop redirecting and shed load instead. Microsoft ended up disabling some redirection by hand.

5. Bulkheads must actually hold

The regional partitions existed on paper but did not stop the cascade. Test isolation with real failure injection: kill a region’s worth of capacity and check that neighbours don’t follow.

6. Recovery needs admission control

Restarted servers were buried by the backlog. Bring capacity back behind rate limits, connection caps and backoff with jitter on the client side, so recovery is not a second outage.

7. Plan rollback for old changes too

Feature flags and flighting only help while the change is fresh. Keep the “last known good” path buildable and deployable for months, and practise reverting a change that is no longer the most recent one.

8. Watch your hubs

Exchange Online is a hub for Teams, Copilot, Defender and more, and it could not fail over “for scale reasons”. Map which services cannot leave a shared dependency; those are your single points of failure.

A quick checklist

  • ☐ Post-rollout monitoring for resource contention, for months, not hours.
  • ☐ Load tests on newest, highest-core hardware with production traffic mix.
  • ☐ Headroom for one failover plus one degraded segment at the same time.
  • ☐ Limit on how much of a fleet automation can remove at once.
  • ☐ Regional bulkheads proven with failure injection.
  • ☐ Rate-limited, staged recovery; clients with backoff and jitter.
  • ☐ Tested rollback path for changes that are months old.
  • ☐ Map of hub services and what cannot fail over.

Open questions

  • What exactly was the “separate issue” that broke the regional bulkheads?
  • What was the original fix in July meant to solve?
  • Why can’t Exchange Online fail over to the alternative gateway, and will that change?

Sources

Facts as of 17 November 2026, based on Microsoft’s preliminary report. The final report is due on 20 November 2026 and some details here may change.

Tags: , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *