This case is complicated and nuanced. There’s a long read here that dives deep into what happened, how it can happen, and what it says about political dissent in the Trump era.
Here's a link to the full story: https://t.co/wOKk9qzTR8
Lots of hot takes on the Microsoft 365 outage. Here's an engineering one.
To be clear: this is speculation. Microsoft hasn't confirmed a root cause. It's my read of their own updates, from 35 years of building and breaking systems.
What Microsoft has actually said:
• Traffic isn't up. Capacity is down. Servers are failing health checks and dropping out.
• A process that establishes secure connections gets overwhelmed, and servers stop processing requests.
• Servers they bring back fall over again as soon as traffic hits them.
• They're reverting a network config change that lines up with when it started. But changes aren't syncing, and some machines are stuck in a bad state.
• No indication of an attack.
My hypothesis: a metastable failure.
Whatever triggered it may be long gone. The outage is now keeping itself alive.
1. Reconnect storm. Every Outlook, Teams and mobile client on the planet is hammering for a fresh secure connection.
Full TLS handshakes are CPU-heavy, and a freshly rebooted server has a cold session cache.
A server comes up, eats a wall of handshakes, CPU pegs, it fails its health check and gets pulled, and its load lands on the next one. Dominoes.
2. The fix shares fate with the failure. If the rollback travels over the same secure channel that's melting, the cure can't reach the patient.
That's exactly what "changes aren't syncing" sounds like to me.
That's why they're throttling traffic types and rebuilding servers from an older config instead of flipping a switch.
You don't get out of this by finding the bug.
You get out by shedding load, restoring capacity in isolated cells, and letting traffic back in slowly.
Where AI fits (also speculation): not "an AI wrote the bad line". It's velocity. Microsoft flagged several recent config changes as suspects and took hours to find one that correlated.
When the volume of change outpaces the humans who understand each change, finding the culprit gets slow and rollbacks get messy. Automate the shipping faster than you automate the understanding, and this is the bill.
The takeaway for every business: your email, chat, files, identity and security tooling now share one failure domain. Today proved it. Have a way to talk to your team that doesn't live in the same tenant.
Happy to be proven wrong by the post-incident review. And I hope this post ages terribly.
With Dems all but certain to prevail in Mississippi and Alaska, we're most likely looking at a 56D-44R Senate and a ~260D-175R House, which would be Dems' largest House majority since 1992.
AP: Patrick Stewart, the classically trained stage actor who scaled the Shakespearean canon and added gravitas to a “Star Trek” franchise and the “X-Men” films, has died. He was 86.
https://t.co/2MGATfsguF
“Mr. Solomon’s presentation, both times, also omitted key context: The field office’s legal doubts came before Mar-a-Lago surveillance footage emerged that showed a Trump aide moving boxes from a storage area before a lawyer came to search it for subpoenaed classified files. The additional evidence resolved legal concerns about probable cause.”
How the White House Manufactured a
Narrative About a Trump Aide’s Call Logs https://t.co/FBwTnjOw1X via @charlie_savage
People are dragging Nikole Hannah-Jones for this, but I think this piece is incredible. Trying to live your values, discovering it’s not working and having an open conversation with your kid about that is eminently laudable — Pubbing about it even more so. https://t.co/aMJuKt9QnR