This is incredible.
Microsoft is unable to recover from the Outlook outage and they want you to know.
This is the outcome of years of centralization of what should’ve been decentralized to begin with.
Centralization will crumble in the Superintelligence era.
‼️Update: Microsoft confirmed that its recent fixes haven't worked as expected. Engineers are exploring other ways to increase capacity and address ongoing traffic imbalances. Administrators can track updates in the Microsoft 365 admin center under EX1550543 and MO1550821.
Details: https://t.co/3i6kwq27MX
🚨 MICROSOFT SAYS ITS FIXES AREN'T WORKING. THE SERVERS IT BRINGS BACK CAN GO DARK AGAIN THE MOMENT TRAFFIC HITS THEM.
More than 10 hours in, it still can't say what's triggering it.
→ 3:19pm ET — Microsoft: no recent change lines up with when this started
→ 5:09pm ET — "our actions taken thus far have not provided the anticipated relief"
→ 7:06pm ET — returned servers "can become unresponsive again once they receive traffic"
→ 7:06pm ET — it rebuilds some servers on a previous configuration and limits some traffic in some locations
→ 10:13pm ET — its own words: the changes are "not taking as expected"
First Microsoft blamed elevated load. Then it said servers went unexpectedly unavailable and dumped their load on the rest.
It paused all deployments, looked into high processor use, and restarted servers with a fix in place. The outage is still not over.
Outlook. Teams. SharePoint. OneDrive. Office on the web. Copilot.
Even Defender XDR, the security platform companies use to watch for attacks.
OpenAI and Anthropic say their Microsoft connectors broke too.
Reports hit 74 countries and territories in about an hour. Asia started its Friday with it still broken.
Electric truck maker Workhorse pushed its earnings from November 12 to November 16 because of it.
No sign of a cyberattack, Microsoft says. No traffic surge. No named cause.
Just a server fleet that won't stay up, and a company that can't say why.
This account stays on the Microsoft outage until it is fixed and will report the confirmed cause. Follow for that update.
https://t.co/mdIeYjdoUa
The servers we returned to service can become unresponsive again once they receive traffic; we're analyzing diagnostic data to determine what's triggering the behavior. In parallel, we're temporarily limiting which types of traffic can reach the affected infrastructure in some locations. Additionally, we're rebuilding a subset of affected servers using a previous configuration to validate if this prevents them from becoming unresponsive. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as https://t.co/pYdahqxDv3 if the admin center is not available to you.
I’m not fucking joking, I’m moving the org off Microsoft products. I have grown tired of their spaghetti code and endless bullshit. Open source managed by AI is the only way forward for anything that matters.
Home Depot blocked my Grok Bot from shopping.
Amazon is blocking it too.
Walmart let it right in.
In 5 years half of us won't shop for ourselves. The stores locking the door now are going to wonder where their customers went.
Who else is getting blocked?
I've been managing Exchange servers for nearly 2 decades.
It is one of the most complex pieces of infrastructure to manage because its databases are enormous and take a long time to repair and rebuild under disaster situations.
Additionally, if the majority of a Database Availability Group (DAG) failed simultaneously, users could experience data loss.
Recovering from backups is an extremely long process and backups could be outdated by a few hours (this is how Exchange backups work).
Imagine moving hundreds of petabytes of data over iSCSI storage. Regardless of how fast the iSCSI and links are, the sheer amount of data makes this a daunting task.
I wouldn't be surprised if this recovery took multiple days and ended up with data loss for multiple users.
I do not miss the days when I'd be woken up at 2am because an Exchange server failed and the DAG was offline.