The Microsoft email outage delayed chlorine safety alerts at a German theme park hotel, leaving 15 guests with breathing problems.
A faulty system pumped too much chlorine into the pool at Hotel Ling Bao, part of Phantasialand. The pool’s alarm system relied on email alerts, but they never arrived because of an Outlook/Exchange outage.
Four guests were taken to hospital, and the pool was temporarily closed.
Appreciate all the humor around the Microsoft outage. Your average joe in the corporate is obviously grateful for it. But once we are done with all the silliness, I need y'all to understand a couple of things:
- regardless of what anyone says, this was not an accident or some bug.
- your entire economy rests on the tech infrastructure that is in place. we are all connected, and that has its pros and cons.
- now think the same outage, but for a longer period of time, and across various platforms and services, and across continents.
- in those conditions, you're talking about big businesses practically coming to a halt, mass disruption in civil services like police, medical, fire brigade, etc. and that's just the tip of it.
Again, if you are not prepared, whether as an individual, group, corporation or government, you will be in trouble very soon.
Lots of hot takes on the Microsoft 365 outage. Here's an engineering one.
To be clear: this is speculation. Microsoft hasn't confirmed a root cause. It's my read of their own updates, from 35 years of building and breaking systems.
What Microsoft has actually said:
• Traffic isn't up. Capacity is down. Servers are failing health checks and dropping out.
• A process that establishes secure connections gets overwhelmed, and servers stop processing requests.
• Servers they bring back fall over again as soon as traffic hits them.
• They're reverting a network config change that lines up with when it started. But changes aren't syncing, and some machines are stuck in a bad state.
• No indication of an attack.
My hypothesis: a metastable failure.
Whatever triggered it may be long gone. The outage is now keeping itself alive.
1. Reconnect storm. Every Outlook, Teams and mobile client on the planet is hammering for a fresh secure connection.
Full TLS handshakes are CPU-heavy, and a freshly rebooted server has a cold session cache.
A server comes up, eats a wall of handshakes, CPU pegs, it fails its health check and gets pulled, and its load lands on the next one. Dominoes.
2. The fix shares fate with the failure. If the rollback travels over the same secure channel that's melting, the cure can't reach the patient.
That's exactly what "changes aren't syncing" sounds like to me.
That's why they're throttling traffic types and rebuilding servers from an older config instead of flipping a switch.
You don't get out of this by finding the bug.
You get out by shedding load, restoring capacity in isolated cells, and letting traffic back in slowly.
Where AI fits (also speculation): not "an AI wrote the bad line". It's velocity. Microsoft flagged several recent config changes as suspects and took hours to find one that correlated.
When the volume of change outpaces the humans who understand each change, finding the culprit gets slow and rollbacks get messy. Automate the shipping faster than you automate the understanding, and this is the bill.
The takeaway for every business: your email, chat, files, identity and security tooling now share one failure domain. Today proved it. Have a way to talk to your team that doesn't live in the same tenant.
Happy to be proven wrong by the post-incident review. And I hope this post ages terribly.
HOLY: Anthropic's Claude Code engineer Addy Osmani is warning about a brutal AI paradox: the more coding you hand to agents, the more you risk losing the very skills needed to supervise them.
He calls it "cognitive surrender." AI-generated code looks pristine, tests pass, and engineers start approving work they no longer understand.
Anthropic's own research warns: "supervising Claude requires the very coding skills that may atrophy from AI overuse."
Osmani makes a striking observation: "When you're producing more than you can stand behind, you're no longer reviewing. You're hoping."
Google is trying to use its monopoly power to force AI into our public schools.
No corporation should have the power to dictate how we teach our children.
Our public schools exist to serve students, not to grow the profit margins of Big Tech.
https://t.co/jfACMS1Sh2
Google is using its search monopoly to coerce schools into adopting AI. This highlights the danger of allowing an unregulated monopolist to control an essential service.
Multiple courts have found Google liable for violating antitrust laws, and this latest move shows it remains unchastened.
As the public increasingly pushes back against the use of AI in schools, Google is making clear it will punish those who don’t fall in line.
Weekend hack: HackerTube. I was annoyed with YouTube's recommendations so I had Claude filter down 43M comments over 20 years of HN to 6,454 videos that keep being recommended. *Significantly* better viewing experience, ~2 hours with Opus 5.5:
https://t.co/JIhXN7OSh1
- Excellent recs without any logins (state is kept in local browser session)
- Thumbnail is a representative frame from middle of the video instead of the super duper A/B test hyper-optimized user uploads
- Underlying data from the comment analysis as one zip file, point your Claude at it and ask for your own version
- HackerTube Longs ✊
A bit more on the process:
I use Claude Code (desktop app for Mac), Opus 5.5 on Medium effort as my default. I only talk to Claude and have it do everything, the only "manual work" I helped with in tandem was in the browser (reminiscent of MenuGen): 1) I had to create some kind of API key from Google Cloud UI to query YouTube for video metadata, 2) I booted up a VPS box for serving and 3) I bought the domain name. Once the box was up I gave Claude the IP and told it update, secure, harden, deploy and optimize the code, and it did the whole thing.
Many awesome projects that would have normally taken weeks/months even can now be few-shotted for breakfast. What else are we not thinking of.