Alex "RubySea" H.

M.S. in Analytics, Gun Nut, Email: [email protected]

[Alex "RubySea" H. @Rubysea]

Reminder I am available https://t.co/jeH8Dr4Aan and on the fediverse https://t.co/DGuI9oQC76 you can also find my website on Ghost https://t.co/egyx6blx9v

[AC 🇪🇸🇧🇷 @alejandrocdelg0]

este es el mayor tweet de desempleado que he visto en la historia

[Luke Elin @LukeElin]

Lots of hot takes on the Microsoft 365 outage. Here's an engineering one.

To be clear: this is speculation. Microsoft hasn't confirmed a root cause. It's my read of their own updates, from 35 years of building and breaking systems.

What Microsoft has actually said:

• Traffic isn't up. Capacity is down. Servers are failing health checks and dropping out.

• A process that establishes secure connections gets overwhelmed, and servers stop processing requests.

• Servers they bring back fall over again as soon as traffic hits them.

• They're reverting a network config change that lines up with when it started. But changes aren't syncing, and some machines are stuck in a bad state.

• No indication of an attack.

My hypothesis: a metastable failure.

Whatever triggered it may be long gone. The outage is now keeping itself alive.

1. Reconnect storm. Every Outlook, Teams and mobile client on the planet is hammering for a fresh secure connection.

Full TLS handshakes are CPU-heavy, and a freshly rebooted server has a cold session cache.

A server comes up, eats a wall of handshakes, CPU pegs, it fails its health check and gets pulled, and its load lands on the next one. Dominoes.

2. The fix shares fate with the failure. If the rollback travels over the same secure channel that's melting, the cure can't reach the patient.

That's exactly what "changes aren't syncing" sounds like to me.

That's why they're throttling traffic types and rebuilding servers from an older config instead of flipping a switch.

You don't get out of this by finding the bug.

You get out by shedding load, restoring capacity in isolated cells, and letting traffic back in slowly.

Where AI fits (also speculation): not "an AI wrote the bad line". It's velocity. Microsoft flagged several recent config changes as suspects and took hours to find one that correlated.

When the volume of change outpaces the humans who understand each change, finding the culprit gets slow and rollbacks get messy. Automate the shipping faster than you automate the understanding, and this is the bill.

The takeaway for every business: your email, chat, files, identity and security tooling now share one failure domain. Today proved it. Have a way to talk to your team that doesn't live in the same tenant.

Happy to be proven wrong by the post-incident review. And I hope this post ages terribly.

[Jeff Geerling @geerlingguy]

Microsoft364

[Alex "RubySea" H. @Rubysea]

@vortmax79 It’s India shift of course it is ESL.

[Alex "RubySea" H. @Rubysea]

@tvheidihatch It’s not they have their heads so far up their ass it is going to take an excavator to remove it.

[Alex "RubySea" H. @Rubysea]

Just tried to check my email for work I am a damn idiot.

[Banana3 @Banana3Stocks]

JENSEN HUANG is fucking my portfolio too

[litquidity @litcapital]

Women: *exist*

Jensen:

[Josh Cook | Microsoft MVP @FlowAltDelete]

Microsoft Office gets killed on this app every week.

Then Monday comes and someone in finance asks you to resend it in Excel.

[The Fat Electrician @Fat_Electrician]

Can someone explain to me why we call this a “honeypot” and not a Trojan Whore?

[Happy Captain @EODHappyCaptain]

If you’ve ever drank a Jägerbomb, thereby disarming it, you may be entitled to an Explosive Ordnance Disposal Badge.

[Melissa Chen @MsMelChen]

Who did this

[Swalt @isw_alt]

Jensen did the meme

[Selena🐝🐬 @chaoticcanon381]

generational meme unlocked

[Garrett Scott 🕳 @thegarrettscott]

I guess Chinese models really were the biggest threat…

[F1 TROLL @f1trollofficial]

Every team leaves Brazil with something

[Mahir 🇹🇷🇬🇧 @ScrewderiaF1]

Verstappen at the Brazil GP

[Wall Street Mav @WallStreetMav]

Utah’s Republican Party has fully embraced crony politics. This is why Republicans lose.

[Lars Mapstead @LibertarianLars]

Government control is never the answer.