Did your email go down last week? This latest Microsoft outage was rather wild. It's starting to feel like the world is imploding. The central premise of #SREGym is to see if AI can stop such outages.
As a sneak peak of SREGym 2.0, we've recreated the Outlook outage. Check it out: https://t.co/Tp6E4nsSKD it's currently very challenging for frontier models.
What other outages would you like to see?
Stay tuned, we have something coming after Thanksgiving 🦃
The pre PIR of the Outlook/Microsoft outage has much more clarify. Thank you @Microsoft !
The outage is indeed a very hard one; many unfortunate events occurred at the same window.
A latent bug exhausting CPU/memory under specific load condition was deployed worldwide, with no impact for 3 months. The bug was triggered by a surge in load, caused by an unrelated failover of Teams and another unrelated loss of 14% of server capacity. The system is then badly overloaded; 80% of servers were unresponsive. Then, the isolation mechanism between fault domains didn't work (it was vague in the PIR), causing cascading failures till a global outage.
Then, the (likely automatic) recovery unsurprisingly hit the classic thundering herd effect.
In this case, root-cause analysis is on the critical path and it took more than 15 hours to figure out the latent bug and revert it (it's really a very nasty bug!)
The long recovery time is quite understandable. Given such a global outage, it has to be paced carefully.
I downloaded the report at,
https://t.co/KgMp7URZwf
1/5
In our previous two studies, we explored how decision models like Jev could diagnose SRE incidents, and how LLM review could improve their results.
This time, we shifted our focus from diagnosing SRE incidents to evaluating diagnosis reports with Jev-as-a-Judge
4/5
Beyond these results, Jev-as-a-Judge could make it practical to evaluate every agent change and provide fast, low-cost feedback for reinforcement learning. The collected reports and verdicts could potentially help train a dedicated judge model in the future.