Pial

MSCS@UIUC

Working on AI for SRE

[Dimitris Papailiopoulos @DimitrisPapail]

One of the worst forms of brainrot AI has cultivated is pessimism about basic research, ie the idea that important work can only happen inside a frontier/neo lab and only with 10k+ GPUs, so the rest should not even bother. What a bleak way to think about science. And it's false.

[Tianyin Xu @tianyin_xu]

This post is long overdue, which I owe to the students from the UIUC++ SRSE program. We finally wrote a blog on how we curated a set of high-quality SRE benchmark problems by turning real-world postmortem reports into reproducible SRE problems atop #SREGym. It may give a good example of community-based data curation. Check it out,
https://t.co/vCLXrf87Zk

The fundamental challenge of AI-for-SRE benchmarks is the quality of the problems, e.g., in terms of realism and the ability of challenging frontier AI. This experience enabled us to understand many faces of the challenge and reflect on important tradeoffs between realism, reproducibility, cost, and utility. We learned many lessons, which would guide the effort on building #SREGym 2.0.

Big thanks to @SaadMRP who (unexpectedly) spent his entire summer managing the program with a heroic effort.

I list all the student contributors at the end of the post. Many of them are on Twitter: @12_AbdAllAh_12 @ermiasmulu19 @sharq_fr @M_ABDz_ @MunimThahmid @Sai_Hari_g @ibnAmjid @SuMaya971503 @TalhaAsif25 @tanzimhromel @tejaspkshukla @Varunihk. They did great work to achieve high-quality problems that challenge AI frontier. If you're looking for students or employees, they are good candidates!

[Tianyin Xu @tianyin_xu]

Well deserved, @HacksonClark and the #SREGym team! AI for SRE is in an unusual situation, where no strong benchmark is available and the technical landscape is rather opaque. The fundamental challenge is to curate high-quality problems that can reflect real-world characteristics of SRE tasks. It's great to see #SREGym continuously pushing on this direction and offering an open platform for everyone, and more importantly, the active community they have built. The #SREGym Slack has 250+ members and the project has 60+ contributors. The best part is to hear from more and more practitioners who are using it.

[Pial @SaadMRP]

Our SREGym paper was accepted to NeurIPS 2026! 🎉
Proud to be part of the team behind this work. It’s been especially exciting to see the benchmark grow with help from 60+ contributors. Huge thanks to everyone who helped make it happen!

[Jackson Clark @HacksonClark]

I'm excited to share that https://t.co/eDFaNGJ0TF has been accepted to NeurIPS 2026! 🎉

Huge thanks to my co-lead @Yiming_Su3, our co-authors @SaadMRP, @lilygn6, Yifan Tian, Hans-Arno Jacobsen, @yinfang_chen , @TianyinXu, and our 60+ contributors!

We have benchmarks for agents that write code.

We built one for what happens after you deploy it.

Can AI resolve production issues? 🧵 [1/N]

[Tianyin Xu @tianyin_xu]

Thanks for the shoutout to TLA+ @bcherny !

As suggested by Markus Kuppe, let me showcase the brilliant work done by colleagues at Specula and the TLA+ community. While Specula (https://t.co/hoV6UJIybf) can effectively model system code in TLA+ (and then use model checkers to find hundreds of deep bugs), we can further prove the correctness of system specs using TLAPS. The TLAPS-Bench (https://t.co/hrYHc4DSow) project aims to offer a proof service for any TLA+ spec of important systems/protocols. Basically, Specula writes the spec; TLAS-Bench generates the proof.

AI writes amazing proofs! We had to retire all Proof-Completion tasks in TLAPS-Bench as they're too easy for frontier AI. We also proved classic protocols and algorithms like 2PC, Paxos, TCP, etc.

However, proving low-level specs of real-world system code _from scratch_ is still non-trivial. @qiancheng9788 ran a subset of TLAPS-Bench using Claude Code w/ Opus 5. The success rate is merely 51.4%. @Muse is great, but Muse Spark 1.3 can only achieve 16.7%.

Ruize has a bag of tricks to push AI agents. Once he pushed Codex to prove one of the nine invariants for the ZooKeeper implementation. Codex struggled 5 days, burned $2000, and did it correctly. We ran out of money to continue. And, @xu_dong_sun shows us that liveness proofs are still difficult for AI without guidance. We have an exciting avenue, but still a lot of work to be done.

If anyone wants to verify their protocols/systems, send us some tokens -- we can do it in TLAPS-Bench. @HacksonClark tells me that people are excited about long-horizon tasks -- TLAPS-Bench may be a perfect benchmark for that.

@bcherny Consider further tuning Claude for TLA+, you may find more bugs and/or prove the correctness of SDK code more efficiently.

[Tianyin Xu @tianyin_xu]

Excellent work on formally verifying the #Kubernetes control plane using a compositional approach led by @Cat_overflow and @xu_dong_sun, with the awesome team of @NikhilDate5 @tylergu_jiawei Cody @tchajed and Oded. It shows a practical path to verify a large control plane by gradually verifying individual controllers progressively.

The compositional verification is enabled by a new spec termed CORE (COmpositional REconciliation), which reuses ESR (the original #anvil spec) to specify each controller's reconciliation and employs rely-guarantee conditions to restrict inter-controller interference. CORE encodes both liveness and safety -- liveness captures correct state reconciliation logic and safety restricts inter-controller interference.

Check out the paper which has a great amount of insights on specification, proof, and implementation.
https://t.co/p5kRoE84xM

[Jackson Clark @HacksonClark]

Can a small and fast decision model make SRE agents more reliable? 🤔
We gave an agent access to @typesafeai’s Jev through the Codex harness and tested it on 10 SREGym-Lite problems.
Three findings stood out. More details in the thread. 🧵

[Pial @SaadMRP]

Excited to see our work on SREGym at the Frontier Data Summit! If you’re attending, stop by @HacksonClark’s poster and say hi!

[Jackson Clark @HacksonClark]

Super excited to be presenting https://t.co/HdYZGstPVj at @SnorkelAI's Frontier Data Summit, Oct 8 in SF. Can AI really solve production issues? Come find the poster. Request an invite: https://t.co/drAHbW8nOU

[Tianyin Xu @tianyin_xu]

Seeing the PR to #SREGym by @niallm, an author of the holy Google SRE book (@srebook) and the Reliable Machine Learning book is rather rewarding.

The PR is very SRE style 🙂
https://t.co/oLyDk6xcPv

Niall, now a Distinguished Engineer @ciroosai, is also a great, patient mentor for students on the #SREGym Slack.

[月球大叔 @uncle_moon_x]

好久好久没有见到如此纯粹的人了。从2007年2026年，还是能在天音身上看到当年那个热血足球少年的影子

“我每年都建议取消best paper”

“给人类做的系统都要给Agent重新做一遍”

……

让我们一起进入徐天音的世界

徐天音：系统，UIUC，教授，最佳论文，Agent Infra，云计算，形式化验证，容错，纯粹，好导师 https://t.co/oqss4uYo6f via @YouTube

[Tianyin Xu @tianyin_xu]

Yeah, great time for reliability research! The scope of reliability is significantly expanded from model reliability, to agent reliability, to (vibe) code reliability, and their interactions in end-to-end systems, not to mention the opportunities of leveraging AI to attack some of of the decades-long reliability challenges in large-scale systems.

What's interesting about AI is that reliability is no longer about edge cases, but a foundation to have functional, stable systems.

[Tianyin Xu @tianyin_xu]

Glad to see more and more folks like @cerebral_system are using #SREGym to evaluate SRE/Prod agents. It's a great sigh that folks are more serious in building useful agents than claiming fake victories.

Results on low-fidelity benchmarks that do trivial fault injections (e.g., flipping a feature flag in AS) is honestly like the emperor's new clothes. We all know that prod issues are never alike.

Fidelity is arguably the hardest problem in SRE benchmarks. #SREGym doesn't have a perfect solution (there's no free lunch), but pushes very hard on it while balancing cost and reproducibility.

Kudos to @HacksonClark @Yiming_Su3 @SaadMRP @lilygn6 @MunimThahm81566 and many other students who treat eval as a way of understanding the problem, not playing the game.

[Tianyin Xu @tianyin_xu]

We released a beta version of Specula, a tool that automatically checks system code using TLA+ based formal methods. Specula instructs coding agents to fully automate formal specification (model + invariants) and mode-code conformance, which were major barriers to adopt formal methods in practice. We find Specula rather useful in finding deep bugs that require formal reasoning. (Specula found hundreds.)

If you need automatic formal reasoning of your code, give it a try. (The tool is push-button.)

Code: https://t.co/gcqFUHYGL6
Paper: https://t.co/4W2p1yX5ia

(While I'm posting it, the students, especially @qiancheng9788 and @SaadMRP, and Ruize Tang did all the hard technical work.)

CC @disalg_spec @Yiming_Su3

[Yiming Su @Yiming_Su3]

The SREGym team (me, @HacksonClark , @lilygn6 in person, @SaadMRP remotely) will be giving a demo at CAIS @CAISconf TODAY at 4:30 PM in San Jose room! Come to chat with us about AI SRE, benchmarking, and more!

Bonus: if you have a laptop, we have a self-contained demo/artifact for you to try without LLM credits here: https://t.co/eAhMIy3AJu.

[Joachim Baumann @joabaum]

Can you boost your AI review scores by asking an LLM to rewrite your paper?
Yes! We call it paper laundering
Our @icmlconf spotlight paper argues current AI reviewers aren't ready to automate peer review, and outlines what a science of peer review automation should look like🧵👇

[Pial @SaadMRP]

Checkout SREGym here: https://t.co/51cRqF9bc2

[Pial @SaadMRP]

Our demo paper "SREGym: A Live Training Ground for AI SRE Agents with High-Fidelity Failure Drills" got accepted at ACM CAIS '26! It benchmarks AI agents on high-fidelity SRE incidents in live Kubernetes environments. Thanks to the team, @HacksonClark, @Yiming_Su3, Lily

[Yiming Su @Yiming_Su3]

We are happy to share that our demo paper on SREGym got accepted into ACM CAIS '26 as a System Demonstration submission! A huge thank-you to all people in the team for making this happen.

SREGym is AI SRE benchmark consists of challenging, high-fidelity SRE incidents to evaluate AI SRE solutions in their diagnosis and mitigation capabilities.

We are very, _very_, happy that the reviewers see and appreciate the importance in the _engineering quality and usability_ of SREGym. We want our user experience to be as smooth as possible, and the reviewers confirm and agree with that (thanks!). Positive signals like this encourages us to keep pushing hard on SREGym and the AI SRE frontier.

See you in San Jose!

[Matei Zaharia @matei_zaharia]

Registration is open for the CAIS conference! We have some awesome workshops and papers in the program.

[Laude Institute @LaudeInstitute]

SREGym/@HacksonClark, Yiming Su (@UofIllinois ) - SRE is where agentic AI gets high-stakes fast: one wrong action can cascade an outage or corrupt data. SREGym introduces safety-first guardrails and realistic benchmarks so AI agents managing production infrastructure are something operators can actually trust.