Yiming Su

Distributed Systems, SRE
PhD student @ UIUC @siebelschool, prev @MSFTResearch

[Yiming Su @Yiming_Su3]

@LakshyAAAgrawal @aiDotEngineer Congrats Lakshya!

[Tianyin Xu @tianyin_xu]

This post is long overdue, which I owe to the students from the UIUC++ SRSE program. We finally wrote a blog on how we curated a set of high-quality SRE benchmark problems by turning real-world postmortem reports into reproducible SRE problems atop #SREGym. It may give a good example of community-based data curation. Check it out,
https://t.co/vCLXrf87Zk

The fundamental challenge of AI-for-SRE benchmarks is the quality of the problems, e.g., in terms of realism and the ability of challenging frontier AI. This experience enabled us to understand many faces of the challenge and reflect on important tradeoffs between realism, reproducibility, cost, and utility. We learned many lessons, which would guide the effort on building #SREGym 2.0.

Big thanks to @SaadMRP who (unexpectedly) spent his entire summer managing the program with a heroic effort.

I list all the student contributors at the end of the post. Many of them are on Twitter: @12_AbdAllAh_12 @ermiasmulu19 @sharq_fr @M_ABDz_ @MunimThahmid @Sai_Hari_g @ibnAmjid @SuMaya971503 @TalhaAsif25 @tanzimhromel @tejaspkshukla @Varunihk. They did great work to achieve high-quality problems that challenge AI frontier. If you're looking for students or employees, they are good candidates!

[Yiming Su @Yiming_Su3]

@vaastav05 @AntoineKau @mpi_jcmace Congratulations Vaastav!

[Tianyin Xu @tianyin_xu]

Well deserved, @HacksonClark and the #SREGym team! AI for SRE is in an unusual situation, where no strong benchmark is available and the technical landscape is rather opaque. The fundamental challenge is to curate high-quality problems that can reflect real-world characteristics of SRE tasks. It's great to see #SREGym continuously pushing on this direction and offering an open platform for everyone, and more importantly, the active community they have built. The #SREGym Slack has 250+ members and the project has 60+ contributors. The best part is to hear from more and more practitioners who are using it.

[Yiming Su @Yiming_Su3]

SREGym is accepted in NeurIPS! There’s still MUCH work to be done on AI SRE, and a solid benchmark is always the first step.

It is also my great honor to be a co-lead of the project with @HacksonClark and work with the brightest team in the world. Let’s keep shipping!

[Jackson Clark @HacksonClark]

I'm excited to share that https://t.co/eDFaNGJ0TF has been accepted to NeurIPS 2026! 🎉

Huge thanks to my co-lead @Yiming_Su3, our co-authors @SaadMRP, @lilygn6, Yifan Tian, Hans-Arno Jacobsen, @yinfang_chen , @TianyinXu, and our 60+ contributors!

We have benchmarks for agents that write code.

We built one for what happens after you deploy it.

Can AI resolve production issues? 🧵 [1/N]

[Yiming Su @Yiming_Su3]

@qizhengz_alex Congrats Qizheng!! Happy for you!

[Tianyin Xu @tianyin_xu]

Thanks for the shoutout to TLA+ @bcherny !

As suggested by Markus Kuppe, let me showcase the brilliant work done by colleagues at Specula and the TLA+ community. While Specula (https://t.co/hoV6UJIybf) can effectively model system code in TLA+ (and then use model checkers to find hundreds of deep bugs), we can further prove the correctness of system specs using TLAPS. The TLAPS-Bench (https://t.co/hrYHc4DSow) project aims to offer a proof service for any TLA+ spec of important systems/protocols. Basically, Specula writes the spec; TLAS-Bench generates the proof.

AI writes amazing proofs! We had to retire all Proof-Completion tasks in TLAPS-Bench as they're too easy for frontier AI. We also proved classic protocols and algorithms like 2PC, Paxos, TCP, etc.

However, proving low-level specs of real-world system code _from scratch_ is still non-trivial. @qiancheng9788 ran a subset of TLAPS-Bench using Claude Code w/ Opus 5. The success rate is merely 51.4%. @Muse is great, but Muse Spark 1.3 can only achieve 16.7%.

Ruize has a bag of tricks to push AI agents. Once he pushed Codex to prove one of the nine invariants for the ZooKeeper implementation. Codex struggled 5 days, burned $2000, and did it correctly. We ran out of money to continue. And, @xu_dong_sun shows us that liveness proofs are still difficult for AI without guidance. We have an exciting avenue, but still a lot of work to be done.

If anyone wants to verify their protocols/systems, send us some tokens -- we can do it in TLAPS-Bench. @HacksonClark tells me that people are excited about long-horizon tasks -- TLAPS-Bench may be a perfect benchmark for that.

@bcherny Consider further tuning Claude for TLA+, you may find more bugs and/or prove the correctness of SDK code more efficiently.

[Yiming Su @Yiming_Su3]

@HacksonClark @typesafeai We loved testing Jev and see how FAST it is. We can't wait to do more crazy ideas with Jev. Thank you @typesafeai for releasing such a great model!

[Yiming Su @Yiming_Su3]

Check out SREGym’s first ever blog post - and it’s on Jev!

Seeing Jev’s release, we immediately think that it can be extremely valuable to SRE agents. Check it out!

[Jackson Clark @HacksonClark]

Can a small and fast decision model make SRE agents more reliable? 🤔
We gave an agent access to @typesafeai’s Jev through the Codex harness and tested it on 10 SREGym-Lite problems.
Three findings stood out. More details in the thread. 🧵

[Jackson Clark @HacksonClark]

Super excited to be presenting https://t.co/HdYZGstPVj at @SnorkelAI's Frontier Data Summit, Oct 8 in SF. Can AI really solve production issues? Come find the poster. Request an invite: https://t.co/drAHbW8nOU

[Yiming Su @Yiming_Su3]

Check out SREGym (https://t.co/TBqWvn0Fkv)!

[Snorkel AI @SnorkelAI]

Frontier Data Summit: Oct 8, SF. @fchollet, @sanmikoyejo, @nikogrupen, @ajratner, @alexgshaw, @StevenDillmann, and others onstage plus 25+ posters with the teams behind Agent's Last Exam, OSWorld 2.0, Terminal Bench Science, Terminal Bench, CollusionBench, PostTrainBench, T² Scaling Laws, SlopCodeBench, LONGRUN, SREGym, BizBench, and more.

[Tianyin Xu @tianyin_xu]

Seeing the PR to #SREGym by @niallm, an author of the holy Google SRE book (@srebook) and the Reliable Machine Learning book is rather rewarding.

The PR is very SRE style 🙂
https://t.co/oLyDk6xcPv

Niall, now a Distinguished Engineer @ciroosai, is also a great, patient mentor for students on the #SREGym Slack.

[Yiming Su @Yiming_Su3]

@OfirPress interesting that you didn't put "scalable" in the tldr; looks like being able to automatically construct tasks is the only way to keep up with model updates. curious to hear your thoughts on that.

[Minjia Zhang @_Minjia_Zhang_]

Excited to share XPress, our new work on accelerating speculative decoding with diffusion drafters: ~30% longer accepted drafts and ~1.3× higher end-to-end throughput over dFlash, reaching up to 8.2× speedup over autoregressive decoding across math, code, and chat benchmarks.

Block-diffusion drafters like dFlash are attractive because they can generate an entire block of draft tokens in a single forward pass. But this parallelism comes with a major limitation: each token is predicted largely from its own marginal distribution, without knowing which tokens were actually selected before it. The resulting tokens can look individually plausible while producing a sequence that the AR target model rejects early.

XPress asks a simple question: can we restore these missing causal dependencies without giving up the parallelism that makes diffusion drafting fast? We introduce a lightweight causal refiner that refines the entire draft block through a few parallel Jacobi refinement iterations. Importantly, the refiner is quite small: it reuses the diffusion drafter’s hidden states and adds only ~80M parameters, with the causal core just 2.8M parameters.

This work is a collaboration with IBM Research, including Naigang Wang, Davis Wertheimer, Fabian Lim, Mudhakar Srivatsa, and Raghu Ganti.

Following up on the paper we released earlier this month (https://t.co/RET7OUuekS), the code and model checkpoints are now public. We are also supporting more models with XPress now. Please stay tuned!

Read more: https://t.co/nSXsmFPbGe

Code: https://t.co/JbRx4l10B1

Model checkpoints: https://t.co/TRzH6vpGz0 and https://t.co/oRCMCg2Ir4

[Yiming Su @Yiming_Su3]

@melissapan did you bet yes or no

[Lewis Jones @thelcjones]

Transformers are cool but diffusion models are probably the most beautiful and elegant discoveries of the last decade

[Tianyin Xu @tianyin_xu]

Glad to see more and more folks like @cerebral_system are using #SREGym to evaluate SRE/Prod agents. It's a great sigh that folks are more serious in building useful agents than claiming fake victories.

Results on low-fidelity benchmarks that do trivial fault injections (e.g., flipping a feature flag in AS) is honestly like the emperor's new clothes. We all know that prod issues are never alike.

Fidelity is arguably the hardest problem in SRE benchmarks. #SREGym doesn't have a perfect solution (there's no free lunch), but pushes very hard on it while balancing cost and reproducibility.

Kudos to @HacksonClark @Yiming_Su3 @SaadMRP @lilygn6 @MunimThahm81566 and many other students who treat eval as a way of understanding the problem, not playing the game.

[Tianyin Xu @tianyin_xu]

Great to see people finding Specula useful. The tool is not perfect; we've already heard problems and are actively improving it (kudos to the students who work very hard @qiancheng9788 @SaadMRP). Any feedback helps us make it a better tool (and is truly appreciated)!

[Tianyin Xu @tianyin_xu]

We released a beta version of Specula, a tool that automatically checks system code using TLA+ based formal methods. Specula instructs coding agents to fully automate formal specification (model + invariants) and mode-code conformance, which were major barriers to adopt formal methods in practice. We find Specula rather useful in finding deep bugs that require formal reasoning. (Specula found hundreds.)

If you need automatic formal reasoning of your code, give it a try. (The tool is push-button.)

Code: https://t.co/gcqFUHYGL6
Paper: https://t.co/4W2p1yX5ia

(While I'm posting it, the students, especially @qiancheng9788 and @SaadMRP, and Ruize Tang did all the hard technical work.)

CC @disalg_spec @Yiming_Su3

[Cyrus (Zikai) Zhou @cyruszzhou]

As agents get more complex, we need better infrastructure for understanding and editing what they actually did, not just reading logs after the fact.