Albert Gong

CS PhD @ Cornell | Yale '23

[Raaz Dwivedi @raazdwivedi]

SWE-bench gave the field a shared standard for whether an agent can fix code. On-call reliability needs the same thing.

The challenge is that SRE has no GitHub equivalent: no large public corpus of incidents with telemetry, investigation context, and agreed-upon root causes. Building a useful benchmark therefore requires defining what realistic incidents look like, how vague the starting context should be, what counts as accurate RCA, and how to evaluate an agent that is helpful without being completely correct.

ORCA-bench is our attempt to establish that standard. Unlike prior RCA benchmarks, it combines production-fidelity task design, realistic telemetry interfaces, source code, expert-validated ground truth, and independently validated evaluation.

Applying that benchmark to five frontier agents produced some sobering results. Even under relatively favorable conditions, the best model reached just 25.3% RCA accuracy on medium-difficulty tasks and 10% on hard tasks, while hallucination rates ranged as high as 40.2%.

Link in comments

[Traversal @traversal_ai]

How ready are today's frontier coding agents for on-call?

Today we're releasing ORCA-bench, a new benchmark from Traversal's AI Research team, built with @Columbia and @cornell_tech, for evaluating AI agents on production root cause analysis.

Paper, benchmark, dataset, and leaderboard: https://t.co/1fDDtEiXJL

Key findings and why they matter below. 🧵

[Anmol Kabra @anmolkabra]

On the heels of Jensen's @NVIDIAGTC keynote - @Cornell 🤝 @NVIDIA. We are the first and only academic lab in the world with early access to the DGX Station. My ICLR paper showed that data isn't a bottleneck for LLM reasoning. With this machine, we confirmed compute isn't either!

[Albert Gong @GongAlbert]

@glamourgirl35 18 more hours!