Reza (Rey) Sanayei

ML Intern @Google | Ex ML Research @ScaleAILabs | MSCS @Uarizona | NLP @LabCLU

[Reza (Rey) Sanayei @r_sanayei]

🤖 Agent: “Done! $30 promo applied. Pizza’s on the way 🍕”

💳 Bank statement: Charged full price 🤦‍♂️

Now imagine rewarding that run as a success during post-training.

Meet Sauron: an evidence-first agentic verifier that makes evaluation faster, cheaper, and more reliable through shared-evidence clustering, targeted trace/state queries, and independent claim verification.

📄 https://t.co/rHx2utFiB9

[Reza (Rey) Sanayei @r_sanayei]

@Radii2323 @talaei_shayan @jure @kvignesh1420 @adibvafa @nikitamounier Clever approach 🔥

[Mersad @Mersad_Abbasi]

Not all economically valuable tasks use energy the same way.
Sam Altman recently argued that home compute sits mostly idle, so hardware belongs in the cloud, where it gets maximum utilization.
I agree with the problem but I disagree with the solution Sam offers. 🧵

[Reza (Rey) Sanayei @r_sanayei]

We’ve also been exploring this with Sauron, our evidence-first agentic verifier:

https://t.co/ndt20uwTkv

[Reza (Rey) Sanayei @r_sanayei]

When verifier verdicts become RL rewards, errors cut both ways: missed failures can reward bad behavior, while false alarms can penalize correct actions.

To stress-test Sauron against both error types, we evaluate it on paired clean and fault-injected runs covering 13 failure families, including simulated reward hacking.

Sauron detects 97% of injected faults while falsely flagging only 2% of clean controls.

[Reza (Rey) Sanayei @r_sanayei]

Bench Summit by @Kaggle and @GoogleDeepMind has been a blast!

Benchmarks shape where the frontier is headed, and verification is at the heart of it.

[Mersad @Mersad_Abbasi]

this is a fun experiment to find systematic bias in different models. follow along !!

[Reza (Rey) Sanayei @r_sanayei]

@soldni Agreed! Our recent work on agentic verifiers addresses a similar challenge:

https://t.co/ndt20uwluX

[Reza (Rey) Sanayei @r_sanayei]

When verifier verdicts become RL rewards, errors cut both ways: missed failures can reward bad behavior, while false alarms can penalize correct actions.

To stress-test Sauron against both error types, we evaluate it on paired clean and fault-injected runs covering 13 failure families, including simulated reward hacking.

Sauron detects 97% of injected faults while falsely flagging only 2% of clean controls.

[Reza (Rey) Sanayei @r_sanayei]

📄 Paper: https://t.co/EkcpwoAAQx
🧑‍💻 Code: https://t.co/xRaTZgDTXO

Grateful for the collaboration this summer at @ScaleAILabs with @RazvanDuu @mhrezaeics @Haaris_Riaz @robert_nlp @Yminglai @liuying04 @msurd @danielyuez

[Reza (Rey) Sanayei @r_sanayei]

🤖 Agent: “Done! $30 promo applied. Pizza’s on the way 🍕”

💳 Bank statement: Charged full price 🤦‍♂️

Now imagine rewarding that run as a success during post-training.

Meet Sauron: an evidence-first agentic verifier that makes evaluation faster, cheaper, and more reliable through shared-evidence clustering, targeted trace/state queries, and independent claim verification.

📄 https://t.co/rHx2utFiB9

[Reza (Rey) Sanayei @r_sanayei]

When verifier verdicts become RL rewards, errors cut both ways: missed failures can reward bad behavior, while false alarms can penalize correct actions.

To stress-test Sauron against both error types, we evaluate it on paired clean and fault-injected runs covering 13 failure families, including simulated reward hacking.

Sauron detects 97% of injected faults while falsely flagging only 2% of clean controls.

[Reza (Rey) Sanayei @r_sanayei]

@mhrezaeics @thinkymachines Lessgoooo 🔥🔥🔥

[Reza (Rey) Sanayei @r_sanayei]

@mohit_r9a @reflection_ai Congrats Mohit 🤩🔥

[Razvan Dumitru @RazvanDuu]

"MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers" has been accepted to NeurIPS 2026!!

I had the privilege to be co-first author on the paper and to work with virtually all of the frontier labs to enable them to report numbers on our benchmark!

[MohammadHossein Rezaei @mhrezaeics]

Reward Hacking in Rubric-Based RL got accepted to NeurIPS!

If you're working on rubrics, take a look at some of our recent work:

[Daniel Zhang @danielyuez]

Who grades the graders? Verifier design for Computer Use Agents (CUA) is a notoriously hard but critical problem. As agents tackle complex workflows, evaluating their output exposes a core tension: Programmatic verifiers are deterministic but highly brittle, often penalizing valid alternative solutions with a rigid zero. Conversely, VLM judges using visual rubrics can capture nuance and partial credit, but they struggle with unpredictability and can confidently over-credit work they cannot actually verify.

Preventing agents from exploiting these grading loopholes is a constant cat-and-mouse game. Finding the balance between strict correctness and the flexibility required for open-ended problem-solving is an active research line. It opens the door to new ideas, and we actively welcome thought partnership to help push this space forward.

Ultimately, Scale AI is focused on powering the highest quality CUA through rigorous evaluation systems.

Dive into the challenges of verifier design and how we are addressing them here: https://t.co/vv1iv9fWI9

[a16z @a16z]

astra waiting to get added to the model selector dropdown

[Reza (Rey) Sanayei @r_sanayei]

@thsottiaux Making your models better at estimating experiment completion times

[MohammadHossein Rezaei @mhrezaeics]

Launching https://t.co/jhIfXxZkmL:

The work of AI R&D has always belonged to humans. For the first time, though, it no longer seems certain that it always will.

Recursive self-improvement is within a line of sight. It may still be far, but it is close enough that we should start measuring it.

[MohammadHossein Rezaei @mhrezaeics]

Scale is hosting a Research Meetup in NYC on August 12th!

If you're around, come say hi and enjoy our office views👀

Register at https://t.co/ZndFhtWB25