Manasi Sharma @ ICLR 2026

research engineer @scale_AI, working on reasoning for frontier models, agents, rl
| prev @stanford, @StanfordAILab, @mitll, @Columbia

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

🚀New @scale_AI paper: 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗥𝘂𝗯𝗿𝗶𝗰𝘀, a benchmark for evaluating Deep Research (DR) agents. Even top agents like Gemini & OpenAI DR achieve <𝟲𝟴% 𝗿𝘂𝗯𝗿𝗶𝗰 𝗰𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲. We built 𝟮.𝟱𝗞+ expert rubrics with 𝟮.𝟴𝗞+ hrs of human labor to measure why.

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

also excited to chat more broadly about agents (deep research agents, multimodal computer-use agents, etc.), evals & post-training

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

i'll be attending @iclr_conf in Rio this week! 🇧🇷

excited to be presenting ResearchRubrics at the 4/23 Thurs afternoon poster session, and a poster on rubric-benchmark sensitivity at the Agents in the Wild workshop

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

excited to share that ResearchRubrics has been accepted to ICLR 2026!

see you in Rio☀️

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

@mariyaivasileva @WiMLworkshop hi @WiMLworkshop, I just ran into this issue as well

[Epoch AI @EpochAIResearch]

Can AI do real digital work? We reviewed three benchmarks to find out: RLI, GDPval, and APEX-Agents.

Our take: progress here will indicate substantial economic value, but tasks are too self-contained to tell us about wholesale automation.

Thread for more:

[Spreadsheet Arena @sheetarena]

TL;DR: Spreadsheet generation is multi-dimensional.

Human preference data captures what users actually value, but different dimensions matter across domains, and some signals surface more clearly than others.

Spreadsheet Arena gives us a powerful foundation for evaluation, and a new lens for improving post-training.

Start a battle at https://t.co/MHGdWX6xxm

Read the paper at https://t.co/6GX4ify8CS

@srkundurthy @claranahhh @Zachkirshner @calvincbzhang @ManasiSharma_ @jhnling

[Calvin Zhang @calvincbzhang]

New paper from @scale_AI & @MeridianAgent: SpreadsheetArena 📄

We evaluated 16 LLMs on end-to-end spreadsheet generation via 4,300+ blind pairwise votes.

Crucially, we move beyond scalar Elo ratings to decompose the latent preference signal into functional, structural, and stylistic components. 🧵

[Scale AI @scale_AI]

🎙️ In our latest Chain of Thought episode we unpack ResearchRubrics, our benchmark for evaluating deep research agent performance.

We explore what meaningful agent evaluation looks like, where today’s agents still fall short, and why clearer evaluation frameworks are critical as agent use accelerates.

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

🎙️check out the full episode and paper: https://t.co/9AuTRbGzuP
https://t.co/VQurPwFeqY

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

i also recently sat down with @calvincbzhang and
@Bckenstler on the CoT podcast at @scale_AI to discuss Deep Research agents & the ResearchRubrics paper in particular 🧵

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

we dug into (1) how to assess deep research queries & reports, (2) where today’s agents still fall short, and (3) why clearer evaluation frameworks are critical as real-world agent use accelerates. :)

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

excited to share that ResearchRubrics has been accepted to ICLR 2026!

see you in Rio☀️

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

🚀New @scale_AI paper: 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗥𝘂��𝗿𝗶𝗰𝘀, a benchmark for evaluating Deep Research (DR) agents. Even top agents like Gemini & OpenAI DR achieve <𝟲𝟴% 𝗿𝘂𝗯𝗿𝗶𝗰 𝗰𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲. We built 𝟮.𝟱𝗞+ expert rubrics with 𝟮.𝟴𝗞+ hrs of human labor to measure why.

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

chatted with other folks at NeurIPS last week working on browser / CUA post-training, and it seems like people are shifting toward vision-only agents, which appear to generalize better

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

@LiuZuxin would love to chat!

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

@996roma would love to chat!

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

@DrJimFan would love to meet!

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

i'll be attending NeurIPS in San Diego next week!

excited to chat about agents in general (deep research agents, multimodal browser agents for long-horizon tasks, etc.), evals for open-ended + economically valuable tasks, and RL / post-training

[Rohan Paul @rohanpaul_ai]

ResearchRubrics is a benchmark that tests how well AI deep research agents handle open ended web questions.

The key finding is that these agents still fall short of expert expectations once checked against rubrics.

The benchmark has 101 prompts from 9 domains, each with a human written checklist about content, reasoning, synthesis, citations, and clarity.

Each checklist item has a weight, some are mandatory, some optional, and a judge marks them as fully met, partly met, or not met.

Tasks get 3 tags describing topic breadth, reasoning depth, and how open ended the goal is.

Top agents stay below 68% compliance because they miss implicit context and fail to connect information across documents, even though language model judges and humans agree when rubrics use concrete examples.

----

Paper – arxiv. org/abs/2511.07685

Paper Title: "ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents"

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

thanks @_akhaliq for featuring our new work!

💻we also recently released the code to evaluate the benchmark: https://t.co/POPYsW0SGR

[AK @_akhaliq]

ResearchRubrics

A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

@atasteoff thanks for sharing! we added a few more details and the link to the dataset here: https://t.co/bjB0AkaB9x

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

🚀New @scale_AI paper: 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗥𝘂��𝗿𝗶𝗰𝘀, a benchmark for evaluating Deep Research (DR) agents. Even top agents like Gemini & OpenAI DR achieve <𝟲𝟴% 𝗿𝘂𝗯𝗿𝗶𝗰 𝗰𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲. We built 𝟮.𝟱𝗞+ expert rubrics with 𝟮.𝟴𝗞+ hrs of human labor to measure why.

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

This work was led by our amazing team at @scale_AI & other institutions: @calvincbzhang, @cbandieth, @clintonjwang, @ankitaich30, @hnghiem_ai, @TahseenRab74917, @yehtetCS, @Rkwmdkrhrh, @sumanabasu21, @Iishiiyaa, @DenisPeskoff, @marcos_scale, @SeanHendryx, @Bckenstler, @vbingliu

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

🚀New @scale_AI paper: 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗥𝘂𝗯𝗿𝗶𝗰𝘀, a benchmark for evaluating Deep Research (DR) agents. Even top agents like Gemini & OpenAI DR achieve <𝟲𝟴% 𝗿𝘂𝗯𝗿𝗶𝗰 𝗰𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲. We built 𝟮.𝟱𝗞+ expert rubrics with 𝟮.𝟴𝗞+ hrs of human labor to measure why.

[Manasi Sharma @ ICLR 2026 @ManasiSharma_]

ResearchRubrics is publicly available 🎉
📄 100+ realistic prompts
🧮 2.5K expert rubrics

📄 arXiv: https://t.co/AEwQL6FLNs
🌐 Website: https://t.co/dtYIR9qmm7
🤗 Dataset: https://t.co/EmwtIHI2zh