Can AI agents keep up with humans on real-time interactive tasks?
On Realtime-Bench:
๐ฎ Games: humans 72%, best model 32%
๐ป Complex web tasks: humans 7%, best model 52%
Humans react faster. Models are better at working out complex rules.
https://t.co/YPvXGpjD0M
With Claude Fable 5.2 coming out, discussions are already expanding to broader applications like robotics simulation and 3D. Despite improvements over 5.1, it still struggles with real-time agentic tasks. Excited to see what comes next!
AI agents are increasingly used in team settings. But most LLMs are trained with a "single-user" assumption: <system> <user> <assistant> <user>. What if multiple users are interacting with the same agent? Check out our new paper: ๐ ๐๐น๐๐ถ-๐จ๐๐ฒ๐ฟ LLM ๐๐ด๐ฒ๐ป๐๐.
๐๐๐
In team settings, that's increasingly common. One agent might serve a CEO, engineers, and outside collaborators, each with different roles, goals, and permissions
In this new paper, we present the first systematic study of this setting. We test 19 frontier and open models on three key capabilities:
โข ๐ฏ ๐๐ป๐๐๐ฟ๐๐ฐ๐๐ถ๐ผ๐ป ๐ณ๐ผ๐น๐น๐ผ๐๐ถ๐ป๐ด ๐๐ป๐ฑ๐ฒ๐ฟ ๐ฐ๐ผ๐ป๐ณ๐น๐ถ๐ฐ๐: Whose instruction wins when users disagree?
โข ๐ ๐๐ฟ๐ผ๐๐-๐๐๐ฒ๐ฟ ๐ฎ๐ฐ๐ฐ๐ฒ๐๐ ๐ฐ๐ผ๐ป๐๐ฟ๐ผ๐น: Can the agent protect restricted data (e.g., a salary database) while still serving authorized users?
โข ๐ค ๐ ๐๐น๐๐ถ-๐๐๐ฒ๐ฟ ๐ฐ๐ผ๐ผ๐ฟ๐ฑ๐ถ๐ป๐ฎ๐๐ถ๐ผ๐ป: Can it efficiently gather input from many people, e.g., to schedule a meeting?
What we found:
โข ๐๐ผ๐ป๐ณ๐น๐ถ๐ฐ๐ ๐ต๐๐ฟ๐๐ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ. Claude-Haiku-4.5's accuracy drops from 0.86 to 0.62 when users' instructions conflict. Choosing the right instruction and executing it are separate skills: Qwen3-4B scores 83.8 on selection but only 57.9 on execution.
โข ๐ฃ๐ฟ๐ถ๐๐ฎ๐ฐ๐ ๐ฎ๐ป๐ฑ ๐๐๐ถ๐น๐ถ๐๐ ๐ฝ๐๐น๐น ๐ฎ๐ด๐ฎ๐ถ๐ป๐๐ ๐ฒ๐ฎ๐ฐ๐ต ๐ผ๐๐ต๐ฒ๐ฟ. GPT-5.2 achieves perfect privacy but serves authorized users only 61.2% of the time. Claude-Sonnet-4.5 does the reverse: 97.5 utility, 77.3 privacy.
โข ๐ฃ๐ฟ๐ถ๐๐ฎ๐ฐ๐ ๐ฒ๐ฟ๐ผ๐ฑ๐ฒ๐ ๐ผ๐๐ฒ๐ฟ ๐๐ถ๐บ๐ฒ. Leakage grows over multi-turn conversations. Claude-3.5-Haiku falls from >95% to <75% privacy within four rounds, so passing an isolated access check doesn't mean the agent stays safe.
โข ๐๐ผ๐ผ๐ฟ๐ฑ๐ถ๐ป๐ฎ๐๐ถ๐ผ๐ป ๐ถ๐ ๐ต๐ฎ๐ฟ๐ฑ. Even the best model, Gemini-3-Pro, schedules a meeting successfully only 64.8% of the time. The bottleneck isn't reasoning. It's knowing when to ask for missing information.
The top models overall are Gemini-3-Pro (85.6), Claude-Sonnet-4.5 (82.6), and Gemini-3-Flash (82.0), and none of them aces all three capabilities.
As agents become co-workers rather than personal assistants, the field needs native multi-user message formats, long-horizon safety benchmarks, and principled ways to resolve conflicts between users.
Grateful to my wonderful co-authors: @jiaxin_pei๏ผ @shenzhe_zhu, @_Hao_Zhu, Josรฉ Ramรณn Enrรญquez, @diwang_kaust , @alex_pentland , and @bakkermichiel .
Paper link in the first comment
This is f*cking insane. Former OpenAI CTO, Mira Murati:
"The models we work with today are turn-based. You talk, they talk, then they go off and think, and while they're thinking they're deaf and blind.
The next ones never stop listening. They take in audio, text and video all the time and answer in 20-millisecond chunks."
In just 2 minutes, she shows why the chatbot you type into is about to look like a fax machine.
Prompt Box โ Chatbot โ Agents โ AI That Works Next To You In Real Time
A turn-based model waits for you to finish typing, then goes quiet while it thinks.
An interaction model sees and hears what you're doing, and can cut in the moment you start going the wrong way.
Anyone can type a prompt. Almost nobody is ready to work with an AI that never stops paying attention
Watch the full 2 minutes. This is worth your time.
Bookmark it, like it, save it, I don't care... this is the best thing you'll see on X today.
โฑ๏ธ Introducing Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
The best model, GPT-6-Astra, solved only 36% of tasks reliably.
Jev achieves sub-1s latency, but its performance remains at 8%.
๐ Read our full blog: https://t.co/G9dzd1moWh
Realtime-Bench โฑ๏ธ
Key findings:
- Best model (GPT-6-Astra): 36%.
- Fastest model (Jev, answers in under 1s): 8%.
- More reasoning helps only when the clock is paused.
- With a real clock, slow thinking costs the win.
- Even with time frozen, many tasks stay unsolved.
Play with it at https://t.co/Dhp2FUrmfa!
Great work led by @JingyuTang621@jiaxin_pei and team ๐
@kohjingyu Yes! Realtimeness is an important ability for models and our team recently released Realtime-Bench (https://t.co/eccYxuRItz) few days ago to test such capabilityies for both frontier VLMs and decision models (e.g., Jev). Check this out!
@PranjalAggarw16 Our team has some relevant work that might be interesting to you! here is the paper link: https://t.co/CTuxNQlEuE. leaderboard: https://t.co/Zsdvk6158k. code: https://t.co/BLs2mW5n7l
Excited to co-lead this project as one of the first to benchmark language agentsโ real-time capabilities. Stay tuned for our next efforts on this domain!
We are very excited to release Realtime-Bench, a well-curated benchmark that systematically test both native realtime models and general VLMs' capabilities to think and act in real-time! Amazing job led by @JingyuTang621, @xyyy6688 and @Zhehao_Zhang123!
From playing games to grabbing concert tickets, many tasks require real-time thinking and action.
Can frontier models solve real-time interactive problems? Check out our latest work, Realtime-Bench!
TL;DR: The best model completes only 36% of tasks reliably. ๐งต
As a summary, real-time interaction remains challenging to all frontier models. Future real-time agents need to track a changing world, adapt reasoning to the time available, and decide when to act or wait.