Victor Barres

AI Researcher

[Victor Barres @VBarres86633]

Requirements buried in ~1,700 files ✅
A client who hasn't told you everything ✅
An API that might be subtly broken ✅
A model menu and a cost budget ✅

Hyper-τ-bench: do agents have what it takes to build agents?

Best: 23.9%. An engineer: 82.2%.

https://t.co/4fchJsXhTq

[Jennifer Wang @jennjwang]

Jobs are modeled as bundles of tasks. But as jobs change, how do we keep those task lists up to date?

Workers often experience new responsibilities and tools before they reach their job descriptions.

We start by listening to them. 🧵

[Brendan (can/do) @BrendanFoody]

It was incredible to work with Stanford, the O*NET Resource Center, and the National Bureau of Economic Research to build TaskNet.

As AI radically changes jobs in the economy, we need a better taxonomy to evaluate what it can do.

[Edward Hu @edwardjhu]

I will be live on MTS at 1:15pm PT to talk about Mercor Research, AI for economically valuable work, model training for enterprise, and more.

[Govind K @t2govind]

The universe is abundant & knowledge has no bounds. We’re just entering a renaissance of discovery. There is still so much worth building.

Watch our GitHub HQ session on training agents for the real world:
https://t.co/rTq1LxhU7Y

[Greg Brockman @gdb]

Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory).

AI starting to help us improve our understanding of the universe! https://t.co/VkfacopR2R

[Victor Barres @VBarres86633]

I completely agree. One thing that I keep going back to is this: there are always incentives behind a measure. The solution that science came up with is not neutrality: this does not exist. It is method that is public and that can be debated.

[Han Fang @Han_Fang_]

Was talking to @karthikabinav about something that's been bugging us: how many evals are just trying to match Twitter vibes?

Before a benchmark even exists, people already have a ranking in mind, and it's mostly Twitter consensus. So when a result looks off, the instinct is to doubt the eval, not the consensus. Benchmarks that confirm the narrative get trusted and shared. The ones that challenge it just disappear.

And a lot of that “consensus” comes from people who've only really used a handful of models. Then a new benchmark validates it. Basically an echo chamber, and it gets even worse when influencers start stirring the pot.

Not sure how much it's distorting how we evaluate models, but I doubt it's small.

[Victor Barres @VBarres86633]

Great evening talking about training and evaluating agents in real life. My message in a nutshell: Environments give you a number. Whether this number is a reward or a measurement makes a huge difference in how you should judge its quality. Use what’s fit for your purpose!

[Sergio Paniego @SergioPaniego]

we had a lot of fun last night at @github HQ in San Francisco talking about training agents

🎥 recording coming soon

special thanks to @t2govind, @VBarres86633, @nblintao and @xdotli!

[Victor Barres @VBarres86633]

🪪 Getting a name right over the phone is a small joint effort:

🔤 you spell it
🔁 they read it back
🔧 you fix what slipped

We do it without noticing. Agents do some of it, but when they check a wrong value, only 24–37% get repaired.

Really fun one to work on with @sohmray!

[Sierra @SierraPlatform]

Voice agents can handle increasingly complex calls, but often fail on the simple details.

τ-Entity is a new benchmark measuring whether an agent can correctly capture details like names, phone numbers and email addresses, recognize when it might be wrong, and repair the mistake.

Read more: https://t.co/m2zYPzemHW

[Victor Barres @VBarres86633]

🌍 A language is much more than a translation: honorifics, forms of address, grammatical gender, how people take turns.

💬 Task completion, interaction and generation don't line up. The most responsive agent isn't the one that sounds most native.

🙌 Congrats @sohmray and team!

[Sierra @SierraPlatform]

How well do voice agents work across languages — and how natural do they sound?

Today we’re introducing 𝜏-i18n (pronounced tau-internationalization), which extends 𝜏-Voice to five more languages: Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin. We also included two new measures for voice quality beyond task completion: interaction, and generation quality.

Read more: https://t.co/7OF7vXGHKg

[Ben Burtenshaw @ben_burtenshaw]

So proud to share that @huggingface is giving OpenEnv to the AAIF and Linux Foundation so that it can be neutrally governed and shared by all.

This is a pivotal moment for RL, agents, and the whole field, and making agents’ learning environments open source is only getting more important by the day.

We believe, now more than ever, that democratizing AI requires a fully open stack. Reinforcement Learning environments are a fundamental part of that stack because they teach agents the capabilities they will use in the real world.

The committee behind OpenEnv is proud to bring AAIF into Reinforcement Learning, and further down the stack, by sharing what we’ve built and used at; Hugging Face, Microsoft, Reflection, Meta-PyTorch, Fleet, Mercor, RadixArk, Nebius, Modal, Prime Intellect, Unsloth, and more.

[Mercor @mercor]

Mercor Research mapped AI benchmarks to actual US jobs.

Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark.

Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽 𝗼𝗳 𝘁𝗵𝗲 𝗨𝗦 𝗘𝗰𝗼𝗻𝗼𝗺𝘆, an interactive breakdown of which economic tasks are covered by AI benchmarks.

Benchmarks are how we measure AI’s progress and anticipate its impact on real work. They’re crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time.

The Benchmark Map is our first step toward building a resource that makes it easier to answer: 𝘋𝘰𝘦𝘴 𝘵𝘩𝘪𝘴 𝘈𝘐 𝘮𝘰𝘥𝘦𝘭 𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘮𝘺 𝘰𝘤𝘤𝘶𝘱𝘢𝘵𝘪𝘰𝘯?

Explore it here: https://t.co/cDmR4S156o

[aden barton @aden_barton]

Looking forward to joining MTS later today to discuss the Mercor Econ team's recent work!

[Victor Barres @VBarres86633]

Some overdue news 🎉 I've joined @mercor as a founding member of the research team!

Agents are now all around us, and they're transforming how we work. Measuring that transformation well is how we understand it, and how we help shape it.

[Mercor @mercor]

Introducing the Mercor Research team.

We’re building a world-class research organization, including @edwardjhu (first author of LoRA), @MinyangTian1 (SciCode, CritPt), Victor Barres (τ²-bench, τ-voice), @hrdkbhatnagar (PostTrainBench), @bertievidgen (APEX-Agents) and many others.

As models get better, the questions get harder:

→ How do you measure a model on a job that AI itself is changing?
→ What data helps when a model fails at real work?
→ How much can a small amount of expert data improve a model?

Mercor Research covers the full AI flywheel. APEX shows where a model struggles. Experts help us understand why and create data for those weak points. We post-train the model and measure it again.

Learn more about our research mission: https://t.co/Hb4SnSnvy0

[Victor Barres @VBarres86633]

@BenShi34 Amazing work!

[aden barton @aden_barton]

New research: we hired 12 CPAs to complete four realistic accounting tasks, then had AI models from the past few years complete the same work.

As of May 2025, accountants outperformed AI. Now they lose out to today's frontier models, which are near perfect on these tasks. 🧵

[Victor Barres @VBarres86633]

Truly impressive results on tau-voice from GPT-Live-1.

At this pace it'll soon take longer to build a benchmark than for models to climb it...

[Ben Shi @BenShi34]

Thanks for reposting!

Got to benchmark astra day 1 and it’s pretty good…

[Daniel Rupawalla @danielrupawalla]

TAU-bench was ahead of its time. had issues b/c of how unstable LLMs were but agent interaction is crucial to making good evals for knowledge work. models need to be able to interact with "people" well. i think we'll definitely see a LOT more RL data / evals focused on this.

[Victor Barres @VBarres86633]

https://t.co/bVmDwKl85h

[Victor Barres @VBarres86633]

Amazing to see TauVoice being part of @ArtificialAnlys new speech-to-speech index! With tau-banking as a core part of their Intelligence index it’s 2-for-2 for the latest tau3-bench!
The whole research team will be at @icmlconf, looking forward to great conversations!

[Artificial Analysis @ArtificialAnlys]

Announcing the Artificial Analysis Speech to Speech Index, our new synthesis metric for native Speech to Speech model quality, comprising of Big Bench Audio, Full Duplex Bench, and 𝜏-Voice

The index provides a single measure of how well native Speech to Speech models perform, assessing Speech Reasoning (Big Bench Audio), Conversational Dynamics (Full Duplex Bench subset), and Agentic Performance (𝜏-Voice). Weighting is equal across all three datasets, and models must have valid results for all three to be included.

Key takeaways
➤ Model performance: @OpenAI GPT-Realtime-2 (High) leads at 77.2%, followed by @xAI Grok Voice Think Fast 1.0 at 75.7%, GPT-Realtime-1.5 at 72.0%, and @GoogleAI Gemini 3.1 Flash Live Preview (High) at 69.5%. Conversational Dynamics and Agentic Performance are key differentiators of frontier models, with GPT-Realtime-2 leading in Conversational Dynamics, and Grok Voice Think Fast 1.0 leading in Agentic Performance.
➤ Speed: Deepslate Opal is the fastest model in the index with a TTFA of 0.44s, followed by GPT-Realtime-1.5 at 0.82s and Grok Voice Think Fast 1.0 at 1.25s. GPT-Realtime-2 (High) records 2.33s, with Gemini 3.1 Flash Live Preview (High) recording 2.98s.
➤ Cost: Gemini 3.1 Flash Live Preview (Minimal) is the lowest cost model in the index at $1.50, then Gemini 3.1 Flash Live Preview (High) at $1.75, Grok Voice Think Fast 1.0 at $3.00, GPT-Realtime-2 (High) at $4.14.
➤ Datasets incorporated: Big Bench Audio - 1,000 reasoning questions across Formal Fallacies, Navigate, Object Counting, and Web of Lies; Full Duplex Bench - pause handling, turn taking, interruption and backchannel handling; 𝜏-Voice - end-to-end customer service task completion across Airline, Retail, and Telecom situations.

As always, we will continue to iterate on these benchmarks and plan to add more models.

More details below ⬇️