Victor Barres

AI Researcher, Tau-Bench guy

[Victor Barres @victorbarres_]

Can AI do your job? Benchmark progress is often read as the answer. But benchmarks are made of tasks, and a job is done with others, who hold information, make decisions and depend on you.

We find only 16% of benchmarks measure the organizational difficulties this creates.

[Victor Barres @victorbarres_]

I guess building a benchmark-of-benchmarks should be a task in a benchmark! Great to see that the initial omission of Tau-Bench has been corrected :)

[Jason Wei @_jasonwei]

Turns out my initial version missed a lot of evals. Claude searched harder and found 36% more mentions of benchmarks: https://t.co/ONtgbBuwEs

It still misses blog posts with fancy UI elements, long PDFs, etc. Guess we need a benchmark for creating benchmark benchmark...

[Victor Barres @victorbarres_]

Top engineers don’t just rely on what they already know. They work in a world that keeps moving, constantly pulling in new knowledge, tools and practices.

That’s what OWE-Bench starts to measure. Great work @jake_cole1, @bertievidgen and the whole @mercor team!

[Mercor @mercor]

Agents can write code. Can they finish real engineering work?

Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.

Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.

OWE-bench top 3 leaderboard:

1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.

Explore the benchmark website: https://t.co/oT0qna1dO0

[MTS @MTSlive]

Mercor co-CEO @BrendanFoody says frontier AI models have jumped from 23% to 80% on the APEX-Agents benchmark since February, and predicts AI will automate over half of all digital tasks within the next few years:

"APEX-Agents we launched in February of this year when frontier models were at 23%, and now they're at roughly 80%."

"RL's becoming so powerful that once we have an eval for something, the models can saturate it."

"The barrier to applying AI to automate everything in the economy is how do we just have some evaluation or some way to measure it so that we can improve model performance against it."

"I more broadly am probably AGI long, but ASI short. I think that we'll automate over 50% of the tasks definitely in the digital world in the next few years."

"There's this long tail of tasks that require a lot of tacit knowledge or are intrinsically human, where people wanna have human-to-human conversations, or everything in the physical world, where it just feels like there's going to be a much longer road."
@mercor

[Brendan (can/do) @BrendanFoody]

We raised our $750 million Series D at a $20 billion valuation from @benchmark, @generalcatalyst, @nvidia, @salesforce, and @zoom.

@mercor scaled from $1M to $3B in run-rate revenue in 27 months, the fastest in history to $3B.

For 2 years, we've organized human intelligence to build evals and training data for every frontier lab.

The labs are the leading indicator for AI startups, then the Fortune 500.

Our growth has accelerated over the past 6 months as enterprises start evaluating and training their own models. For many, owning their intelligence is existential.

[Victor Barres @victorbarres_]

Attention is growing fast, from 14% of benchmarks in 2023 to 33% in 2026. AI agents are starting to work inside organizations, and we hope this taxonomy helps the field measure whether they can play well with others.

📝 https://t.co/iXJ3vxQql6
📄 https://t.co/byuj76cRol

[Victor Barres @victorbarres_]

Can AI do your job? Benchmark progress is often read as the answer. But benchmarks are made of tasks, and a job is done with others, who hold information, make decisions and depend on you.

We find only 16% of benchmarks measure the organizational difficulties this creates.

[Victor Barres @victorbarres_]

We then applied it to prominent AI benchmarks. Fewer than a quarter claim any organizational dimension, and when one appears it usually takes a narrow form, such as authority written as a policy in the prompt.

[Sergio Paniego @SergioPaniego]

weekend watch: 2h on post-training agents with open models

4 talks + a panel + Q&A from Microsoft, Mercor, RadixArk and BenchFlow

https://t.co/fT0ub9XnUZ

[Victor Barres @victorbarres_]

Truly incredible to see the progress full-duplex voice models have made on Tau-voice in less than a year!

[Avartha @AvarthaAI]

Introducing Avartha.

We're building the infrastructure to bring frontier-grade intelligence to real-time applications. Our first release, Ava Zero, is a voice agent entirely running open models using our inference infrastructure and ranks #1 on Sierra's tau-voice benchmark with highest task completion rate and generation quality and just costs $2.5 per hour.

Open models are now at the frontier of voice AI.

[Soham Ray @sohmray]

Congrats to the @AvarthaAI team for topping the Tau Voice 2.0 leaderboard! This means they win against the big labs across 5 languages, on not just task completion but also speech fidelity and multilingual naturalness: tau is more robust now! And all using open weight models.

[Soham Ray @sohmray]

Phonic takes gold on Tau Voice English!

Its the first time that a model not produced by one of the big labs (openai/xai/gemini) has come out on top. Congrats to @NikhilKMurthy, @moinnadeem and the Phonic team, this is huge!

Phonic-1: https://t.co/uNroPiEw12
Leaderboard: https://t.co/jmAweQNzDx

[Jennifer Wang @jennjwang]

Jobs are modeled as bundles of tasks. But as jobs change, how do we keep those task lists up to date?

Workers often experience new responsibilities and tools before they reach their job descriptions.

We start by listening to them. 🧵

[Brendan (can/do) @BrendanFoody]

It was incredible to work with Stanford, the O*NET Resource Center, and the National Bureau of Economic Research to build TaskNet.

As AI radically changes jobs in the economy, we need a better taxonomy to evaluate what it can do.

[Edward Hu @edwardjhu]

I will be live on MTS at 1:15pm PT to talk about Mercor Research, AI for economically valuable work, model training for enterprise, and more.

[Govind K @t2govind]

The universe is abundant & knowledge has no bounds. We’re just entering a renaissance of discovery. There is still so much worth building.

Watch our GitHub HQ session on training agents for the real world:
https://t.co/rTq1LxhU7Y

[Greg Brockman @gdb]

Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory).

AI starting to help us improve our understanding of the universe! https://t.co/VkfacopR2R

[Victor Barres @victorbarres_]

I completely agree. One thing that I keep going back to is this: there are always incentives behind a measure. The solution that science came up with is not neutrality: this does not exist. It is method that is public and that can be debated.

[Han Fang @Han_Fang_]

Was talking to @karthikabinav about something that's been bugging us: how many evals are just trying to match Twitter vibes?

Before a benchmark even exists, people already have a ranking in mind, and it's mostly Twitter consensus. So when a result looks off, the instinct is to doubt the eval, not the consensus. Benchmarks that confirm the narrative get trusted and shared. The ones that challenge it just disappear.

And a lot of that “consensus” comes from people who've only really used a handful of models. Then a new benchmark validates it. Basically an echo chamber, and it gets even worse when influencers start stirring the pot.

Not sure how much it's distorting how we evaluate models, but I doubt it's small.