Sharing an early result: Sierra verified Cortex at 73.2% on τ³-benchmark.
We build algorithmic complements to LLMs. We don’t fight what models do well. We fill in the rest.
https://t.co/MmTS0HVc3E
Congrats to the @AvarthaAI team for topping the Tau Voice 2.0 leaderboard! This means they win against the big labs across 5 languages, on not just task completion but also speech fidelity and multilingual naturalness: tau is more robust now! And all using open weight models.
Introducing Avartha.
We're building the infrastructure to bring frontier-grade intelligence to real-time applications. Our first release, Ava Zero, is a voice agent entirely running open models using our inference infrastructure and ranks #1 on Sierra's tau-voice benchmark with highest task completion rate and generation quality and just costs $2.5 per hour.
Open models are now at the frontier of voice AI.
Phonic takes gold on Tau Voice English!
Its the first time that a model not produced by one of the big labs (openai/xai/gemini) has come out on top. Congrats to @NikhilKMurthy, @moinnadeem and the Phonic team, this is huge!
Phonic-1: https://t.co/uNroPiEw12
Leaderboard: https://t.co/jmAweQNzDx
Hinglish! but you can adjust codeswitching percentage if you want. Within Hindi we try out 2 personas, but you can create your own, one with high codeswitching hinglish vs urdu inflected less codeswitching hindi. just 2 of many types of hindi speakers ofc, counting on the community to expand this!
Congrats to the @GoogleDeepMind team on an awesome model; 3.8 is a step change in gemini voice performance.
And shoutout to movie-star-in-the-making @valeriawu_
Explore the latest advances in native audio models and the challenges of building better real-time voice experiences with @GoogleDeepMind + @SierraPlatform.
Learn about conversational latency, multilingual code-switching, agentic voice tasks, and more →https://t.co/wrzm4SeD09
Community has always been at the heart of Tau. To celebrate the exciting work happening across our ecosystem, we’re highlighting some contributions here https://t.co/HWbQeP80oy.
If you have built something with Tau, please make a PR and we'll take a look. 🚀
Introducing τ-i18n (pronounced internationalization), extending τ-Voice to more languages and raising the bar for conversation quality.
At Sierra, our work with customers serving non-English speakers reinforces a clear standard: voice agents need to complete the task and deliver a conversation that feels native. Clay Bavor calls this “unreasonable localization”—getting even the small details right. Every language has its own nuances, from noun gender in Hindi to tones in Mandarin. The meaning may be clear, but those mistakes keep a conversation from feeling native.
In τ-Voice, accents caused the largest average drop in task completion among the factors we tested. That motivated us to expand beyond English. τ-i18n brings the same airline, retail and telecom tasks to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin. Tasks, policies, tools, and success criteria stay fixed, allowing comparisons across languages. We also examine how database localization affects performance.
Working with native speakers, we built callers and evaluation rules that reflect local conventions—from name order and code-switching to forms of address and honorifics. These conventions affect outcomes: an agent can hear a family-first name correctly and still fail authentication by putting it into the wrong fields.
Across 4,500 calls, task completion in Spanish, Portuguese, and Hindi stays within 3.2 percentage points of English. Korean and Mandarin trail by 14.7 and 8.4 points. But completing the task is only part of a good call. Roughly 30–43% of evaluated utterances fail naturalness checks. A caller can get what they need and still have a bad experience. Our evaluations need to capture both.
These five additional languages are a starting point. We’ll soon open source our codebase and LanguageFactory, our workflow for building localized user simulators and evaluation criteria, so native speakers can contribute new language packs. If your language is missing, we’d love your help bringing it to the benchmark: build representative callers, review conversations, and define the conventions agents should meet—so voice agents can better serve speakers around the world.
We’re also tackling the hardest part separately with τ-Entity–how reliably do voice agents collect names and numbers, decide when to verify, and correct mistakes?
Built with friends at @SierraPlatform (Edgard dos Santos Paiva Neto, Ruben Valenzuela, Keshav Dhandhania, Vijay Iyengar, Ajeet Grewal), @mercor (Victor Barres), and @Princeton (Karthik Narasimhan).
Papers: τ-i18n https://t.co/rAzGsUi6MY, τ-Entity https://t.co/phTXDclidS
Submission instructions, codebase, and updated leaderboard coming soon.
Introducing τ-Entity, a benchmark for how voice agents collect, verify, and correct the details callers give them.
In τ-Voice, capturing names and other identifying information was a recurring bottleneck: get a detail wrong, and authentication fails before the agent can help. τ-Entity breaks that exchange down. Can an agent collect one exact value, recognize when it needs to check what it heard, and repair a mistake?
We test 200 tasks across ten entity types, including names, phone numbers, addresses, and emails. Each call asks for a single field, with a cooperative caller who spells accurately when asked. Most calls last less than two minutes. And even here, reliability is far from solved.
When agents choose their own verification strategy, only 14–41% of tasks succeed across all tested conditions. A brute-force approach—prescribing spelling, read-back, and confirmation—raises that to 37–54%, but makes calls longer. The challenge is finding the balance: knowing when to trust a capture and when to spend more effort checking it.
Across systems, agents recognize hard and unfamiliar entities and verify them more carefully. But they do not increase verification for the caller voices they struggle with most, and their response to noise is inconsistent. And verification often fails to produce a repair: just 24–37% of wrong captures that agents verify end up correct. Agents need to get better at both deciding when to check and using the answer to fix the record.
We’re introducing τ-Entity alongside τ-i18n, and plan to extend these entity-collection tasks across languages with the upcoming codebase release. Together, they help us measure whether voice agents can both handle the details accurately and deliver a conversation that feels native.
Thanks to Victor Barres at @mercor for collaborating on this.
Papers: τ-Entity https://t.co/phTXDclidS, τ-i18n https://t.co/rAzGsUi6MY
Codebase and multilingual extension coming soon!
The way you manage agents on the Sierra platform has evolved from signing into Sierra to engaging with Ghostwriter in Slack. The first time Ghostwriter proactively @-mentions you when it notices an issue or suggests running an A/B test you hadn't thought of, it changes your conception of how enterprise software should work. Really proud of the team that built this https://t.co/2uQzTJK18T
This makes total sense. The benchmarking cycle is insanely fast now: new benchmark drops -> data companies crank out training data -> benchmark saturated in months. Real progress requires specialization—the high-value difficulty is in the nuances generalized data misses. I'm interested to see how this evolves, and grateful for the partnership with @FireworksAI_!
Today we're launching the Specialized Intelligence Index (SII): one destination for real-work benchmarks across industries, built by the teams that use them every day.
Hear from Fireworks co-founder @the_bunny_chen on the importance of specialized benchmarks:
Say hello (literally) to Gemini 3.8 Live and 3.8 Live Extended Thinking, our new SOTA live audio models, available with frontier price + performance.
3.8 Live supports 97 languages (can seamlessly switch), async tool calls, and more!
How does GPT‑Live‑1 perform on the benchmarks that matter most for production voice agents?
We focused on task completion, back-and-forth conversation and turn-taking, how quickly the model responds, and tool use.
Here are the results:
Awesome stuff from Sierra favorites Ben Shi, Keshav Dhandhania, Victor Barres and Karthik Narasimhan.
Hypertau lets you benchmark coding agents, their harnesses, and their self improving capabilities.
How do coding agents decide to spend their budget?
How do they decide which models to pick for sub agents?
How do they decide when to run simulations, and how many?
How do they resolve ambiguities? Reach out to the client? Or look for answers in simulated slack threads, websites, snapshots or knowledge base artifacts?
Learned so much from this, and we see these gaps everyday at Sierra. First of many tau releases, stay tuned!
Today we’re releasing hyper-𝜏-bench, a new evaluation that measures how good coding agents are at building agents.
𝜏-bench asked whether models could be good agents. Hyper-𝜏-bench asks whether they can build them.
https://t.co/aaLKgo3IGQ