Drop 4/15: Announcing the Voice Arena Dictation Bench - the first open benchmark of AI dictation apps.
Not their APIs: the real apps. We played 761 recordings from 87 speakers into each of 10 apps on a Mac, then 5 LLM judges asked if each draft could be sent as-is.
Wispr Flow: 79%. Apple: 35% 🧵
🏆 A new leader on the Voice Arena Diarization + ASR board: AirCaps A5S Async v1 joins the 14 systems already scored and enters at rank #1.
To put this result in context, we evaluated all 15 systems on the same 139 held-out conversations with 2–8 speakers, counting overlapping speech throughout. cpWER measures how accurately a transcript captures words and assigns them to the right speakers.
AirCaps A5S Async v1 scores 23.89% cpWER overall and leads in both settings, with 24.91% cpWER in-person and 17.47% in online calls.
What's particularly interesting is where this lead comes from. AirCaps and Nemotron 3.5 Transcribe (#2) miss a similar share of words (9.29% vs. 8.32%), but AirCaps has a substantially lower substitution rate (10.88% vs. 18.43%) and fewer insertions (3.72% vs. 5.68%).
AirCaps maintains its lead when we resample the 139 conversations. Its overall cpWER showing a 95% confidence interval of 19.47%–28.58%.
Congratulations to the @tryaircaps team on the launch! 👏🏻
Drop 2/15
Are we actually measuring the things we ultimately want our voice AI models to achieve?
The best AI voices are becoming hard to tell apart.
From each other. Not from humans.
🚀 Introducing Turing TTS: an evaluation that measures how often human listeners mistake AI-generated speech for real human speech.
We tested leading TTS models against real human speech. Even the best model fooled listeners only ~10% of the time.
With Turing TTS, we hope to contribute to a broader discussion within the research community about how we measure progress in voice AI.
If one of our goals is to develop speech that is indistinguishable from human speech, how well do our current evaluation methods capture progress toward that goal?
We don’t claim that Turing TTS is the right way forward, or that human indistinguishability should be the goal of every TTS system.
Rather, we hope these findings encourage the community to collectively examine whether blind pairwise human evaluations of naturalness are sufficient, and what else we might need to measure.
Results are now live on @voicearena_ai.
Introducing Diarization Bench, @voicearena_ai's benchmark for who spoke when.
The metric we measure is diarization error rate, or DER. It measures how much of a conversation a model attaches to the wrong person. Every second it misses, invents, or gives to another speaker counts against it.
Here's what we did. 139 conversations recorded for this board, in rooms and on calls, with people interrupting each other throughout. 12 systems scored under a single goal: testing models in the real-world conditions they are actually deployed in. @nvidia's Nemotron 3 leads at 14.72%. 🧵
We’re going to be in Sydney next week talking about all the research we’ve done this year and dropping some very exciting new releases!
Drop by our booth D1 at ICC Sydney from the 28th of September and say 🙋♀️
Excited to share some great news. @voicearena_ai is the Diamond Sponsor of Interspeech 2026.
I’ve been attending Interspeech for the last 2 years. My journey of falling in love with research and solving bottlenecks in voice ai started here.
We’ll be sharing our latest evaluations, datasets and research with the community.
If you are going to be there, do come by!
📍 Booth D1, ICC Sydney
📅 28 September–1 October
Introducing Jarvis Bench v0.5, @voicearena_ai's conversational agent benchmark.
We've been obsessed with one question at VoiceArena: why do voice agent demos sound incredible, benchmarks say models are near-perfect, and yet you probably didn't have a single real conversation with a voice agent in the last 24 hours?
Here's what's different about the Jarvis Bench. Real humans have live conversations with voice agents. A second group of humans blind-votes pairwise on two questions: which sounded more human, which got the job done. And one of the "agents" on the leaderboard is a human.
Trying to get mileage without context is something people on the internet really must stop doing. Having known Megha for 15 years her choices are her own and out of respect.
Dear @theliverdoc, I think you have made a rather large assumption about a person you know nothing about, based on a clip.
You have no idea what I stand for, what I believe in, or the context of that conversation.
And there is nothing wrong with being seated on the floor. I was perfectly comfortable there and it was out of my own choice.
Sometimes respect has less to do with where someone is seated and more to do with how we choose to speak about people we don’t know.
Perhaps a little more context, and a little less assumption, would serve us all well.
Wishing you a low Resting Entropy
Meta's Muse rises on three Voice Arena STT boards.
US English: 6.87% → 5.31% WER, #18 → #9.
Vietnamese: 20.67% → 13.75%, #10 → #5.
Brazilian Portuguese: 10.71% → 7.65%, #13 → #5.
Three boards, three gains over Meta's own omniASR-LLM-7B. Same audio for every model within a language.
@AIatMeta
Hosted the first edition of Voice Research Club in SF yesterday with @rjrshr and @nishnik!
We’ll be hosting this monthly. The next edition is on October 8.
VRC is a volunteer-driven community. If you’d like to help shape future editions - from deciding which papers get presented to helping organize the sessions - DM any of us.
@voicearena_ai
Cartesia Sonic 3.5 is the #1 streaming TTS model on Voice Arena US English Leaderboard.
In the overall leaderboard (streaming + non-streaming) it jumped from rank #9 → #2, moving ahead of Grok TTS, ElevenLabs v3 & OpenAI's gpt-4o-mini-tts.
Sonic-3.5 is the latest TTS model from @cartesia . It supports 42 languages, with 500+ voices available out of the box. The model has been highly preferred among raters on @voicearena_ai .
Results backed by 11,110 blind, head-to-head listener votes
Introducing Human-1 by @JoshTalksLive - the first open full-duplex conversational speech model for Hindi.
Key result: natural conversational behaviour transfers to a new language when a duplex speech model is trained on sufficiently large-scale real dialogue.
Built on Moshi's full-duplex architecture, Human-1 listens and speaks simultaneously processing both speakers' audio and Hindi text in parallel.
Training data:
26,000 hours of real two-person conversations
14,695 speakers
stereo recordings
This allows the model to directly learn conversational dynamics most speech datasets lack:
• interruptions
• overlap
• backchannels
• turn-taking
Interestingly, the model converged in ~4,000 training steps, suggesting that the model quickly learned conversational patterns from the data and that much larger conversational datasets may unlock further capability gains.
Human evaluation:
85% rated as human-like interaction
Naturalness MOS 4.10 (human: 4.55)
66.9% tie with human speech
Human-1 is not a production system - it’s a research proof point toward much larger conversational speech corpora.
Paper, code, playground and demos:
https://t.co/vb8bTzdnqf
We’ve released one of the most rigorous large scale evaluations of speech recognition ever conducted for Indian languages. Check out the thread for more.
Introducing: Voice of India - a benchmark for speech recognition (STT/ASR) models
Built by AI4Bharat (IIT Madras) and Josh Talks, it measures how well AI models understand real-world Indian speech across languages, accents, code-switching, and noisy conditions.
This isn’t about blaming global labs.
The issue is simple: India specific evaluation infrastructure hasn’t existed.
You can’t improve what you can’t measure.
Voice of India is that measurement layer.
Drop 12/14: Models, products, impact - today something different, very different. Launching Sarvam Kaze, our foray into getting our models into the your hands with our devices - designed and built here in India!
Stress tested Sarvam AI, India's sovereign TTS model through a Large-Scale Indic TTS Evaluation of @SarvamAI's Bulbul V3 vs @elevenlabs's v3 Alpha and 2.5 Flash vs @cartesia's Sonic-3 with over 44,000+ votes from 1000+ humans.
More on this in the next few days with audio samples where each model excelled vs the others.
There’s a bigger layer to @SarvamAI story.
The application and services ecosystem that can be built on top of Sarvam is ENORMOUS. Their Indic speech, OCR, and language stack unlocks use cases across governance, education, finance, healthcare, and citizen services.
I still remember @AshwiniVaishnaw saying that India will lead at the application layer and this is exactly what that vision looks like: strong local AI infrastructure that others can build real-world services on.
Sarvam isn’t just a model company , it’s an enabler layer.