I wanted Granola-style notes without sending my calls anywhere, so I built onoats. It doesn't care which meeting app you use. It taps mic and speaker audio at the OS level, so Zoom, Meet, and Teams all work and no bot joins the call. 🧵 https://t.co/3bOCp1fXg3
Something like 350 million people regularly speak Hindi with lots of English words and phrases mixed in. "Hinglish" might be the most widely spoken language that, unless you live in India, you haven't heard of.
But if you do live in India, and you're building voice AI agents for use cases like customer support, the need for AI models that are very good at Hinglish is going to be very much on your mind.
Merve made an explainer video about the Sarvam models: LLM, TTS, STT, and decision models trained in 10 Indian languages, and uniquely good at mixed-language understanding, thinking, and speech.
The hard part of a multilingual voice agent isn’t translation, but instead how people switch languages mid-sentence and expect it to follow.
In this video, Merve explores how far you can get with @SarvamAI models across the whole Pipecat pipeline: hearing, thinking and speaking in Hinglish.
Sarvam models in Pipecat:
https://t.co/42tJ3uG4oN
Demo from this video:
https://t.co/YYKuObllGt
Big models that "do everything" are truly amazing. We live in extraordinary times. But agents that wrap big models are fundamentally hard to debug, observe, maintain, extend, etc.
This is currently an unsolved problem. Possible paths forward include:
- Tooling that helps developers with context engineering, observability, and evals. Lots of people are doing good work in these areas. But we are very early.
- New kinds of tooling that allows us to understand why deep learning models do what they do. Broadly and loosely, this is "mechanistic interpretability". Lots of good academic papers; but so far a completely unsolved problem.
- Use small(-er) models that we can individually test, observe the inputs and outputs of, and fine-tune. We can do this today.
@kwindla i like this pattern a lot. a small fast classifier answering one narrow question, like is this person frustrated, is way easier to test than hoping the main model picks up on tone every turn.
The biggest single pain point we see with the enterprise customers is that people just keep adding stuff to their system prompt. The more real-world corner cases you find, the bigger the prompt gets, and the more fragile and hard to debug the agent behavior becomes. Which is a vicious cycle.
(Also, of course, latency. I'm not allowed to tweet without mentioning latency at least once a day. But that's another topic.)
[ System One | Decision | Classifier ] models are the hot new thing. Pipecat 1.12 added a Classifiers abstraction. Since that release, we've been building a *lot* of things on top of that abstraction.
Google released Decision 1 yesterday, a classifier model that can take audio input.
Check out Jon's example (code and a video) below, which uses Google Decision 1.
Jon's PerceptionReader class is in Pipecat 1.16 (also released yesterday). PerceptionReader makes it easy to create a voice agent that understands how a user is speaking: positive, sad, frustrated, etc.
Pipecat 1.16 adds PerceptionReader, something I'm really excited about. How someone speaks can matter just as much as what they say, and voice agents have mostly been missing that.
We’ve been testing it with Google Decision 1, released yesterday. Stream it audio and you get a pretty accurate read on the prosody, tone and sarcasm of what the user said.
Try talking to it at this demo here: https://t.co/VlsJz3j054
We've exploring many different ways of measuring expressivity... more to share on that soon!
➡️ Pipecat Docs: https://t.co/AuIwJOcJvW
➡️ Google Decision 1: https://t.co/UjE85OEriS
I'm excited about a bunch of things:
- the emotion/tone stuff that Jon has been showing how to build for
- making decisions that we have to make today using LLM tool calling, but faster and more flexibly
- capturing user intent in a situation where the user might be talking to multiple agents, or multiple people
- fast classification of voice commands when voice is driving a user interface
One of the really fun things about this generative AI moment is how much innovation and interesting work is happening all over the world. Wherever I travel to, these days, there's someone on the DeepMind team to hang out with and talk about models, inference, APIs, and hardware. And when they come to SF they throw a great party!
I sat down with @tarush_agarwal_, CEO of @cekuraAi, to talk about his company's recent work on benchmarks. Tarush will be at VapiCon on Wednesday and Thursday this week in San Francisco. So if you're interested in benchmarks for voice AI, and will be at VapiCon, find him and say hi.
Cekura has spent the last two years building simulations and evals for teams that deploy voice AI agents at scale.
We talked about why Cekura is putting a lot of effort into building general benchmarks for public release, a little about their "secret sauce," what's hard about building good benchmarks, and what they're working on next.
0:00 Intro
0:29 Tarush and Cekura
0:47 Benchmarks and VapiCon
1:40 The Cekura real-world voice AI benchmark
3:51 Evals loops and how you build a multi-turn benchmark
4:51 Making simulations adaptive
6:33 Different approaches to benchmarks
7:23 Testing speech-to-speech models
8:58 How to use published benchmarks
10:58 What's next: benchmarking naturalness
12:29 Why naturalness is the next phase of voice AI
Also, I highly recommend VapiCon. Last year, it seemed like the whole Voice AI world showed up in San Francisco for the conference: https://t.co/m2zHEWftbY
Going to the Tavus office and seeing the behind the scenes development work on the kiosk stuff is really fun. Realtime AI is a lot more than just voice and video generation (though voice and video are obviously central to the experience). Object and face tracking, directional audio signals, broader context about what's happening in the environment, richer context from integration into backend systems, the personality and tone of the agent ... all of these play a big role in making an AI agent useful and natural.
Last time I visited Tavus I was there for a meeting with someone named Quinn. My name is also Kwin, same pronunciation, different spelling. I told the Tavus PAL receptionist at the office front door why I was visiting, and she said: "I'll let Quinn know you're here. Feel free to sit in the red chairs behind me. By the way, two Quinns is a lot of Quinns in one day." I couldn't disagree.
We put a Tavus PAL on a coffee kiosk and had someone try to hijack every order.
No directional mics. Our audiovisual isolation model tracks who’s speaking to it, so he (the faded captions) never got an answer.
We’re opening a kiosk mode pilot. Want in? https://t.co/quwookMLHq
A big benchmarks release from @cekuraAi today. Good benchmarks are very difficulat develop for a bunch of reasons:
- Automated testing of complex, multi-turn, conversational systems is a fundamentally hard problem. Do you use an LLM to simulate a user? That's the most common approach, these days. But how do you design things so that your benchmark isn't mostly just testing how good your sim is? (The opposite of what you actually want!)
- What are you testing? Task success? On what tasks? What is success? For example, if an agent gets a buch of things wrong along the way, but an infinitely patient simulator sticks around long enough, you can probably register success on a task. On the other hand, self-correcting during a conversation is a good thing, so you want to allow some flexibility in the "path" the agent takes. Individual turn-by-turn performance? (How do you judge a good turn vs a bad turn response. There's a lot of grey area.) Specific tool calling or instruction following metrics? What about latency, interruption handling, and subjective voice quality?
- There's a very large surface area for benchmark bugs.
The Cekura team has been thinking hard about how to build good benchmarks.
Introducing Cekura Bench: the world’s leading Voice AI Benchmarks
We benchmarked 50+ models from openAi, Google, Nvidia, xAI, Inworld, Gradium, Assembly, Fish audio and more
Now you can:
→ Compare naturalness, performance and cost
→ See how the best voice agents are being built.
→ Find the best fit for what you are building.
Our tests cover Speech-to-Speech models, Speech-to-Text models, Text-to-Speech Models, and Voice Agent platforms. We recorded real humans doing tasks in different domains and created the most representative voice benchmark out there!
Explore the benchmarks: https://t.co/eZAFy7P0Nj
We would love to collaborate with more teams on these benchmarks. Reach out if you are interested.
@pbogosta Yes. Smaller, fine-tuned models are starting to be used in a significant percentage of the agents we help enterprise customers deploy and scale.
We've gotten used to crazy progress every month in the world of frontier LLMs. And more recently, mid-sized, open weights LLMs (~20B - 200B parameters) hit that same fast-progress, upwards curve.
But the new, new thing is super fast progress in small, specialized models.
@JonPTaylor is training new models every week, now, for things like detecting whether speech is sarcastic. These models can take text input, audio input, or both. They can be small enough to run on CPU in <50ms. You can combine them with LLMs, decision models like Jev, other classifier models, etc.
Specialized model training/fine-tuning is one of the things that coding agents make it possible to do 100x faster. So go make your consumer GPU go brrrrrrrr and train some tiny models.
New week, new tiny model for Pipecat.
Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm.
It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering.
This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation.
Available in two weights (base and tiny): https://t.co/tb0KJq8k0K