kwindla

Infrastructure and developer tools for real-time voice, video, and AI. @trydaily // ᓚᘏᗢ // @pipecat_ai

[kwindla @kwindla]

https://t.co/ne9yJo9LWi

[kwindla @kwindla]

One of the really fun things about this generative AI moment is how much innovation and interesting work is happening all over the world. Wherever I travel to, these days, there's someone on the DeepMind team to hang out with and talk about models, inference, APIs, and hardware. And when they come to SF they throw a great party!

[Thor 雷神 ⚡️ @thorwebdev]

Always a pleasure catching up with @kwindla! Exciting times for him and the @trydaily team and happy to keep collaborating on @pipecat_ai 🙌

[kwindla @kwindla]

@raiyanj3_ Indeed. We don't have good ways to even talk about, much less to actually measure, "naturalness."

[kwindla @kwindla]

I sat down with @tarush_agarwal_, CEO of @cekuraAi, to talk about his company's recent work on benchmarks. Tarush will be at VapiCon on Wednesday and Thursday this week in San Francisco. So if you're interested in benchmarks for voice AI, and will be at VapiCon, find him and say hi.

Cekura has spent the last two years building simulations and evals for teams that deploy voice AI agents at scale.

We talked about why Cekura is putting a lot of effort into building general benchmarks for public release, a little about their "secret sauce," what's hard about building good benchmarks, and what they're working on next.

0:00 Intro
0:29 Tarush and Cekura
0:47 Benchmarks and VapiCon
1:40 The Cekura real-world voice AI benchmark
3:51 Evals loops and how you build a multi-turn benchmark
4:51 Making simulations adaptive
6:33 Different approaches to benchmarks
7:23 Testing speech-to-speech models
8:58 How to use published benchmarks
10:58 What's next: benchmarking naturalness
12:29 Why naturalness is the next phase of voice AI

[kwindla @kwindla]

Also, I highly recommend VapiCon. Last year, it seemed like the whole Voice AI world showed up in San Francisco for the conference: https://t.co/m2zHEWftbY

[kwindla @kwindla]

Going to the Tavus office and seeing the behind the scenes development work on the kiosk stuff is really fun. Realtime AI is a lot more than just voice and video generation (though voice and video are obviously central to the experience). Object and face tracking, directional audio signals, broader context about what's happening in the environment, richer context from integration into backend systems, the personality and tone of the agent ... all of these play a big role in making an AI agent useful and natural.

Last time I visited Tavus I was there for a meeting with someone named Quinn. My name is also Kwin, same pronunciation, different spelling. I told the Tavus PAL receptionist at the office front door why I was visiting, and she said: "I'll let Quinn know you're here. Feel free to sit in the red chairs behind me. By the way, two Quinns is a lot of Quinns in one day." I couldn't disagree.

[Tavus @tavus]

We put a Tavus PAL on a coffee kiosk and had someone try to hijack every order.

No directional mics. Our audiovisual isolation model tracks who’s speaking to it, so he (the faded captions) never got an answer.

We’re opening a kiosk mode pilot. Want in? https://t.co/quwookMLHq

[kwindla @kwindla]

A big benchmarks release from @cekuraAi today. Good benchmarks are very difficulat develop for a bunch of reasons:

- Automated testing of complex, multi-turn, conversational systems is a fundamentally hard problem. Do you use an LLM to simulate a user? That's the most common approach, these days. But how do you design things so that your benchmark isn't mostly just testing how good your sim is? (The opposite of what you actually want!)
- What are you testing? Task success? On what tasks? What is success? For example, if an agent gets a buch of things wrong along the way, but an infinitely patient simulator sticks around long enough, you can probably register success on a task. On the other hand, self-correcting during a conversation is a good thing, so you want to allow some flexibility in the "path" the agent takes. Individual turn-by-turn performance? (How do you judge a good turn vs a bad turn response. There's a lot of grey area.) Specific tool calling or instruction following metrics? What about latency, interruption handling, and subjective voice quality?
- There's a very large surface area for benchmark bugs.

The Cekura team has been thinking hard about how to build good benchmarks.

[Tarush Agarwal @tarush_agarwal_]

Introducing Cekura Bench: the world’s leading Voice AI Benchmarks

We benchmarked 50+ models from openAi, Google, Nvidia, xAI, Inworld, Gradium, Assembly, Fish audio and more

Now you can:
→ Compare naturalness, performance and cost
→ See how the best voice agents are being built.
→ Find the best fit for what you are building.

Our tests cover Speech-to-Speech models, Speech-to-Text models, Text-to-Speech Models, and Voice Agent platforms. We recorded real humans doing tasks in different domains and created the most representative voice benchmark out there!

Explore the benchmarks: https://t.co/eZAFy7P0Nj

We would love to collaborate with more teams on these benchmarks. Reach out if you are interested.

[kwindla @kwindla]

@dexhorthy Congratulations!

[kwindla @kwindla]

@pbogosta Yes. Smaller, fine-tuned models are starting to be used in a significant percentage of the agents we help enterprise customers deploy and scale.

[kwindla @kwindla]

We've gotten used to crazy progress every month in the world of frontier LLMs. And more recently, mid-sized, open weights LLMs (~20B - 200B parameters) hit that same fast-progress, upwards curve.

But the new, new thing is super fast progress in small, specialized models.

@JonPTaylor is training new models every week, now, for things like detecting whether speech is sarcastic. These models can take text input, audio input, or both. They can be small enough to run on CPU in <50ms. You can combine them with LLMs, decision models like Jev, other classifier models, etc.

Specialized model training/fine-tuning is one of the things that coding agents make it possible to do 100x faster. So go make your consumer GPU go brrrrrrrr and train some tiny models.

[Jon Taylor @JonPTaylor]

New week, new tiny model for Pipecat.

Pipecat Voice Reader listens to how something is said: tone of voice, energy, laughter, speaking rate and potential sarcasm.

It's a fine-tuned version of LAION’s affect-trained Whisper encoder, in two sizes (base or tiny), and runs on CPU in well under 50 ms per turn. In a Pipecat pipeline it runs beside the text tone reader, which reads what was said in the context of the previous exchange, providing broader context to the harness and metering.

This is one area where a cascaded pipeline has a real edge over speech-to-speech... each turn gets explicit, calibrated readings of both the words and the delivery, with context of the single turn and the entire conversation.

Available in two weights (base and tiny): https://t.co/tb0KJq8k0K

[kwindla @kwindla]

@vkyaligar I 100% agree that modularity helps with testing and observability.

[kwindla @kwindla]

@deepwhitman That’s awesome!

[kwindla @kwindla]

More from Jon on this topic: https://t.co/JLG4es3W6f

[kwindla @kwindla]

@nadeemcool47 We can definitely help with this. Do you want to DM me here with your email address or WhatsApp number?

[kwindla @kwindla]

https://t.co/ne9yJo9LWi

[kwindla @kwindla]

specialized models 🤝 orchestration-layer standards

[Jon Taylor @JonPTaylor]

Using purpose-trained classifiers that are small and fast enough to not impact the voice loop can help close the gap in naturalness between a speech-to-speech and cascade pipeline.

For intent and tone though, text obviously only gets you so far. But audio models can be small too!

Check out Pipecat Backchannel Detector: https://t.co/UBo37NzvCH

It's just 734k parameters, and does a pretty great job of figuring out when a user "mhmm"s or "oh right"s without interrupting the bot.

Oh, and these backchannel events are all part of the RCS and RTVI standard we have defined in Pipecat, so supported with typed events in the client libraries too!

[kwindla @kwindla]

@vtahowe Thank you, @vtahowe!

[kwindla @kwindla]

@ivanleomk That’s a great idea!

[kwindla @kwindla]

@lina_colucci Thank you, @lina_colucci!

[kwindla @kwindla]

@angushf I really enjoyed talking!

[kwindla @kwindla]

@vikasreddy Thank you!

[kwindla @kwindla]

@pbbakkum Thank you, @pbbakkum !