In interaction models, there are a few limitations, but these are more of engineering and data problems than cons.
Mostly data and evaluation. You need a strong dataset to train the model on when to hand off and when to answer by itself. If you miss it here, you might get denials or, even worse, hallucinations, which could have been correctly answered by the background reasoning model.
Similarly, when you hand off a task to the background model, it might take a few seconds to get the answer then you have to train and evaluate the model on when and how to interrupt the user for that answer. So that it feels natural to the user.
Overall, interaction models are bound by data, evals, and real-time inference bottlenecks, and I'm working on these.
In asynchronous backend architecture, there are a few limitations, but these are more of engineering and data problems than cons.
Mostly data and evaluation. You need a strong dataset to train the model on when to hand off and when to answer by itself. If you miss it here, you might get denials or, even worse, hallucinations, which could have been correctly answered by the background reasoning model.
Similarly, when you hand off a task to the background model, it might take a few seconds to get the answer then you have to train and evaluate the model on when and how to interrupt the user for that answer. So that it feels natural to the user.
Overall, interaction models are bound by data, evals, and real-time inference bottlenecks, and I'm working on these.
3 bottlenecks of real-time voice AI:
1. Data
- Need more natural conversation recordings
- Can't generate synthetic data because it lacks EQ, expressiveness, and human emotions
- Can't train on YouTube videos or podcasts because those are narrative conversations for an audience, not natural human conversations
2. Evals
- Lacks benchmarks to evaluate EQ, expressiveness, tone of voice, and other metrics
- Hard to quantify human emotions
- LLMs are still not reliable for identifying human emotions properly to be used as a judge
3. Inference
- Hard to manage long streaming sessions
- Need effective context management for longer sessions
- Need a custom inference engine to stream 200ms frames effectively
#VoiceAI #voice #ai
In cascade voice models, endpointing is part of ASR, not VAD.
VAD just acts like a sensor that detects whether there is "speech" or "no speech".
But ASR runs a fixed timer every time there is "no speech"; if the timer goes off, it gives the turn to the LLM to speak.
That's called endpointing.
Opening a Slack or Discord channel doesn't mean you have created a community.
If you think through it, you'll realise: you have just created a communication channel, not a community.
To have a community, you need 3 layers:
1. Discovery
2. Communication
3. Collaboration
In the discovery layer, people need to find relevant prospects. Just imagine finding an engineer, founder, or investor in your Slack or Discord channel.
You need to have a web interface to list, filter, and browse different members, with the ability to contact them.
That's your starting point for community interaction.
For communication, you already have your channels. So no need to comment on that.
The last is collaboration.
You have to have at least 1 vertual/in-real meetups every month, and the focus of that meetup should be meeting or introducing key people in the community.
If I say it in one line, in a healthy community, people should find, communicate, and collaborate with each other.
#Tech #Community
@_josh_meyer_ Yeah, that's why most of them now rely on handoff to a more capable reasoning model. Still huge room for improvement in handoff, coordination, and real-time streaming.
I still believe algorithm tests are a horrible interview medium for startups.
Unlike at big tech, you're not working on a tiny part of a feature at startups. Most likely, you have to implement the complete feature from end-to-end with impl, testing, deployment, etc.
So, I think we should check for complete product development rather than LeetCode-style interviews.
Speech-to-speech doesn't mean full-duplex models.
Speech-to-speech describes the modality of a model, while full-duplex refers to the turn-taking capability of that model.
A model can be speech-to-speech and still not full-duplex.
Think of a "walkie-talkie" as a speech-to-speech model: you can send and receive audio, but can't do both at the same time.
But full-duplex models are more like phone calls, meaning both parties can speak and listen at the same time.
Speech-to-speech means that the model takes, processes, and generates audio natively, while full-duplex means it can take audio while generating an output.
To make a speech-to-speech model a full-duplex one, you need:
- Natural turn-taking
- Ability to speak while it listens
- Backchanneling
- Learned interruption
- Ability to know when to speak and when to remain silent
#AIVOICE #voiceagents #voiceai #fullduplex #models
In a normal human conversation, the average gap between a question and its answer is about 200 ms.
Keep this 200ms in mind; you will encounter it quite often during your Voice AI journey.
Cons of a Cascade-Style AI Voice model:
- Artificial Turn-taking
- High latency
- Error Propagation
- Loss of Acoustics
- No Human Emotions/Tone of Voice
- No Speaker Identity
- No Backchanneling
- 0 Overlap
- No Interruption