@Kcodess writing the whole code from scratch gives a more clear understanding instead of no-code tool, but if you have time constraints then it's good to use them but for prototypes nd all
Building a voice AI pipeline that actually talks back fast (1/3)
So I've been obsessed with a question for a while: why do most voice assistants feel like you're talking to a walkie-talkie? You say something, then there's this awkward pause, and then it replies. It never feels like an actual conversation.
I wanted to figure out if I could fix that. So I built a real-time voice pipeline from scratch.
The whole thing is a chain running over a single WebSocket connection: your mic goes to speech-to-text (Deepgram's Nova-3) which shows interim text and takes care of end of turn, that gets fed to an LLM (Claude Haiku 4.5, through OpenRouter), and the reply then streams to text-to-speech (Sarvam's Bulbul) which then streams back out your speaker. The backend tying it all together is FastAPI. Simple enough on paper. The hard part is making it fast enough that it feels natural.
A few things that made the biggest difference:
It starts talking before it's done thinking. Instead of waiting for the LLM to finish the whole answer, I split its output into sentences on the fly. The moment the first sentence is ready, it's already being handed to Sarvam and turned into speech while the rest is still being generated. That overlap is most of what kills the awkward pause.
No wasted milliseconds on audio. I kept everything as raw PCM audio the whole way through, no compressing to MP3 and decompressing again at every step. Sounds like a small thing but those little delays add up and you can hear them.
You can talk over it. This was the one I cared about most. There's a tiny voice-detection model (Silero VAD) running right in your browser that's always listening, and the second you start talking over the AI, it shuts up and listens, instantly. No push-to-talk, no buttons. Just like interrupting a person who's rambling. Meanwhile Deepgram figures out when you've actually finished a sentence on its own, so there's no "waiting for the beep" moment either.
The audio doesn't stutter. Getting little chunks of sound to play back-to-back without clicks and gaps in a browser is genuinely annoying. I ended up scheduling every chunk precisely on the browser's audio clock with a small look-ahead buffer to absorb network jitter. Took a while to get properly seamless.
End result: sub-second, back-and-forth voice that actually feels like a conversation instead of a turn-based game.
This was really the foundation for everything I built after.
Demo attached below.
github: https://t.co/KK0125GFX3
Next post I'll get into what happened when I gave this thing hands and a memory