Rohan Siva

research @decagonai
ml & robotics research @utaustin

[Rohan Siva @_rsiva]

@DecagonAI @alon_rag might be a genius

[Decagon @DecagonAI]

Conversational voice agents need to know when a different person is speaking, but not every background voice should count.

We combined speaker embeddings with a post-trained audio-language model to determine when a speaker change matters. https://t.co/Cg5HKtyaUf

[Alon Ragoler @alon_rag]

multimodal LLMs are great but they have some key limitations preventing them from accomplishing general audio tasks, including some as basic as speaker-change detection!

[Jesse Zhang @thejessezhang]

I've written a lot about how low latency is critical in voice agents, and it's quite a hard problem when you also need to keep accuracy super high.

Here's some insight into how our team approaches a portion of this!

[Rohan Siva @_rsiva]

multimodal inference feels pretty underexplored, especially when you move beyond standard transformer architectures. fun to share what we learned when scaling our text to speech model to 10x throughput

[Decagon @DecagonAI]

Fast text serving ≠ fast speech.

Here's how we got our time to first audio under 30ms with nearly 10x more audio throughput. https://t.co/cGPMpyPw6v

[Cyrus @cyrusasg]

multimodal inference doesn't fit the mold of standard text autoregressive generation. text-to-speech models feature different architectures, different states, and different batch shapes. we rebuilt our tts serving around that and simultaneously reduced our time to first audio while improving throughput by several fold.

[Decagon @DecagonAI]

Fast text serving ≠ fast speech.

Here's how we got our time to first audio under 30ms with nearly 10x more audio throughput. https://t.co/cGPMpyPw6v

[Rohan Siva @_rsiva]

@cyrusasg Let’s gooo

[Rohan Siva @_rsiva]

tts naturalness is hard. sft has its limits, but rl is tricky for flow-matching models since they lack the log-probs dpo and grpo rely on.

excited to share work on adapting both methods for TTS with @cyrusasg at @DecagonAI!

[Decagon @DecagonAI]

Modern TTS models can sound great — and still fail badly on pacing, pauses, and prosody.

We adapted DPO + GRPO to flow-matching models to tackle the tail end of TTS behavior: https://t.co/PnE1apjxg6

[Decagon @DecagonAI]

Modern TTS models can sound great — and still fail badly on pacing, pauses, and prosody.

We adapted DPO + GRPO to flow-matching models to tackle the tail end of TTS behavior: https://t.co/PnE1apjxg6