Dante E

Voice AI Research @ Decagon

[Dante E @EveraertDante]

https://t.co/r62dQCxCtc

[Dante E @EveraertDante]

@jaminball @DecagonAI Thanks for having me!

[Dante E @EveraertDante]

Had a blast talking with @jaminball and @palak_go on First Pass about why transcription shouldn’t be on the critical path for voice agents, and what it actually takes to go audio-native. Thanks for having me!

[Jamin Ball @jaminball]

Next up on First Pass: A conversation with @EveraertDante, a voice AI researcher at @DecagonAI on Duplex Models

LLMs are turn based. Duplex Models are constantly "listening" streaming audio in and out simultaneously.

[Decagon @DecagonAI]

Effortless.

[Dante E @EveraertDante]

@NielsRogge @togethercompute @FireworksAI_HQ @PrimeIntellect Verticalizing away from ~0 margin businesses/establishing lock-in from an otherwise easily swappable product, just like OAI/Anthropic

[Dante E @EveraertDante]

SF in October 🇺🇸

[Dante E @EveraertDante]

We open sourced PACT protocol today.

Personal agents already call businesses on behalf of customers, and there's no way to verify who they represent or what they're allowed to do. PACT fixes that on top of A2A + OAuth.

Protocol work isn't glamorous but this one's going to matter.

[Decagon @DecagonAI]

Today, we're open-sourcing the Personal Agent Consent & Trust Protocol (PACT), co-developed by @Instinct. We're also joining the Personal Agent Protocol working group by @Muse.

Built on A2A and OAuth, PACT enables businesses to verify who a personal agent is acting for and what the customer has authorized it to do.

Excited to shape the industry standard for how personal agents will interact with businesses.

[Dante E @EveraertDante]

Voice 3 launched yesterday and I presented the duplex architecture I've been working on at @DecagonAI Dialogues. While building it, I was surprised that even if you tell a model there's another one working with it, its reasoning traces still show it thinks it's the only one.

I believe this is because most models are post-trained to be a solo act, and most of our research went into beating that out of them. The industry could use a lot more models post-trained to work alongside other models, imo.

I'll share a deeper dive next week.

[Dante E @EveraertDante]

Big day. Two voice research projects I've been building at Decagon for months are finally out:

1/ A next-gen duplex architecture. Interrupt the agent, make small talk, ask something else, get a progress update, all while it keeps working. It feels like talking to a person and I can't wait to show more next week.

2/ Chord, our first in-house trained speech model. In a blind test 90% of people couldn't tell it from a real human

Also, please be gentle about my acting in the launch video 🙏

[Decagon @DecagonAI]

Voice 3 is here.

Our most advanced voice experience yet combines Chord, our new voice model from Decagon Labs, with a duplex architecture that lets agents process incoming audio even while speaking.

Chord is post trained specifically for customer conversations, giving agents more natural pacing and intonation, while the duplex architecture enables smoother interruptions and less dead time as the agent keeps working.

Also launched today at Decagon Dialogues:
- Personal Agent Gateway: a dedicated communication layer for handling interactions with personal AI agents.

- Agent Modules: a foundation for expanding agents into new customer journeys.

- Duet Apprentice: a learning layer grounded in your company’s knowledge and the teammates who know the business best.

[Dante E @EveraertDante]

Last week I posted that transcription in voice agents is dead. That sparked some debate, so here are some followup thoughts:

1. No, transcription is not dead in general. It just does not have to be on the critical path anymore in voice agents. It can still be useful for debugging, or totally different use cases like live captioning.

2. People seem to have missed that operating on audio directly has many other advantages. Even a perfect transcription can't tell you if the audio was muffled, if it's a side conversation, or if the customer is frustrated. You get strictly more capabilities by going audio native.

3. Cascade was the right call until recently. A year ago I would've also built transcription into the pipeline. When Big Bench Audio came out, Whisper + GPT-4o scored 92%, +26pps over GPT-4o native speech - but now audio-native models have maxed it out at 99.7% so that reason is gone.

4. People assumed I meant full S2S will be replacing it. I actually meant any audio-native LLM can replace it, and those are already here tbh. You can keep the rest of your pipeline (e.g. TTS) and many great open source audio-native LLMs already exist (e.g. MiMo was literally just launched last week).

[Dante E @EveraertDante]

https://t.co/r62dQCxCtc

[Cyrus @cyrusasg]

a coding agent delegating work and a consumer agent negotiating a refund are both “multiagent systems.” but only one lets you design both sides of the interaction.

wrote about what changes, from context and verification to incentives and training. https://t.co/lHkT7BDpT8

[Dante E @EveraertDante]

@rdesh26 Completely agree! It just doesn’t need to be a piece of the voice agent itself anymore

[Dante E @EveraertDante]

https://t.co/r62dQCxCtc

[Cyrus @cyrusasg]

qwen-audio-3.1 is a concrete example of voice agent post-training converging with text llm recipes: sft, rl-trained specialists, and on-policy distillation.

three parts stand out:

1. transfer capabilities from text. after audio/text sft, the audio student generates its own responses. a text teacher sees the equivalent text input and provides next-token supervision along the student’s trajectory. a frozen audio reference helps retain audio-conditioned behavior.

2. specialize, then consolidate. domain experts train w/grpo for things transcripts miss like emotion, intent, acoustic context. multi-teacher on-policy distillation folds those capabilities back into the main audio model.

3. train against executable outcomes. environments contain tools, databases, policies, and validators. rewards check whether the task actually completed, whether writes were allowed, and whether required confirmations happened. tasks progressively get harder as the model improves.

voice models can inherit a lot of what text llms have already learned. combine that with increasingly rich synthetic training environments and the gap in reasoning and task execution could close quickly. no more cascaded asr->llm pipelines needed.

paper: https://t.co/Omu5WZv3wq

[Dante E @EveraertDante]

@CVestergaard_ Thank you!

[Dante E @EveraertDante]

Had a ton of fun filming for Decagon Dialogues. Lots of exciting voice announcements coming Oct 1

[Decagon @DecagonAI]

Introducing DuetBench-2, our latest benchmark for measuring whether self improving agents create improvements that last.

After passing 93% of diagnostic tasks and outperforming humans on complex agent building tasks, Duet quickly outgrew the original benchmark. 🧵

[Dante E @EveraertDante]

Multiple-speaker identification sounds simple and solved (e.g. diarization), but it’s actually completely mismatched to real-time use cases. if you take action based on a speaker flag, false positives are critical: to stay below 1% over a 10-turn conversation you need a per-turn FP rate under 0.1%, and current methods are nowhere close. so we built a suite of methods that outperform existing approaches several-fold.

Built with @alon_rag

[Decagon @DecagonAI]

Conversational voice agents need to know when a different person is speaking, but not every background voice should count.

We combined speaker embeddings with a post-trained audio-language model to determine when a speaker change matters. https://t.co/Cg5HKtyaUf

[Cyrus @cyrusasg]

me and the boys this weekend

[Dante E @EveraertDante]

@arakharazian Not having ZDR for Fable is a big issue for enterprises too

[Dante E @EveraertDante]

Great thoughts on one of the biggest topics in how to get AI products out there, worth a read

[Jesse Zhang @thejessezhang]

https://t.co/LCU8avjBpC