dan

Product @assemblyai 🧙 Voice Agent API

[dan @dan_aai]

@kai_wen_eth @verdictprotocol @AssemblyAI hey @kai_wen_eth! for adding state persistence, e.g. if you want to continue from a past conversation, it's typically recommended to store past conversations on your end, and then inject the context into the prompt for the next session

We're working on ways to make this easier!

[dan @dan_aai]

@alps_n_apps @alexcooldev @amosrazi Yes 🤝

[dan @dan_aai]

@alps_n_apps @alexcooldev @amosrazi we have a startup program too :) so it could be $0 a month https://t.co/XxKDSw3Y0v

[dan @dan_aai]

trying something new - come chat! https://t.co/d9m3RsrtFS

[dan @dan_aai]

Would you need to use the extracted email immediately (e.g. for booking a meeting)?

If you can manage some delay or even wait until the end of the call, one common method we do see voice agent devs use, is transcribing the audio snippet containing the email with an async/batch model since they achieve better alphanumeric accuracy than realtime models

Another thing you can try for a realtime experience is having the user spell out the email with the phonetic alphabet (A for Alpha, B for Bravo etc) as that can be more robust

Overall every realtime model is not quite there yet for 95%+ accuracy across all providers, so it does require some workarounds, but it's possible to handle it gracefully and we humans need confirmation sometimes too

[dan @dan_aai]

@gxespino Lol fair feedback, building the best performing speech to text models is always the main focus 😉

[dan @dan_aai]

First month I've ever used up all my Windsurf credits

Lots to ship

[dan @dan_aai]

Launch day! Check out our new speech understanding, guardrails and LLM Gateway features today as well as a new docs redesign

We just made it a whole lot easier to build Voice AI apps 😎

[AssemblyAI @AssemblyAI]

Today, we’re introducing new tools and model updates to help you build, deploy, and scale Voice AI applications.🎙️

🆕 Speech Understanding: Turn transcripts into actionable data with speaker identification, custom formatting & translation
🆕 LLM Gateway: One API for your voice-to-intelligence pipeline, with GPT, Claude, Gemini & more
🆕 Voice AI Guardrails: End-to-end protection for safe, compliant content

Model Upgrades:
✅ 99 languages with auto code-switching
✅ 64% fewer speaker counting errors
✅ 57% better accuracy on critical terms with 1,000-word context

Real results:
🔥 Calabrio: 80% boost in customer satisfaction, 22% revenue increase
🔥 Siro: "10/10 customers say 'wow, that insight was crisp'"

Build & scale Voice AI in minutes! Try it in our Playground or check the docs. 🚀

[dan @dan_aai]

@uselayercode congrats team! 🔥

[dan @dan_aai]

plz like this tweet so we can convince our boss to buy this

[Mart Schweiger @martschweiger]

i am once again begging for an @AssemblyAI mascot

[dan @dan_aai]

this is 🔥🔥🔥

love to see agents working well with our API, the team has been grinding to improve docs readability for LLMs

[matt palmer @mattyp]

Transcribe a 75 minute video in under 2 minutes

This app splits video from audio (locally), making upload size about 100x smaller

Then, it uploads the audio to AssemblyAI for transcription

Finally, transcripts are corrected with Gemini Flash

Built with Agent 3, of course

[dan @dan_aai]

@mattyp this is so fire, ffmpeg wasm is too quick 🔥🔥🔥

and glad to hear you like our API and that Replit agent can pick up the docs perfectly!

[dan @dan_aai]

@stevenvillarino @mattyp we support diarization :) https://t.co/3svYanB4fj

[dan @dan_aai]

if you were tasked with transforming a call centre with AI, how would you do it?

here's the playbook i'd run:

3 stages

1 - start tracking your best performers with post call transcription and LLM as a judge analysis, you wanna start collecting the human data on what responses perform best and how questions should be answered, to power the next stages

2 - implement agent assist, use the data you collected from your best performers to power the agent assist responses. see how often agents are using the guidance, and call success rates to determine how well its performing

3 - add voice agents, once you know you're generating good responses and you have the right data, it's time to scale up with voice agents. you don't have to worry about improving agent responses as you already have the data from stage 1 & 2 and can focus just on making it sound natural

most people try skipping straight to 3 and end up battling between improving responses and making it sound natural, hence why 95% of AI projects fail because you're jumping the gun

no amount of prompting can make it sound human, you need to provide references of human responses

[dan @dan_aai]

the ttfb (time to first byte) metric for measuring STT latency for voice agents is inaccurate, here's why you should be using ttct (time to complete transcript) instead:

ttfb only measures silence -> start of speech -> first transcript chunk

it completely ignores the more critical path which is ttct:
end of speech -> silence -> final complete transcript (to be sent to the LLM)

this can lead to some interesting conclusions:
- streaming models that emit fast but inaccurate partials get rewarded (easy to game)
- async models ran in a low latency fashion look like they perform terrible with ttfb due to no mid-speech partials (even tho they can work well in a voice agent)

but the thing is: you're not using partials to generate the LLM response, you're using finals, so that is what you should measure

models also have different configs for emitting partials vs finals, with partials you want to optimise for speed, finals you want to optimise for accuracy (without affecting latency too much) - so you will get different speeds

at AssemblyAI we're laser focused on optimising for TTCT while maintaining industry leading accuracy, and we're shipping a new update today to our streaming model to return transcripts faster than any other provider via a new field called "Utterance" which emits finals on any 160ms silence so you can generate your LLM response as quick as possible - give your voice agents a noticeable speed up today by testing it out ;)

[dan @dan_aai]

@icegiantx @pgrache @AssemblyAI @liveatc Sure thing! Dm me the URL :) @icegiantx

[Mart Schweiger @martschweiger]

New improvements to @AssemblyAI's Universal-Streaming:
- 21% accuracy improvement for repetitions in speech
- 20ms faster response time
- 66% better recognition of your custom keyterms
- 3% improvement in accuracy
- 4% better recognition for accented speech
- 7% improvement at recognizing names, brands, and places

[PHP Architect @phparch]

📣 Hey, look what’s new!
Real-Time Call Transcription with FreeSWITCH and AssemblyAI: A Debugging Journey
https://t.co/4gsFAD37Cj

[dan @dan_aai]

@icegiantx @pgrache @AssemblyAI @liveatc @icegiantx check out our getting started guide in our docs: https://t.co/Cd649gDeOc

if you need any support feel free to email our team: [email protected] or create a ticket via the popup in the bottom right of the docs :)