Amy

building in voice AI @withdavidai || currently thinking about voice evals and how to improve voice AI models

[Amy @pineaultamy]

Can we stagger the developing situations a little

I only have 4% battery left and several developments that require me to “monitor the situation”

[Amy @pineaultamy]

Can confirm I choose flights based on the odds of good Wi-Fi.

I religiously check how likely the plane is to have Starlink. Will pay a premium and take a slightly worse departure time for better internet.

[Sawyer Merritt @SawyerMerritt]

United Airlines CEO in new interview on @Starlink Wi-Fi:

"Wi-Fi is the No. 1 reason that people pick an airline flight. I think it's the biggest near-term competitive advantage that we've ever had in United's 100-year history. And it really is something that's causing customers to try United because they want Starlink. By the way, all the JFK flights 100% are going to be on Starlink when we start JFK [soon]."

[Amy @pineaultamy]

today I was talking to one of the voice models on DIAL, our multiturn S2S benchmark, and instead of staying silent while it’s thinking, the model verbalized in full seriousness “NO SPEECH”

and then proceeds to be silent…and then carried on, business as usual 😳

[Amy @pineaultamy]

I really like how the photos app surfaces pics from different points in the album history

Today it surfaced these pics from one of David AI’s first ever offsites, back when we were a wee 15 person company that fit under one large roof

we’ve grown 5x since then in headcount

[Amy @pineaultamy]

@andrewdsouza yep it's giving canada

[Amy @pineaultamy]

Darn midwestern Americans & Canadians out here saying “yeah no” to mean no and “no yeah” to mean yes 🫠

[Amy @pineaultamy]

More details for the inquisitive

[Amy @pineaultamy]

On ASR Bench, missing a language will penalize the model's overall score.

We made this deliberate choice because the overall ranking should rewards broad multilingual performance.

This means a model cannot simply skip its weaker languages and have its overall ranking reflect only its strongest ones.

If you're interested, the full methodology is listed on Research at David AI.

[Amy @pineaultamy]

Only about 8% of speaker-labeling error in our published ASR Bench analysis came from models confusing the speakers.

Most of the errors came from missing speech or marking silence as speech.

Timing accounts for a lot of the score. More speech-to-text musings to come. Thinking a lot about ASR this week.

[Amy @pineaultamy]

A speech model can make your transcript easier to read and less accurate at the same time.

Imagine someone speaking in Egyptian Arabic and receiving a transcript rewritten into formal Modern Standard Arabic.

Or, for our @meetgranola and @WisprFlow users, saying “I, I think we should, um, start on Friday.”

but the transcript comes back: “I think we should start on Friday.”

The result now reads as one smooth sentence, but it has changed what the person actually said.

ASR Bench published by @withdavidai uses verbatim references, keeping fillers, repetitions, false starts and other features of natural speech. A model that edits those away can lose accuracy against that reference (even if the output reads more naturally).

That feels especially relevant when people say a new transcription model is “better.” Better at producing readable notes and better at preserving the original speech pulls in somewhatdifferent directions.

TLDR - "better" is once again a term to pick apart...and in this case, readability and faithfulness are different things to evaluate!

[Amy @pineaultamy]

My conversational satisfaction would increase greatly if voice agents can tell the difference between

“okay”

“okay…”

and

“OKAY.”

[Amy @pineaultamy]

@pablogimenezz yah the per language breakdown is across the top of the graphic! you can look down the ES column here https://t.co/Nqe3xPHPyq

[Amy @pineaultamy]

New model drop! Lots of great speech-to-text activity recently

[David AI @withdavidai]

New models have been added to ASR Bench (DAI-ASR-I18N)!

Here’s where they land by average rank by Transcription Accuracy:

🆕 Scribe v3 by @ElevenLabs : #2
🆕 Reson8 Rumble-2 by Reson8: #11
🆕 Nova 2 Sonic by @amazon : #17
🆕 Parakeet TDT 0.6B v3 by @nvidia: #18
🆕 Granite Speech 5.0 2B by @IBM: #20

Lower average rank is better, with language coverage and per-language error rates shown alongside each model’s results.

You can explore the language-level breakdowns and compare open-weight and proprietary models directly on the benchmark.

[Amy @pineaultamy]

@ElevenLabs Congrats on a good model! We eval'd this on ASRBench, our multilingual ASR benchmark, and Scribe V3 ranked #2 on Transcription Accuracy

[Amy @pineaultamy]

https://t.co/Nqe3xPHPyq

Details here if you're interested! Updated rankings & methodology can be found on Research @ David AI

[Amy @pineaultamy]

We added @ElevenLabs's new Scribe V3 model through ASR Bench (DAI-ASR-I18N), our multilingual ASR benchmark!

Congrats to the team for ranking #2 on Transcription Accuracy 🗣️

[Amy @pineaultamy]

@chetan_raj Hey Chetan, you can click around our benchmark here if you're interested https://t.co/kr7kWywMQu

This is the stacking when sorted on naturalness overall (grouped by thinking)

[Amy @pineaultamy]

When a voice model ranks #1 on a leaderboard, what is it actually #1 at?

On DIAL, you can rank models by overall preference, naturalness, audio quality, or content. Each answers a different question about the experience of talking to a model.

We’ve added more models this week, with more coming over the next few weeks. As you compare the rankings, it helps to know what’s behind them.

ICYMI, here are the PMs at @withdavidai behind DIAL walking through what goes into those leaderboard scores (it also gives you a sneak peak of the David AI office!)

[Amy @pineaultamy]

update i responded and said I wasn’t interested, and then it sent me nearby group fitness classes

instinct’s commitment to my health is commendable

[Amy @pineaultamy]

a few days ago i asked my @instinct to cancel a gym membership, and today it started messaging me about finding "a top personal trainer in sf"

should i be pleased that it cares about my health or ???

[Amy @pineaultamy]

@MaxCarter_01 Thanks! We're expanding to new domains and scenarios & also adding new models to the leaderboard in the coming weeks