Desh Raj

Speech + LLMs @nvidia | Previously: @Meta MSL, @jhuclsp, @IITGuwahati

[Desh Raj @rdesh26]

I’m happy to share that I’m starting a new position as Senior Research Scientist at @nvidia!

Looking forward to open science for speech full-duplex models :)

[Desh Raj @rdesh26]

After 2 wonderful years, I left Meta this week. During this time, I worked on several projects related to speech and LLMs:

- Built the first multi-channel audio foundation model with M-BEST-RQ (https://t.co/Uwx3XLuvIZ)
- Made ASR with SpeechLLMs faster (https://t.co/ED8meCVzuD) and more accurate (https://t.co/pk4CkDgFgg)
- Shipped the first production-ready full-duplex voice assistant (https://t.co/X1pwmHUW3u)
- Improved Moshi’s reasoning capability with chain-of-thought (https://t.co/6Lv8svqodU)

I am grateful to my managers for having my back on critical projects, and fortunate to have collaborated with several brilliant researchers and engineers during this time.

As to what's next, I am still in NYC and continuing to do speech research. More on that later!

[merve @mervenoyann]

idk I try to help youngsters high schoolers etc in my DMs if I can be helpful

this is the reason why I don't like SF, feels like it's full of social hierarchies where numbers/markers matter

not as much in Europe

[Desh Raj @rdesh26]

@sathwikmandava1 @ArtificialAnlys Qwen3.8-27B is dense model, meaning that it activates 27B param per token, vs 12B active for Super-3.5.

[NVIDIA AI @NVIDIAAI]

Meet NVIDIA Nemotron 3.5 Super VL

Our new open model combines reasoning and tool use with text, image, document, and video understanding for long-running AI agents.

120B total parameters with 12B active 🧵

[Desh Raj @rdesh26]

@harsh___d The easiest is to add a bunch of noise augmentation during training (you can check ESPNet recipes for how to do this). Having real noisy data would also be helpful, if expensive.

[Desh Raj @rdesh26]

I love full-duplex models and GPT-Live is awesome, but this claim is dubious at best.

From Josh's post, most of the models hold up under "street noise" and "cafe chatter", but fail under "background TV", despite all of them being added at low energy levels. The only conclusion that we can draw from this is that most models are trained with noise augmentation resembling ambient noises or on in-the-wild data where it is unlikely to find TV noise.

On the TV noise scenario itself, GPT-Live degraded by ~39% while Grok degraded only by 15%, despite not being a full-duplex model. From xAI's blog post: "We've focused on making Grok Voice Think Fast 2.0 perform exceptionally well in real-world settings, with substantial background noise and telephony compression." In short, they trained with a lot of real-world noise augmentation.

Speech folks have worked on noise robustness and target-speaker recognition for a decade now, and came up with several cool architectural tricks. But if you look at ASR recipes from Kaldi/ESPNet, the only constant is noise augmentation and far-field simulation.

[Peter Bakkum @pbbakkum]

AFAIK gpt-live-1 is the only model here where turn detection is built into the audio model itself rather than separate, which is why it’s so much better perceiving turns in noisy environments. Enabled by the full duplex architecture

[Desh Raj @rdesh26]

I think it does well for the same reason that it does great on Tau-Voice: the post-training with delegation on enterprise agentic tasks is top notch.

I get the point you're trying to make, and I agree that combining "when to respond" with "how to respond" is essential for a great voice agent, but I don't think this benchmark really captures that aspect. A more suitable benchmark would be something like this from @liva_ai: https://t.co/fRIRTsCRbQ

[Desh Raj @rdesh26]

@jessearmand It's hard to say with certainty, but in general, if the frontier lab hasn't explicitly advertised "full-duplex" in the promotions, it is very unlikely that the model is FDX. Most voice models that advertise "real-time" are either cascades or half-duplex.

[Desh Raj @rdesh26]

@_josh_meyer_ I think they just used i-vectors extracted from the utterance itself, mainly as a way of speaker adaptation (rather than extraction). But the same idea has been used for TSE with SpeakerBeam (from NTT).

[Desh Raj @rdesh26]

In full disclosure, Kaldi recipes did append speaker i-vectors along with the frame-level features for the acoustic model, but this was mostly a remnant of the older speaker-discriminative training of HMM-GMM models. Later recipes (with E2E models, larger training data, and more noise augmentation) dropped these without any noticeable performance drop.

Target-speaker recognition (or "device directedness" as it is called in industry) is a real problem when the energy level of the interfering speaker is similar to the main speaker, but again sonething can be reasonably tackled through appropriate augmentation strategies.

[Desh Raj @rdesh26]

@_josh_meyer_

[Josh Meyer @_josh_meyer_]

How to kill a voice agent? Just turn on the TV ☠️

We benchmarked 14 voice agents with a TV playing in the background.

Task completion fell from 38.7% to 8.6%

🧵/

[Desh Raj @rdesh26]

@_josh_meyer_ Ahh time to dig up the old papers on noise-robust and target-speaker ASR!

[Desh Raj @rdesh26]

@rissa_cao Could you link the model weights as well, thanks!

[Desh Raj @rdesh26]

@FishAudio I didn't find any links for the model weights?

[Joma Tech @jomatech]

Introducing ChatGPT for Dishwashing

GPT-6 achieves state-of-the-art results on dishwashing benchmarks for long-running jobs and difficult stains.

The Navier–Stokes Millennium Prize Problem is a $1 million challenge about whether the equations describing fluid motion can break down. In this video, I accidentally found a counter-example while trying to optimize my dishwasher. But in reality, OpenAI used 10,000 AI agents to find a proof in 88 hours: https://t.co/6HTwFfXnm1

Parody of the original GPT-6 Astra ad

@OpenAI #ChatGPT_Partner #chatgpt

Credits:

Written and Directed by Joma
Co-Writer - Henry Connor Coan IV @coan_henry
Production by Dauntlus Studios @dauntlus.studios
Producer/Co-Director - Marcus Liew @dauntlus
Camera Operator - Amos Lee @amoselijahlee
Art Director - Alethea Soo @aletheasoo
Art Assistant - Alicia Lim @aliciatkl_
Assistant Editor - GPT-6 Astra

Music: Tensions Run High by Soundridemusic
https://t.co/kBUsd2DK76

[Allan @niemerg]

I think I'm going to get a lot of mileage out of this one.

[Desh Raj @rdesh26]

@voicearena_ai @smallest_AI If you are ranking diarization+ASR systems, why are you using DER for ranking? There are several metrics for speaker-attributed transcription such as cpWER and tcpWER. You can use MeetEval (https://t.co/LB6hzzIAnv) for scoring.

[tuk’s ˚⋆ @lilieslores]

love how in 90s and 2000s romcoms, nyc was never just a backdrop but a character of its own

[Desh Raj @rdesh26]

@shinjiw_at_cmu Congratulations Shinji! Huge milestone, 200 coming soon :)

[Desh Raj @rdesh26]

this was not in my bingo card for 2026

[Ben Burtenshaw @ben_burtenshaw]

PSA: @mervenoyann is destined to teach @pewdiepie about local agents and post-training.

upvote her plea and make a dream come true.

https://t.co/GygBRgbG6Z