Hongyu Gong

Hongyu is an NLP and speech researcher, currently a research scientist at Meta.

[Hongyu Gong @AnnFirst111]

Shout-out to amazing collaborators: @tuochao @b_veluri @ShyamGollakota

[Hongyu Gong @AnnFirst111]

Excited to share our ACL 2026 paper: AV-Dialog 🎉
The first full-duplex spoken dialogue model that combines audio and visual input to converse in noisy, multi-speaker settings — much like humans solve the "cocktail party problem."
Paper: https://t.co/vQ0jDtmfav

[Hongyu Gong @AnnFirst111]

The future of voice assistants isn't just about hearing — it's about seeing who's speaking, knowing when to jump in, and handling the chaos of real-world conversations. AV-Dialog is a step toward that future.

[Hongyu Gong @AnnFirst111]

[4] Multi-stage training overcomes data scarcity. We align text, audio, and vision on large monadic datasets, then fine-tune on real conversations with interference.

[Hongyu Gong @AnnFirst111]

[3] Explicit turn-taking supervision matters. A dedicated stream that predicts when to speak before deciding what to say dramatically improves response timing in noisy conditions.

[Hongyu Gong @AnnFirst111]

[2] Vision helps identify who to listen to. Adding visual grounding lets the model lock onto the target speaker in crowded scenes.

[Hongyu Gong @AnnFirst111]

We share key findings: [1] Acoustic tokens > semantic tokens in noise. By preserving voice characteristics, our model distinguishes speakers and cuts word error rate from 67% → 31.7% under strong multi-speaker interference.

[Hongyu Gong @AnnFirst111]

Ever tried having a conversation at a noisy party? You naturally look at who's speaking to follow the conversation. Current spoken dialogue models don't do this — they rely on audio alone, making them struggle with multiple speakers, turn-taking, and noisy environments.

[Bandhav Veluri @b_veluri]

Like humans on a phone call, can Spoken LLMs adapt to network latency?

Our #EMNLP2024 paper, SyncLLM (jointly done at @uwcse & @AIatMeta), shows they can learn such an ability... consequently, full-duplex voice AI systems can hide latency with high token throughput! (1/4)

[arXiv Sound @ArxivSound]

``MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservation,'' Yifan Peng, Ilia Kulikov, Yilin Yang, Sravya Popuri, Hui Lu, Changhan Wang, Hongyu Gong, https://t.co/Mo6ruz1TtY

[Yifan Peng @pengyf21]

"Semantic units" and "acoustic units" are widely used in speech language models based on discrete speech representations. After analyzing text-free VALL-E and Voicebox models, we have several findings:
https://t.co/BwrkV5FQJY

[Hongyu Gong @AnnFirst111]

I am excited to share our research in expressive speech-to-speech translation and a suite of Seamless Communication models!

[AI at Meta @AIatMeta]

Yesterday we introduced SeamlessExpressive — a new model that preserves unique vocal styles & expression for speech translation, built on our SeamlessM4T v2 foundation model.

More details on the family of Seamless Communication models ➡️
https://t.co/pPgkrTvryg

[AI at Meta @AIatMeta]

10 years of FAIR.
10 years of advancing the state of the art in AI through open research.

We're celebrating the 10th anniversary of Meta's Fundamental AI Research team and continuing that legacy by sharing our work on three exciting new research projects today.

Details below 🧵

[AI at Meta @AIatMeta]

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model.

This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task.

Details ⬇️

[AI at Meta @AIatMeta]

We believe SeamlessM4T represents a significant breakthrough and as part of our open approach, today we're publicly releasing this work under a CC BY-NC 4.0 license so that others can continue to build on this important field of study.

Get the code ⬇️
https://t.co/K516Icb61C

[AI at Meta @AIatMeta]

Compared to cascaded approaches, SeamlessM4T's single system approach reduces errors & delays, increasing translation efficiency & quality, delivering state-of-the-art results.

Want to see it for yourself, try the demo ➡️ https://t.co/jEbWCE0sL1

[Hongyu Gong @AnnFirst111]

Our teams at Meta AI are hiring research interns in NLP, speech & multimodal translation. If interested, please apply here or reach out directly.
https://t.co/lFVbYlEZDg

[Hongyu Gong @AnnFirst111]

https://t.co/XUxjCtfupC

[AI at Meta @AIatMeta]

Meta AI is sharing new research on speech-to-speech translation (S2ST) that doesn't rely on text generation as an intermediate step. Our method outperforms previous approaches & is the first of its kind trained on real S2ST data for multiple language pairs.https://t.co/AGnZYHKRnm

[Juan Pino @juanmiguelpino]

We are currently hiring for 2 RS/RE positions as well as 1 Postdoctoral position. We are conducting research on exciting topics related to speech-to-speech translation with the aim to solve live multilingual communication. Please reach out if you are interested to learn more!