Ming Tu

Research scientist on Speech and Natural Language Processing.

My tweets are my own and can be crawled as training data freely.

[Ming Tu @tuming628]

@JulianSlzr @kmisiunas Glad to see it actually works even without audio reconstruction loss. Will play with it locally in my Hermes agent.

[Ming Tu @tuming628]

@_alex_kirillov_ Congrats on the release

[Ming Tu @tuming628]

@levinstanley @OpenAI Cool was the video edited? It looks so fast

[Ming Tu @tuming628]

Hope it's a full-duplex model

[Sam Altman @sama]

people are really starting to use voice to interact with AI, especially when they have a lot of context to dump.

GPT-Realtime-2 comes to the API today; it is a pretty big step forward.

(we are working on improvements to voice in chat.)

[Ming Tu @tuming628]

it's actually not

[Ming Tu @tuming628]

@rdesh26 could be the full-duplex voice mode

[Ming Tu @tuming628]

@rdesh26 could be the full-duplex voice mode

[Ming Tu @tuming628]

The world doesn't wait its turn. Neither should conversational AI. Let's step into the Full-Duplex era.

[Ming Tu @tuming628]

For the past year, I have been working on the development of Seeduplex. Today, we officially launched the industry's first native full-duplex speech LLM, completely replacing the half-duplex and turn-by-turn system released early last year. https://t.co/rukxhsesFp

[Ming Tu @tuming628]

This architecture is now fully deployed in production on the Doubao App, processing continuous, real-time voice interactions for hundreds of millions of users. Read the technical blog for more details.

[Ming Tu @tuming628]

@JulianSlzr @rdesh26 True. It doesn't need to be an end-to-end model to achieve full-duplex experience, especially if we think text2speech is a tool that can be called when the model/system believe it's time to speak something.

[Ming Tu @tuming628]

Our recent work uses Reinforcement Learning (GRPO) and MLLM-based rewards to fine-tune audio-driven video models, significantly improving lip-sync and natural expressiveness.

[arXiv Sound @ArxivSound]

Weiting Tan, Andy T. Liu, Ming Tu, Xinghua Qu, Philipp Koehn, Lu Lu, "FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation," https://t.co/ZpRIVcx7BN

[Ming Tu @tuming628]

@CarlZha the doort mat with "taoyuanli" kills it. Should be in Chinese caligraphy

[Ming Tu @tuming628]

@rdesh26 One example is using LLM to generate both slides and presentation. In this case, the slides content and spoken content may be different. Then, LLM needs to decide when to take the action to use TTS.

[Ming Tu @tuming628]

For text-only systems, a system includes AI models, tools and harness (tool usage, context&memory management, etc). In this sense, both ASR and TTS can be considered as tools.

[Desh Raj @rdesh26]

Voice AI has grown a lot recently, and definitions of models/systems have become somewhat vague. Let's put down some basics.

1. AI "models" are not AI "systems".
Models are the core units that build up a system. For text-only systems, the two are trivially equivalent (discounting the BPE tokenizer/detokenizer), but not for voice. For voice AI systems, examples of model may be ASR, TTS, LLM, SpeechLLM, OmniLLM, etc.

2. A model is the smallest replaceable unit within a system.
For example, an STT model (user speech in / agent text out) often contains a speech encoder + an LLM, but neither of these components can be replaced without having to train the model again.

3. A speech-to-speech "system" (often called a voice agent) may take many forms and comprise many components, but it is always based on two requirements:
(A) response generation --> what/how to respond
(B) duplex control --> when to talk.
Traditionally, (A) has been handled through an ASR/LLM/TTS cascade. Most of the current S2S modeling research aims to replace this pipeline with fewer models (either STT+TTS or S2S).
Most systems still rely on external VADs and WebRTC for (B), with the famous exception of "full-duplex" models like Moshi.

4a. A SpeechLLM is a model that takes text+speech input, but only generates text output. It is also called a "speech understanding" model.
4b. An OmniLLM is a SpeechLLM that also generates speech (either codecs or continuous latents). It is also called a "speech generation" model (not to be confused with a TTS).

5. A speech-to-speech system is considered "realtime" if it satisfies 3 conditions: low latency (< 1s), streaming audio in/out, and barge-in/interruption handling. It can also be called a full-duplex system (not to be confused with a full-duplex "model").

[Ming Tu @tuming628]

@rdesh26 I believe chatgpt (the system an user actually uses through the app) is such a system. It's not just the model weights.

[Ming Tu @tuming628]

@Teamily_AI @yize_Wang369 @AidenChaoyangHe @avestime how to join the community? is there a discord channel?

[Hao HONG 洪灝, CFA @HAOHONG_CFA]

Seedance made this MV ft. Kayne West. It’s so well made that it’s going viral on Chinese internet.

[Ming Tu @tuming628]

@oran_ge 但是豆包还是要上春晚冲dau。精英阶层的生产力提升需求在规模上是否真的能match普罗大众用豆包p图的需求。dau不是负债，免费token数才是负债

[Ming Tu @tuming628]

It got accepted to ICLR 2026. See ya!

[Ming Tu @tuming628]

Excited to share our new paper, "ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction"!