Patrick Pérez

Research director, @GoogleDeepMind

[Patrick Pérez @ptrkprz]

As promised, we are sharing the technology behind Moshi: paper+models+inference code for everyone.

[kyutai @kyutai_labs]

Today, we release several Moshi artifacts: a long technical report with all the details behind our model, weights for Moshi and its Mimi codec, along with streaming inference code in Pytorch, Rust and MLX. More details below 🧵 ⬇️
Paper: https://t.co/mMInmjiBIC
Repo: https://t.co/PFak47FMrm
HuggingFace: https://t.co/bqG4IS0ntg

[Patrick Pérez @ptrkprz]

changing air, entering blue sky, same handle

[Patrick Pérez @ptrkprz]

New sharing step on our journey towards easy-to-use fully-open models.

[kyutai @kyutai_labs]

Meet Helium-1 preview, our 2B multi-lingual LLM, targeting edge and mobile devices, released under a CC-BY license. Start building with it today!
https://t.co/X4Dbx2T1cJ

[Alexandre Défossez @honualx]

I’ll be presenting a deep dive into how Moshi works at the next NLP Meetup in Paris, this Wednesday the 9th at 7pm. Register if you want to attend ! 🧩🔎🟢

https://t.co/1ZPb105JKX

[Patrick Pérez @ptrkprz]

Serious stress testing!

[Neil Zeghidour @neilzegh]

Voice AIs handle speaker turns & interruptions with Voice Activity Detection. VAD is brittle and will trigger due to background noise, creating frequent hiccups. Moshi gets rid of it completely, so you can use it in the most chaotic settings. I myself couldn't hear Moshi here 😅

[Andrej Karpathy @karpathy]

Moshi is a very nice/fun conversational AI audio 🔊 model release from @kyutai_labs .

Are you slowly losing faith in the objective reality and existence of Advanced Voice Mode? Talk to Moshi instead :) You can talk to it on their website: https://t.co/5QaFspEMkj
Or even locally on your Apple Silicon Mac with just:
$ pip install moshi_mlx
$ python -m moshi_mlx.local_web -q 4

I find the Moshi model personality to be very amusing: it is a bit abrupt, it interrupts, it is a bit rude but somehow in a kind of endearing way, it goes off on tangets, it goes silent for no reason sometimes, so it's all a bit confusing but also very funny and meme-worthy. This video "it's just the pressure" / "i just like working on projects" is a good example, soooo funny:
https://t.co/Go1nQMkBnj

But in any case, it's really cool that I can even run this kind of voice interaction with my Macbook, that the repo is out on GitHub along with a detailed paper, and I certainly look forward to effortlessly talking to our computers in end-to-end ways, without going through intermediate text representations that lose a ton of information content.

[Aaron @hertzfelt_io]

Watch @kyutai_labs #moshiai and @OpenAI #gpt4o discuss #AGI. #AI #gpt4ovoice

[Patrick Pérez @ptrkprz]

can even be explored on a vacation beach or a conference center, as Moshi is robust to noisy environments

[kyutai @kyutai_labs]

"Hippie" Moshi tells its love for Hendrix...but "skeptical" Moshi is less enthusiastic about psychedelic rock. Moshi can play 70+ emotions, will you catch them all? Try now at https://t.co/lU2sqa8wMQ

[Patrick Pérez @ptrkprz]

Meet our ambassador!

[kyutai @kyutai_labs]

If you're attending ICML and want to learn more about Kyutai and Moshi, reach out to Edouard!

[Patrick Pérez @ptrkprz]

Staying in real-time connection with voice AI in Paris while being in Vienna

[Edouard Grave @EXGRV]

Moshi goes to #ICML2024 in Vienna! Try the demo at https://t.co/weFG6cmhDT

[Patrick Pérez @ptrkprz]

The attentive listener will notice that even when speaking over Alex, Moshi still listens (taking into account the "in space" instruction for the second poem)

[Alexandre Défossez @honualx]

Some Moshi extracts!
Get your own at https://t.co/SVQZQ9UlEN
Don't forget to click the "Download video" at the end (if it's good) 🟢

[Patrick Pérez @ptrkprz]

And our demo runs in the US thanks to a donation from @huggingface

[Patrick Pérez @ptrkprz]

Thanks @Thom_Wolf Moshi experimental voice AI is indeed a crazy adventure / a radical innovation / a new technology / a surprising experience / a research prototype / a shared resource / a starting point…. not a productized conversational bot.

[Patrick Pérez @ptrkprz]

Thanks @Thom_Wolf Moshi experimental voice AI is indeed a crazy adventure / a radical innovation / a new technology / a surprising experience / a research prototype / a shared resource / a starting point…. not a productized conversational bot.

[Thomas Wolf @Thom_Wolf]

The @kyutai_labs fully end-to-end audio model demo of today is a huge deal that many people missed in the room

Mostly irrelevant are the facts that:
- they come a few week after OpenAI ChatGPT-4o
- the demo was less polished than the 4o one (in terms of voice quality, voice timing…)

Relevant:
- the model training pipeline and model archi are simple and hugely scalable, with a tiny 8+ people team like Kyutai building it in 4 months. Synthetic data is a huge enabler here
- laser focus on local devices: Moshi will soon be everywhere. Frontier model builders have low incentive to let you run smaller models locally (price per token…) but non-profits like Kyutai have very different incentives. The Moshi demo is already online while the OpenAI 4o one is still in limbo.
- going under 300 ms of latency while keeping Llama 8B or above quality of answers is a key enabler in terms of interactivity, it’s game changing, This feeling when the model answer your question before you even finished asking is quite crazy or when you interrupt the model while it’s talking and it react… Predictive coding in a model, instantly updated model of what you’re about to say...

Basically they nailed the fundamentals. It’s here. This interactive voice tech will be everywhere. It will soon be an obvious commodity.

[Patrick Pérez @ptrkprz]

Research internships at @kyutai_labs are fun, beside the hard work! A good session by @RamaAdrien

[kyutai @kyutai_labs]

Moshi is not an assistant, but rather a prototype for advancing real-time interaction with machines. It can chit-chat, discuss facts and make recommendations, but a more groundbreaking ability is its expressivity and spontaneity that allow for engaging into fun roleplay.

[Patrick Pérez @ptrkprz]

It feels so good to have shared at last what we have been up to in the past 6 months. We worked hard on this unique voice AI, carefully training it on a mix of text and speech, making it multi-stream and real-time, and putting it in an online demo for everyone to experience it.

[kyutai @kyutai_labs]

Yesterday we introduced Moshi, the lowest latency conversational AI ever released. Moshi can perform small talk, explain various concepts, engage in roleplay in many emotions and speaking styles. Talk to Moshi here https://t.co/a4EbAQiih7 and learn more about the method below 🧵.

[Patrick Pérez @ptrkprz]

Please @abursuc keep one for me!

[Andrei Bursuc @abursuc]

We've just launched our BRAVO robustness and reliability challenge for semantic segmentation.
I and @tuan_hung_vu will be giving away these nice stickers @CVPR
Ping us or catch us at the posters to find out more!
#CVPR2024

[Amir Zamir @zamir_ar]

We are releasing 4M-21 with a permissive license, including its source code and trained models. It's a pretty effective multimodal model that solves 10s of tasks & modalities. See the demo code, sample results, and the tokenizers of diverse modalities on the website.

IMO, the multitask learning aspect of multimodal models has really taken a step forward. We can train a single model on many diverse tasks with ~SOTA accuracy. But a long way to go in terms of transfer/emergence.

🌐 https://t.co/TsN5JK2Dud
⌨️ https://t.co/CY7NZ2yoFy

Joint work w/ @EPFL_en @Apple.

[valeo.ai @valeoai]

📢We introduce the ScaLR models (code+checkpoints) for LiDAR perception distilled from vision foundation models
tl;dr: don’t neglect the choice of teacher, student, and pretraining datasets -> their impact is probably more important than the distillation method #CVPR2024
🧵
[1/8]

[F. Güney @ftm_guney]

we’ve got multiple PhD and postdoc positions funded by my #ERCstg project ENSURE.

if you’re interested in computer vision and self-driving, please consider applying.
graduate students: apply ASAP! details at https://t.co/kfcbRFyXZe
postdocs: send me an email with your CV and brief research interests at [email protected]

we offer competitive (euro-based) scholarships and salaries for Turkey standards.

[Ian Hogarth @soundboy]

1/ Today the UK's AI Safety Institute is open sourcing our safety evaluations platform. We call it "Inspect": https://t.co/7trBzgw9hw