Today, we release several Moshi artifacts: a long technical report with all the details behind our model, weights for Moshi and its Mimi codec, along with streaming inference code in Pytorch, Rust and MLX. More details below 🧵 ⬇️
Paper: https://t.co/mMInmjiBIC
Repo: https://t.co/PFak47FMrm
HuggingFace: https://t.co/bqG4IS0ntg
Meet Helium-1 preview, our 2B multi-lingual LLM, targeting edge and mobile devices, released under a CC-BY license. Start building with it today!
https://t.co/X4Dbx2T1cJ
I’ll be presenting a deep dive into how Moshi works at the next NLP Meetup in Paris, this Wednesday the 9th at 7pm. Register if you want to attend ! 🧩🔎🟢
https://t.co/1ZPb105JKX
Voice AIs handle speaker turns & interruptions with Voice Activity Detection. VAD is brittle and will trigger due to background noise, creating frequent hiccups. Moshi gets rid of it completely, so you can use it in the most chaotic settings. I myself couldn't hear Moshi here 😅
Moshi is a very nice/fun conversational AI audio 🔊 model release from @kyutai_labs .
Are you slowly losing faith in the objective reality and existence of Advanced Voice Mode? Talk to Moshi instead :) You can talk to it on their website: https://t.co/5QaFspEMkj
Or even locally on your Apple Silicon Mac with just:
$ pip install moshi_mlx
$ python -m moshi_mlx.local_web -q 4
I find the Moshi model personality to be very amusing: it is a bit abrupt, it interrupts, it is a bit rude but somehow in a kind of endearing way, it goes off on tangets, it goes silent for no reason sometimes, so it's all a bit confusing but also very funny and meme-worthy. This video "it's just the pressure" / "i just like working on projects" is a good example, soooo funny:
https://t.co/Go1nQMkBnj
But in any case, it's really cool that I can even run this kind of voice interaction with my Macbook, that the repo is out on GitHub along with a detailed paper, and I certainly look forward to effortlessly talking to our computers in end-to-end ways, without going through intermediate text representations that lose a ton of information content.
"Hippie" Moshi tells its love for Hendrix...but "skeptical" Moshi is less enthusiastic about psychedelic rock. Moshi can play 70+ emotions, will you catch them all? Try now at https://t.co/lU2sqa8wMQ
The attentive listener will notice that even when speaking over Alex, Moshi still listens (taking into account the "in space" instruction for the second poem)
Thanks @Thom_Wolf Moshi experimental voice AI is indeed a crazy adventure / a radical innovation / a new technology / a surprising experience / a research prototype / a shared resource / a starting point…. not a productized conversational bot.
Thanks @Thom_Wolf Moshi experimental voice AI is indeed a crazy adventure / a radical innovation / a new technology / a surprising experience / a research prototype / a shared resource / a starting point…. not a productized conversational bot.
The @kyutai_labs fully end-to-end audio model demo of today is a huge deal that many people missed in the room
Mostly irrelevant are the facts that:
- they come a few week after OpenAI ChatGPT-4o
- the demo was less polished than the 4o one (in terms of voice quality, voice timing…)
Relevant:
- the model training pipeline and model archi are simple and hugely scalable, with a tiny 8+ people team like Kyutai building it in 4 months. Synthetic data is a huge enabler here
- laser focus on local devices: Moshi will soon be everywhere. Frontier model builders have low incentive to let you run smaller models locally (price per token…) but non-profits like Kyutai have very different incentives. The Moshi demo is already online while the OpenAI 4o one is still in limbo.
- going under 300 ms of latency while keeping Llama 8B or above quality of answers is a key enabler in terms of interactivity, it’s game changing, This feeling when the model answer your question before you even finished asking is quite crazy or when you interrupt the model while it’s talking and it react… Predictive coding in a model, instantly updated model of what you’re about to say...
Basically they nailed the fundamentals. It’s here. This interactive voice tech will be everywhere. It will soon be an obvious commodity.
Moshi is not an assistant, but rather a prototype for advancing real-time interaction with machines. It can chit-chat, discuss facts and make recommendations, but a more groundbreaking ability is its expressivity and spontaneity that allow for engaging into fun roleplay.
It feels so good to have shared at last what we have been up to in the past 6 months. We worked hard on this unique voice AI, carefully training it on a mix of text and speech, making it multi-stream and real-time, and putting it in an online demo for everyone to experience it.
Yesterday we introduced Moshi, the lowest latency conversational AI ever released. Moshi can perform small talk, explain various concepts, engage in roleplay in many emotions and speaking styles. Talk to Moshi here https://t.co/a4EbAQiih7 and learn more about the method below 🧵.
We've just launched our BRAVO robustness and reliability challenge for semantic segmentation.
I and @tuan_hung_vu will be giving away these nice stickers @CVPR
Ping us or catch us at the posters to find out more!
#CVPR2024
We are releasing 4M-21 with a permissive license, including its source code and trained models. It's a pretty effective multimodal model that solves 10s of tasks & modalities. See the demo code, sample results, and the tokenizers of diverse modalities on the website.
IMO, the multitask learning aspect of multimodal models has really taken a step forward. We can train a single model on many diverse tasks with ~SOTA accuracy. But a long way to go in terms of transfer/emergence.
🌐 https://t.co/TsN5JK2Dud
⌨️ https://t.co/CY7NZ2yoFy
Joint work w/ @EPFL_en@Apple.
📢We introduce the ScaLR models (code+checkpoints) for LiDAR perception distilled from vision foundation models
tl;dr: don’t neglect the choice of teacher, student, and pretraining datasets -> their impact is probably more important than the distillation method #CVPR2024
🧵
[1/8]
we’ve got multiple PhD and postdoc positions funded by my #ERCstg project ENSURE.
if you’re interested in computer vision and self-driving, please consider applying.
graduate students: apply ASAP! details at https://t.co/kfcbRFyXZe
postdocs: send me an email with your CV and brief research interests at [email protected]
we offer competitive (euro-based) scholarships and salaries for Turkey standards.