Mike Lewis

Llama3 pre-training lead. Partially to blame for things like the Cicero Diplomacy bot, BART, RoBERTa, attention sinks, kNN-LM, top-k sampling & Deal Or No Deal.

[Chanakya Ekbote @thecekbote]

🧵 [1/14]: How should we loop a transformer when model size 📦, training compute 🧮, and inference efficiency ⚡ all matter?

Looping is inherently simple: take the hidden representations produced by a transformer (or part of it) and feed them back through the same transformer (or part of it), reusing its weights. 🔁

But every extra loop costs compute. Choosing which layers to loop, how many times, and how to execute those loops determines whether the trade-off is worthwhile.

We explore these choices in our new paper:
“How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference”

Here’s what we found. Thread below 🧵👇

📄 Paper link: https://t.co/kEiYswTrsY

[Nicholas Roberts @nick11roberts]

Great read and congrats on GLM-5.3! Here’s my paper that @jietang cites—optimal tokens/param is task-dependent: memorization wants params, reasoning wants data https://t.co/X9t4eawWDe

He also mentions inference/reasoning cost+scaling. We do this with T² https://t.co/aR07gUxwFY

[Jesse Dodge @JesseDodge]

My team at meta is hiring a 1-year role! Come work with me!
https://t.co/HEfxllsq0z

Qs I'm working on:
What to eval on/hill climb during pretraining to maximize post-train perf?
What evals are most correlated/predictive across pretrain -> midtrain -> post-train?
How to datamix?

[Junlin Han @han_junlin]

We live in a multimodal world. We see, talk, act, and dream.

Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch?

Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition

So what’s the path forward?

Introducing: Towards Physics of Multimodal Pretraining (https://t.co/xgfbdDPFq4)

We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

[Mike Lewis @ml_perception]

@ehsanik @Vercept_ai @AnthropicAI Amazing, congrats!!

[Mike Lewis @ml_perception]

Excited to see what the amazing @sarahookr and team build here!

[Sara Hooker @sarahookr]

Beginnings are very special. Today is an important day for @adaptionlabs.

Today a handful of one-size-fits-all-models are optimized for the average use case.

Averages erase the exceptional.

Everything intelligent adapts. So should AI.

[Jason Weston @jaseweston]

Our team in FAIR at Meta is hiring a (full-time) researcher!

We work on the topics of Reasoning, Alignment and Memory/architectures (RAM) for self-improvement & co-improvement.

Apply here:
https://t.co/Vukp3u8rfu
Location: NY, Seattle or Menlo Park.

Some of our recent work to give flavor:
Co-Improvement (position): https://t.co/XPwbsuCmSy
SPICE (Self-Play in Corpus Environments): https://t.co/47BarIqsFe
Self-Challenging Agents: https://t.co/qgDLmcgPjp
RL from Human Interaction: https://t.co/wmC2fVB0zu
AggLM (parallel aggregation): https://t.co/Fg0E31agT0
StepWiser (CoT-PRM RL): https://t.co/QbfBVYwxcu
DARLING (diversity-trained RL): https://t.co/J9ZSs8GnJp
J1 (RL-trained LLM-as-Judge): https://t.co/yG6xAPafTv
CoT-Self-Instruct: https://t.co/dHMYRxsXfJ
Multi-Token Attention: https://t.co/4kfUe8JQKl

[Mike Lewis @ml_perception]

@edward_milsom You see this because at the start of each epoch, many of your samples were seen recently at the end of the previous epoch. By the end of your epoch, each sample hasn't been seen for at least one epoch, so is less memorized.

[Yen-Ju Lu @Yen_Ju_Lu]

🚀 Introducing the Latent Speech-Text Transformer (LST) — a speech-text model that organizes speech tokens into latent patches for better text→speech transfer, enabling steeper scaling laws and more efficient multimodal training ⚡️

Paper 📄 https://t.co/4nUsbC1YKF

[Mike Lewis @ml_perception]

Love seeing these incredibly creative new evaluations! Optimizing benchmarks is easy, the real challenge is in generalizing to the tasks that don't exist yet

[henry @arithmoquine]

new post. there's a lot in it. i suggest you check it out

[Guangxuan Xiao @Guangxuan_Xiao]

I've written the full story of Attention Sinks — a technical deep-dive into how the mechanism was developed and how our research ended up being used in OpenAI's new OSS models.

For those interested in the details:
https://t.co/0EAi2KQMMx

[Mike Lewis @ml_perception]

@niloofar_mire @CMU_EPP @LTIatCMU @AIatMeta @kamalikac Amazing, congrats!!

[Sharan Narang @sharan0909]

Don’t miss this - I’ve worked with Mike (@ml_perception) very closely at Meta and his talks are super informative and fun.

[Nicholas Roberts @nick11roberts]

📉📉NEW SCALING LAW PHENOMENON 📉📉

We find that knowledge and reasoning exhibit different scaling behaviors!

Super excited to finally tell you all about our paper on the compute optimal scaling of skills:
https://t.co/SH3YCMyIeG

[1/n]

[Mike Lewis @ml_perception]

@achowdhery @reflection_ai Congrats!!

[Qizhen (Irene) Zhang @IreneZhang30]

✨New Preprint✨We introduce 𝐁𝐫𝐚𝐧𝐜𝐡-𝐓𝐫𝐚𝐢𝐧-𝐒𝐭𝐢𝐭𝐜𝐡 (𝐁𝐓𝐒), an efficient & flexible method for stitching together independently pretrained LLM experts (i.e. code, math) into a single, capable generalist model.

Key Takeaways:
✅BTS achieves the best average generalist performance across a variety of tasks 👊
✅We stitch together 4 x 2.7B specialized expert LLMs, where only the lightweight stitching layers (<300M params in total‼) are trained while the experts’ params remain frozen. This makes BTS super modular, flexible, and easy to train! 👊

https://t.co/fIlnCJXj3r
Work done at @AIatMeta w/ @prajjwal_1, Chloe Bi, Chris Cai, @j_foerst @imjeremyhi @punitkoura, Ruan Silva, @shengs1123 @em_dinan* @ssgrn* @ml_perception*
* Joint last author

🧵👇(1/5)

[Artidoro Pagnoni @ArtidoroPagnoni]

🚀 Introducing the Byte Latent Transformer (BLT) – An LLM architecture that scales better than Llama 3 using byte-patches instead of tokens 🤯

Paper 📄 https://t.co/5QGrlJdK0y
Code 🛠️ https://t.co/jCdDI5BXwe

[Weixin Liang @liang_weixin]

How can we reduce pretraining costs for multi-modal models without sacrificing quality? We study this Q in our new work: https://t.co/tjwblRVy8u

At @AIatMeta, We introduce Mixture-of-Transformers (MoT), a sparse architecture with modality-aware sparsity for every non-embedding transformer parameter (e.g., feed-forward networks, attention matrices, and layer normalization). MoT achieves dense-level performance with up to 66% fewer FLOPs!

✅ Chameleon setting (text + image generation): Our 7B MoT matches dense baseline quality using just 55.8% of the FLOPs.
✅ Extended to speech as a third modality, MoT achieves dense-level speech quality with only 37.2% of the FLOPs.
✅ Transfusion setting (text autoregressive + image diffusion): MoT matches dense model quality using one-third of the FLOPs.
✅ System profiling shows MoT achieves dense-level image quality in 47% and text quality in 75.6% of the wall-clock time**

Takeaway: Modality-aware sparsity in MoT offers a scalable path to efficient, multi-modal AI with reduced pretraining costs.

Work of a great team with @liliyu_lili, Liang Luo, @sriniiyer88, Ning Dong, @violet_zct, @gargighosh, @ml_perception, @scottyih, @LukeZettlemoyer, @VictoriaLinML.👏

**Measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs.

[Mike Lewis @ml_perception]

@microth @AnetteMFrank Congrats Michael!!

[Mike Lewis @ml_perception]

@apjacob03 @MIT_CSAIL @jacobandreas @KonstDaskalakis @gabrfarina @roger_p_levy @polynoamial @adamlerer @em_dinan Huge congrats Jacob!!