Cengiz Pehlevan

Theoretical neuroscience, theory of neural computation, physics of learning and intelligence. Associate Professor of Applied Mathematics @Harvard SEAS

[Cengiz Pehlevan @CPehlevan]

We collected lecture notes and blog posts by group members about recent topics in deep learning theory here. Hope it is useful!

https://t.co/DWSeSkXtnS

[Kempner Institute at Harvard University @KempnerInst]

Learn more about the broad science-of-pretraining effort that #KempnerInstitute Co-Director @ShamKakade6 discussed in yesterday's #ICML2026 invited talk: https://t.co/Ne8eEaiHIG @alexmeterez @CPehlevan @alex_damian_

[Hanlin Zhang @_hanlin_zhang_]

Presenting our work at the #ICML26 Oral session:
Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

We study a practical question for language models:
given a pre-training compute budget, what downstream performance can we reliably expect after post-training?

📍 #ICML Oral Hall C
🕧 Thu, Jul 9, 4:30 PM–4:45 PM Local Time

[Cengiz Pehlevan @CPehlevan]

@behboodiarash @WangBinxu Thanks!

[Cengiz Pehlevan @CPehlevan]

Our paper on the consistency of diffusion models received an Honorable Mention for the Outstanding Paper at #ICML2026! 🎉 Huge thanks to my brilliant collaborators @WangBinxu and Jacob Zavatone-Veth!

Paper: https://t.co/udAlnQJ6up

[ICML Conference @icmlconf]

🏆Five research papers are recognized as Honorable Mention for the Outstanding Paper award (2/2):

(cont) ...

A Random Matrix Perspective on the Consistency of Diffusion Models

To Grok Grokking: Provable Grokking in Ridge Regression

[Cengiz Pehlevan @CPehlevan]

@neurovium @WangBinxu Thanks, Nima!

[Mary Letey @ ICML @maryiletey]

Transformers are sequence models, but much of in-context learning theory considers prompts of independent examples / tokens.

We asked: what changes when ICL prompts are actually sequential?

I’ll be presenting this at #ICML2026 HiLD Workshop!

[Kempner Institute at Harvard University @KempnerInst]

Congratulations to #KempnerInstitute researchers Binxu Wang, Cengiz Pehlevan, and Jacob A. Zavatone-Veth, whose paper received an #ICML2026 Outstanding Paper Honorable Mention!

Read more about the award-winning paper here:
https://t.co/pDsxxTWkDw

@WangBinxu, @CPehlevan #AI

[Surya Ganguli @SuryaGanguli]

Check out all the amazing work from our @SimonsFdn Collaboration on the Physics of Learning and Neural Computation (https://t.co/TfOKlQxCrE) presented at the main meeting of @ICMLconf #ICML2026

Tuesday

Efficient Learning of Compositional Targets with Hierarchical Spectral Methods,Hugo Tabanelli, Yatin Dandi, Luca Pesce, and Florent Krzakala
https://t.co/wix8AkXVcl

CompleteP for RL: Maintaining Feature Learning When Scaling Deep Reinforcement Learning
M Ganesh Kumar, Adam Lee, Blake Bordelon , Cengiz Pehlevan
https://t.co/Ox6AgFt5LU

Universal One-third Time Scaling in Learning Peaked Distributions
Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore
https://t.co/QzIQf5ANde

Wednesday

A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models, Leonardo Defilippis, Florent Krzakala, Bruno Loureiro, Antoine Maillard
https://t.co/x4zZVfTHP9

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, Lenka Zdeborová
https://t.co/CKWt5qGyOR

A Solvable High-Dimensional Model Where Nonlinear Autoencoders Learn Structure Invisible to PCA While Test Loss Misaligns With Generalization
Vicente Mendes, Lorenzo Bardone, Cédric Koller, Jorge Medina Moreira, Vittorio Erba ⋅ Emanuele Troiani, Lenka Zdeborova
https://t.co/rgDY2ieGJR

Deep networks learn to parse uniform-depth context-free languages from local statistics
Jack T. Parley, Francesco Cagnetta, Matthieu Wyart
https://t.co/yrqjxZTrAp

Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
Indranil Halder, Cengiz Pehlevan
https://t.co/ERCgkiOyIt

On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classification
Matteo Vilucchio, Lenka Zdeborova, Bruno Loureiro
https://t.co/KDpelqRLla

Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation
Zier Mensch, Lars Holdijk, Samuel Duffield, Maxwell Aifer, Patrick Coles, Max Welling, Miranda C. N. Cheng
https://t.co/ikqMhKQOfJ

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Shobhita Sundaram, John Quan, Ariel Kwiatkowski, Kartik Ahuja, Yann Ollivier, Julia Kempe
https://t.co/wjgSrAbJ25

Thursday

Deriving Neural Scaling Laws from the Statistics of Natural Language
Francesco Cagnetta ⋅ Allan Raventos ⋅ Surya Ganguli ⋅ Matthieu Wyart
https://t.co/b6nKYqNulh

Symmetry in language statistics shapes the geometry of model representations
Dhruva Karkada, Daniel Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri
https://t.co/Y58fJt9qmk

A Random Matrix Perspective on the Consistency of Diffusion Models
Binxu Wang, Jacob A Zavatone-Veth, Cengiz Pehlevan
https://t.co/snM0EAxv3Q

Hyperparameter Transfer with Mixture-of-Expert Layers
Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin
https://t.co/dIe32NuQTS

Analytic Bijections for Smooth and Interpretable Normalizing Flows
Mathis Gerdes, Miranda C. N. Cheng
https://t.co/kWCw1EHGn4

Efficient RL Training for LLMs with Experience Replay
Charles Arnal, Vivien Cabannnes, Taco Cohen, Julia Kempe, Remi Munos
https://t.co/EjmNErAFpC

Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation
Dhrupad Bhardwaj, Julia Kempe, Tim G. J. Rudner
https://t.co/MknrTodSaX

What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT
Yunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain, Anthony Hartshorn
https://t.co/S18yTP5fy5

From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers
Ziming Liu, Surya Ganguli, Andreas Tolias
https://t.co/H7eFnYtYQC

[Kempner Institute at Harvard University @KempnerInst]

📢 Just announced!

Join us for the #KempnerInstitute workshop “Learning Dynamics in Natural and Artificial Intelligence: Evolution, Adaptation, and the Foundations of Efficient Learning.”

Learn more, register, or submit an abstract 👉 https://t.co/Qp1g4UnUDO

[Kempner Institute at Harvard University @KempnerInst]

New blog post: Jailbreak Scaling Laws for #LLMs
Prompt-injection attacks can boost jailbreak success from slow polynomial to exponential growth as inference-time samples increase.

New on the Deeper Learning blog: https://t.co/5m0vogzXuB
#AI @_ihalder @cpehlevan @BanerjeeAnnesya

[Scott Linderman @scott_linderman]

I'm excited to share what we're building at Engram! This team is incredible, and we're working on one of the most interesting problems in AI right now: how to build models that are tailored to each person and continually learn from experience. Come join us!

[Matthieu wyart @MatthieuWyart]

@MLStreetTalk Great fun, Tim. The part I'd underline: the same compositional structure that makes data learnable from polynomially few examples is what lets a model recombine them into exponentially many new valid ones: creativity and sample-efficiency are two sides of one coin.

[Machine Learning Street Talk @MLStreetTalk]

> Natural data is "generated" from a constrained hierarchical / compositional function.

> Deep networks learns that hidden structure from polynomially few examples and creatively generate exponentially many valid new ones.

> The depth of the network is what's important to overcome the curse of dimensionality, and potentially invalidate Chomsky's poverty of stimulus argument.

> Prof @MatthieuWyart, a physicist (Johns Hopkins / EPFL) was the senior author of the Random Hierarchy Model.

[Cengiz Pehlevan @CPehlevan]

@KempeLab @amilabs @ylecun Congratulations!

[Dayal Kalra @dayal_kalra]

How do you know if a parameterization (e.g., µP) or a fitted Hyperparameter (HP) scaling law actually gives reliable transfer?

@MBarkeshli and I propose a three-metric framework to quantify the quality of transfer and use it to show that µP’s advantage over SP in Transformers trained with AdamW comes from training the embedding layer fast enough.

Below: speeding up the embedding LR in SP (SP+Embd) recovers µP-like transfer, and slowing it down in µP (µP-Embd) wrecks training with severe instabilities.

A thread 🧵

1/n

[Machine Learning (ML) Papers @Memoirs]

Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover

Indranil Halder, Annesya Banerjee, Cengiz Pehlevan
https://t.co/FWDbqBkpoM [𝚌𝚜.𝙻𝙶 𝚌𝚜.𝙰𝙸]

[Statistics (Machine Learning) Papers @StatsPapers]

An Asymptotic Theory of Chain-of-Thought in In-Context Learning

Kaito Takanami, Cengiz Pehlevan
https://t.co/YGQugkyFfK [𝚜𝚝𝚊𝚝.𝙼𝙻 𝚌𝚘𝚗𝚍-𝚖𝚊𝚝.𝚍𝚒𝚜-𝚗𝚗 𝚌𝚜.𝙻𝙶]

[Binxu Wang 🐱 @WangBinxu]

Happening today!!
looking forward to seeing u in the full day analytic diffusion workshop!
i ll be giving a talk and tutorial at 10:50 ~ 12:00 about the linear lens into diffusion model and how far does this get us!
Featuring works with friends and colleagues @johnjvastola @CPehlevan Jacob ZV !
website https://t.co/AiQUxfK5ps

See our speaker lineup @ArtemKRSV @MasonKamb @ZKadkhodaie @yuancy @CScarvelis synthesizing smoothness, linearity, locality into a coherent story of diffusion generalization thru their epic sequence of papers!

[Mason Kamb @MasonKamb]

If you’re @CVPR: come by our tutorial tomorrow, June 4th 8:30-5:00, on Analytic Understanding of Diffusion Models.

We’ll be covering how and why diffusion models generalize, learning about state-of-the-art analytical theories for their behavior, and covering key open questions.

[Cengiz Pehlevan @CPehlevan]

@hankyang94 Congratulations!