Richard Sutton

Student of mind and nature, libertarian, chess player, cancer survivor. @ UAlberta, amii, The Royal Society, Turing Award

[Richard Sutton @RichardSSutton]

AI researchers seek to understand intelligence well enough to create beings of greater intelligence than current humans.

Reaching this profound intellectual milestone will enrich our economies and challenge our societal institutions. It will be unprecedented and transformational, but also a continuation of trends that are thousands of years old.

People have always created tools and been changed by them; this is what humans do.

The next big step is to understand ourselves.

This is a quest grand and glorious, and quintessentially human.

[Richard Sutton @RichardSSutton]

@vinalgadhiya See the Oak Architecture and the STOMP progression

[Csaba Szepesvari @CsabaSzepesvari]

https://t.co/SxXB2wiVCJ
UofA Math Dept has an opening for Tenure-Track Assistant Professor, subtitled "Mathematical Foundations of AI and Formal Reasoning". Please spread the word! Math dept is excellent, we have a lot of fun together. DM if you are interested and have questions

[Marlos C. Machado @MarlosCMachado]

The Department of Computing Science at the University of Alberta is hiring for two new tenure-track positions this year. Please spread the word:

- Computer Systems for AI and High-Performance Computing: https://t.co/rMJoJXwJFs

- AI for Health: https://t.co/SHBJYJdqIP

[jane ~ in SF @jane_gatsby]

thrilled to have this deep dive published with @PirateWires!!!

a sincere investigation into why so many prominent AI doomers are also extremely kinky, & what this overlap reveals about the movement's underlying psychology

[Chris Watkins @ChrisJCHWatkins]

@RichardSSutton Full announcement and citation here: https://t.co/IpwT6YlPB1

[Chris Watkins @ChrisJCHWatkins]

I am honoured to be awarded the 2026 NEC C&C prize with Andrew G. Barto and Richard S. Sutton @RichardSSutton, for Pioneering Contributions to the Establishment of Reinforcement Learning Concepts and Algorithms.

[Marko Njegomir @njmarko]

There is a lesson to be learned here, taught by @RichardSSutton.

It's bitter, so not everyone will like it at first. But don't worry, it's an acquired taste 🙂

Expert systems, feature engineering, prompt engineering, Tesla FSD that Elon mentioned, and most recently, Harness engineering, just to name a few.

[Marko Njegomir @njmarko]

The Bitter Lesson is knocking on the door of harness engineering. ☠️ 👀

[Richard Sutton @RichardSSutton]

Agree on all three points

[Dan Williams @danwilliamsphil]

On the one hand, I think a strong case can be made that we have made much less progress on AI capabilities than many assume. Frontier AI systems still struggle to learn from limited examples, learn continually from experience, and interact effectively with the physical world. More broadly, their competence is highly uneven. Superhuman performance on tasks they excel at (e.g., maths and coding, where training data can be generated at scale) doesn’t mean they will excel at the much messier tasks involved in most white-collar work.

On the other hand, nobody should find this assessment that comforting. It suggests there is vast room to improve the capabilities of existing systems by changing their design and algorithms. In other words, we could have vastly more powerful systems with the same amount of computing power.

Given that many of the world’s smartest people, assisted by already very capable AI systems and backed by vast resources, are working to accelerate AI research and development, even those sceptical of the current paradigm need to be prepared for the possibility of explosive change once certain capabilities are cracked.

[Richard Sutton @RichardSSutton]

Modern Americans, you are not responsible for slavery. You are not responsible for stealing the land and killing almost all native Americans. Those things happened long ago, before you were born. But you are responsible for the stealing of land and the genocide of Palestinians by the state of Israel. These are directly caused by your bombs and your votes today.

[Richard Sutton @RichardSSutton]

And here is the new PhD thesis of Gautham Vasan (@Gautham529), whose committee I was recently on. I thought this thesis was particularly well done, including the AVG algorithm in Chapter 6, which is a new streaming RL actor-critic (different from the Stream-X algorithms).

Title: Robots That Learn on the Fly Through Real-World Interaction
URL: https://t.co/92MD0ASb2d

Abstract:
A robot that learns from its own runtime experience can improve its behavior over its operational lifetime and adapt to conditions that were never anticipated. The prevailing practice in robot learning provides no such ability: practitioners train a control policy before deployment, from human-provided data or from simulated interaction, and hold it fixed afterward. When performance degrades after deployment, a human engineer may need to diagnose the failure, collect more data or refine the simulator, and retrain and redeploy the policy. Reinforcement learning (RL) from ongoing interaction could reduce reliance on humans for data collection, retraining, and redeployment.
Real-time reinforcement learning names the setting in which an agent senses, acts, and learns from ongoing interaction as wall-clock time advances for both the agent and the environment. Real-time RL on physical robots imposes three practical demands: (a) learning must operate within the robot’s limited computational and memory resources; (b) learning must reach competent performance reliably within a practical wall-clock budget; (c) learning must use rewards that are computable from the robot’s own sensing and simple to specify across tasks. I present three complementary contributions that address the three demands.
The first contribution, the Remote-Local Distributed (ReLoD) framework, supplements the robot’s limited computational resources with a remote workstation. The framework distributes action computation and learning updates between the two computers. Experiments on vision-based tasks showed that the performance of an agent using Soft Actor-Critic, a widely used batch RL algorithm that learns from stored transitions, degraded under limited onboard computation. A robot using ReLoD recovered performance by drawing on remote computation. Across repeated runs, competent policies were learned from scratch within a few hours.
The second contribution is an empirical study of the minimum-time task specification, which assigns a reward of −1 at each timestep and terminates the episode when the goal is reached. Given matched interaction budgets, final policies learned under the minimum-time specification matched or outperformed those learned under guiding reward specifications, which reward progress toward the goal. Using the minimum-time specification, four physical robots learned goal-reaching policies from raw-pixel observations within hours and without external instrumentation.
The third contribution is Action Value Gradient (AVG), a computationally cheap streaming RL algorithm that learns with neural networks. An agent using AVG updates its parameters once from each transition (an observation, action, reward, and next observation) and stores no transitions for reuse. On simulated tasks, agents using AVG outperformed agents using existing streaming RL methods, often achieving final performance comparable to batch RL methods. On physical-robot tasks, agents using AVG learned competent policies within hours on an edge device.
The three contributions address the three demands in complementary ways. ReLoD and AVG meet the computational resource demand from opposite directions: ReLoD supplements a robot’s onboard computer with remote computation when feasible, and AVG is computationally cheap, enabling on-device learning. The minimum-time specification reduces specifying a goal-reaching task to one requirement, detecting the goal from onboard sensing. Across the contributions, physical robots learned competent behavior from scratch within hours. Together, the contributions advance real-time reinforcement learning on physical robots toward the longer-term goal of continual adaptation and long-term autonomy.

Gautham is now a research scientist at Zyphra.

[Richard Sutton @RichardSSutton]

The PhD thesis of my 16th PhD student, Fernando Hernandez Garcia, is now available.

Title: Selective Reinitialization Algorithms for Preventing Plasticity Loss in Artificial Neural Networks
Url: https://t.co/waIzJDCbS0

Abstract:
In this dissertation, I study systems based on artificial neural networks that learn from nonstationary data. Learning from non-stationary data requires continual adaptation of the system, and is often referred to as continual learning. Developing systems capable of continual learning is a longstanding goal in artificial intelligence. In deep learning, the field concerned with designing and training deep neural networks, this goal remains elusive.
This dissertation addresses a fundamental challenge in continual learning: loss of plasticity, a phenomenon where artificial neural network systems progressively lose their ability to learn from new data. While the phenomenon has been noted several times over the last three decades, it has remained understudied until recently. The work in this dissertation constitutes the first systematic demonstrations of plasticity loss, highlighting its persistence and importance.
In this work, I provide a systematic demonstration of plasticity loss in a wide variety of deep learning systems. Across systems based on fully-connected networks, convolutional networks, residual networks, and vision transformers, plasticity degrades when learning continually. Notably, even systems employing normalization techniques, residual connections, and regularization—design choices that improve training stability—remain susceptible to plasticity loss. This evidence establishes that plasticity loss is pervasive and that deep learning systems trained with backpropagation are not suitable for continual learning.
I explore the idea of selective reinitialization to prevent plasticity loss. This idea has been used in the past for improving generalization performance in deep learning systems. However, its use for preventing plasticity loss is a recent innovation pioneered by the continual backpropagation algorithm. The algorithm periodically sets new values for units in the network, making initialization a continuous process rather than a one-time operation. I demonstrate the effectiveness of continual backpropagation in preventing plasticity loss across a wide variety of settings. These demonstrations establish that plasticity loss, while pervasive, is not inherent to deep learning systems.
I generalize continual backpropagation through an algorithm I call selective unit reinitialization. This general algorithm has three key components: a utility measure that ranks units by importance, a pruning criterion that selects which units to reinitialize, and a reinitialization method that assigns new values to the selected units. Continual backpropagation involves a specific choice of utility measure, pruning criterion, and reinitialization method. However, the general algorithm is not limited to the choices used in continual backpropagation. I present a study of how different choices for the three components of selective unit reinitialization affect its effectiveness at maintaining plasticity. This study establishes selective unit reinitialization as a general approach that can be tailored to each learning system to maintain plasticity.
Finally, I propose a different approach to implementing selective reinitialization that operates at the weight level. I call the corresponding algorithm selective weight reinitialization. Reinitializing at the unit or weight level involves different trade-offs. Unit reinitialization minimally disrupts the network outputs, preserving stability during learning. However, the definition of a unit varies across network architectures, requiring additional engineering to make the approach effective. Weight reinitialization, in contrast, can be readily applied to any arbitrary network architecture, but can substantially affect network outputs, reducing training stability. The reduced learning stability can be remedied with L2 regularization, which stabilizes selective weight reinitialization but introduces an additional hyperparameter. Both selective unit and weight reinitialization successfully maintained plasticity across the systems tested, providing flexible approaches for different architectural and engineering constraints.

Fernando is now a research scientist at Zyphra.

[Richard Sutton @RichardSSutton]

The PhD thesis of my 15th PhD student, Shibhansh Dohare (@ShibhanshD), is now available.

Title: Learning Forever using Artificial Neural Networks
Url: https://t.co/Zp2GxZ03aw

Abstract:
In this dissertation, I study the ability of artificial neural networks to learn new things. I refer to this ability as plasticity. Plasticity is a desirable property of intelligent systems in the real world. Systems that interact with the real world can become outdated if they do not learn from the new information revealed to them by the extremely large and partially observable real world. All naturally intelligent systems continually learn new things to keep up with the changes in the world. Many applications like predicting markets or weather, learning human preferences, controlling factories or prosthetics, etc require AI systems that continually learn new things.
I provide direct and thorough demonstrations of loss of plasticity in deep learning systems. The demonstrations range from simple feed-forward networks to deep residual networks and cover a wide range of algorithmic choices. While seeking an understanding of the phenomenon, I found that loss of plasticity occurred simultaneously with hidden units becoming dormant and becoming similar to each other, hampering the ability of the networks to learn new things.
Next, I describe the continual backpropagation algorithm. I designed this algorithm, in collaboration with others, to maintain plasticity in artificial neural networks. Continual backpropagation is a simple extension of the conventional backpropagation algorithm. The conventional backpropagation has two main parts: initialization with small random weights and gradient descent at each step. In addition to gradient descent, continual backpropagation reinitializes a small fraction of units at each step. Continual backpropagation indefinitely maintained plasticity in many continual supervised learning problems.
Finally, I study plasticity of deep reinforcement learning systems. First, I show that plasticity loss has an effect even in stationary reinforcement learning problems. It is unsurprising that plasticity loss affects stationary reinforcement learning because it requires continual learning due to changing policies and bootstrapping. This effect is visible under long-term learning. I found that some commonly used RL algorithms do not scale with data. Their performance drops as they continue interacting with the environment. Plasticity loss hampers the ability of these algorithms to recover and relearn when the performance drops, resulting in agents that do not get better with experience. Continual backpropagation allows these algorithms to recover when performance drops, leading to agents whose performance keeps improving with experience. Second, I show that loss of plasticity is a significant problem in continual reinforcement learning problems, and continual backpropagation maintains plasticity in these problems.

Shibhansh is now a research scientist at Keen Technologies.

[Richard Sutton @RichardSSutton]

In here is a video of The Bitter Lesson as a pretty-good country-music song. Enjoy.
https://t.co/kMLW892Gw1

[Richard Sutton @RichardSSutton]

I do not seek to dominate, or be dominated by, machine intelligences. I seek harmonious coexistence.

[Steven Pinker @sapinker]

An Open Letter to Scott Alexander: The dangers of AI are real, but doomerism may be counterproductive.

https://t.co/HY481etIwD

[Richard Sutton @RichardSSutton]

This looks interesting…

[David Siegel @PullNews]

Please repost the crap out of this until @elonmusk or @Gwynne_Shotwell replies to it:

Grokipedia is dead. I'd like to rescue it, starting with scientific data, which is critical to intelligence (artificial or not) ...

1. LLMs have a GIGO problem. They ingest tons of wrong information and spew it out with certainty.

2. Peer-reviewed research is a disaster. It's a $10 billion industry with bad incentives and terrible results. Feynman would not be happy.

3. Far too many global crises are caused by fake, weaponized, political, and simply incompetent science.

I believe the ONLY way to fix it is not with programming and training, but with an open, two-sided market mechanism where you make money if you're right and you lose money if you're wrong. It's not a prediction market, it's a validation market.

This skin-in-the-game approach will produce the world's most accurate science data, and much more. It's better than community notes. It could become the basis for an entirely new media company that complements https://t.co/gMiSGoXqvo.

DM me and I'll send you my deck.

Sincerely,

David Siegel

@chamath @bgurley @hthieblot @jefielding @larjo280 @naval @pmarca @Jason @paulg @bhorowitz @balajis @fredwilson @msuster @Jeff @hunchventures @jasonmendelson @sether @jeremysliew @VCMike @ChristopherA @jeffnolan @ryanhoover @hstebbings @garrytan @alexf @jaltman @jasonlk @jessicalessin @joshua @khoslaventures @nicole @petersuciu @BillAckman @saranormous @JeffDean @kristianfreeman @alexwg @daveblundin @PeterDiamandis @kcoleman @matthew_pines @shaunmmaguire @dwarkesh_sp @C_Angermayer @RichardSSutton @deedydas @ID_AA_Carmack

[Richard Sutton @RichardSSutton]

@EachWayAI Or, to see the slides, go to https://t.co/dOa3rfOPhn

[Erik Voorhees @ErikVoorhees]

Frontiers are wild, uncontrollable, anarchic.

Frontiers are not balanced, regulated, unified.

Frontiers do not have embedded evaluators.

Frontiers cannot be paced.

That's what makes them frontiers.

[Erik Voorhees @ErikVoorhees]

People killed by AI: zero

People killed by governments: millions

Let's give control of AI to the governments and call it safety