Noetic

The experiential learning company

[Noetic @Noetic_Labs]

Our first bit of work on learning from experience

[Matt @0xLienid]

RL is a false god

Its had real impact on reasoning, but compressing feedback into scalar rewards is naive

Why not leverage a model’s priors for credit assignment over richer, token-space signals like code tracebacks or user messages?

Enter the era of Experiential Learning

[Noetic @Noetic_Labs]

RL is not good for continual learning because continual learning is not only about forgetting

Continual learning in any meaningful way requires:
- Learning from outcomes
- In an unstructured output space and environment
- With minimal forgetting

RL is bad at the second

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

RL is actually pretty good at continual learning (depending on how you define it). These results make me feel like RL-based continually learning is possible quite achievable...

Continual learning has recently become a popular topic of discussion. In order to maximize the utility of an LLM, this model needs to actively learn as it is used (i.e., on-the-job learning like humans). Currently, model utility is difficult to unlock because LLMs struggle to adapt or improve autonomously as they are used.

Setup. As a proxy for continual learning, consider a continual post-training setup with T datasets {D_1, D_2, ..., D_T}. Given a base model (e.g., an existing Instruct model), we can sequentially train over each of these datasets. This mimics continual learning behavior, where the LLM is exposed and trained on new tasks over time. This is (obviously) not a perfect proxy, but it's an informative empirical setup.

Metrics. To evaluate the model in terms of continual learning ability, we can consider two metrics:

1. Average accuracy of the model on test sets for all datasets after continual post-training completes.
2. Forgetting measure captures average difference between final accuracy of a task and the best accuracy achieved for that task throughout continual training.

When we compare multi-task SFT training, sequential SFT and sequential RL (GRPO / RLOO) over a set of downstream training datasets, we see that

(1) SFT leads to catastrophic forgetting of previously learned tasks during continual post-training on downstream tasks. Final average accuracy of 54% (much lower than multi-task learning with SFT, 62.9%).

(2) SFT degrades general capabilities too; e.g., 52.1% → 40.1% on MMMU.

(3) RL is naturally robust to forgetting. The forgetting measure is only -2.3%, and final average accuracy is 60%, which is close to upper bound of 62.9% achieved by multi-task SFT.

(4) RL maintains (and even improves) general model capabilities. For example, MMMU improves by 2.1% and POPE improves by 1.9% when running sequential GRPO-based RL training.

RL seems to be very good at maintaining and adding to model capabilities over time naturally! There are not even any replay buffers used for these results. As a disclaimer, there is still a ton of work left to figure out how continual learning would be defined in the real world, and it's possible this sim-to-real gap between proxy benchmarks like this and real life is huge.

“Without any data replay, continual post-training with RL can achieve comparable performance with that of multi-task training, which is not achievable even when equipping SFT with continual learning strategies.”

[Matt @0xLienid]

I very very strongly agree. It is incredibly inefficient, and not at all how intelligent creatures learn. Intelligence scales with converting unstructured experience into knowledge and behaviors.

https://t.co/jdqfklEvY8

[Matt @0xLienid]

Wow. Wonder who could be working on that...

[Matt @0xLienid]

Update:

We've now matched GRPO performance. Again, no scalar rewards, no trajectory filtering for SFT.

Now +36% over GSM8K baseline with Experiential Learning

[Noetic @Noetic_Labs]

If you want more information about Experiential Learning, check out our initial blog post on it here: https://t.co/6kXrgqSQE4

[Noetic @Noetic_Labs]

A bit of progress on Experiential Learning. We're taking into Gym environments, with Frozen Lake being first up.

Using no RL, and no SFT on successful trajectories the 3.2 1B is able to learn how to make proper moves via <move> "tools"!

[François Fleuret @francoisfleuret]

Remember that RL is finding the exit in a pitch black building you never visited before.

[Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex]

Noetic develops «experiential learning framework», with the goal to move beyond RLVR or VR-CLI and their dependence on gold standard answers, make proper use of rich feedback for the general case, and usher in @RichardSSutton's Era of Experience.

[Matt @0xLienid]

We built a way for models to learn from arbitrary experience to move past naive SFT or scalar rewards

The results:
- +17% on holdout HumanEval set from just seeing printouts
- +33% on GSM8K training set from natural language feedback

No labels. No reward. Just experience.

[Matt @0xLienid]

Paper: https://t.co/jdqfklDY8A

[Noetic @Noetic_Labs]

We’ve made some initial steps in addressing this Original Sin of RL at Noetic

Expect improvements and a playground soon
https://t.co/6kXrgqSQE4

[Noetic @Noetic_Labs]

The reduction of all signal to scalars that is foundational to RL is really a scourge on the process of learning from experience

Given enough attempts you can still learn, but it’s not the optimal way of figuring out the optimal behavior

[Andrej Karpathy @karpathy]

Imagine you do 1 hour of intellectually difficult work just to learn that your grade is 0.32 lol

[Noetic @Noetic_Labs]

Andrej points to intellectual work, which is fair, but I Iike to think about it in the context of games

Conscious processing of the failure mode, missing the net left or right, or checkmated by a piece you didn’t notice, is where the greatest progress comes from

[Noetic @Noetic_Labs]

Instead what’s ideal is a purely neural approach relying on some set of excellent priors for processing a state

This is what Experiential Learning does
https://t.co/6kXrgqSQE4

[Noetic @Noetic_Labs]

Dwarkesh is correct here on a couple important angles

First is about learning based on the way in which you failed. In all of RL, the signal gets compressed to a scalar and you have to basically guess at credit assignment

[Dwarkesh Patel @dwarkesh_sp]

.@TrentonBricken and @_sholtodouglas are back

Timestamps:
00:00:00 – Claude 4 & how far RL can scale
00:16:27 – Is continual learning a key bottleneck?
00:31:59 – Model self-awareness
00:50:32 – Taste and slop
01:00:51 – How soon to fully autonomous agents?
01:15:17 – Neuralese
01:18:55 – Inference compute will bottleneck AGI
01:23:01 – DeepSeek algorithmic improvements
01:37:42 – Why are LLMs ‘baby AGI’ but not AlphaZero?
01:45:38 – Mech interp
01:56:15 – How countries should prepare for AGI
02:10:26 – Automating white collar work
02:15:35 – Advice for students

Up on YouTube, Spotify, Apple Podcasts, etc. Enjoy!

[Noetic @Noetic_Labs]

It also doesn’t really scale beyond scenarios you can carefully craft simulated environments for

Trenton is right that in real life there are lots of implicit dense reward signals, but the blunder is in thinking you should explicitly define all of them in a simulation