we started a company!!
so, we’re tackling continual learning: what’s the learning algorithm to take arbitrary data — documents, conversations, the models’ own experience — and make better models? how do we scale compute in the same way we’ve already seen with pre-training and inference time, but scaling on the same data we see as humans, day after day with no labels, no rewards?
A lot of the ingredients are out there already (rl, distillation, long-context, sparse / param-efficient architectures, etc.). our team is at the frontier of these topics, and we’re singularly focused on this. we want to understand this problem better than anyone else in the world.
nobody’s solved this problem yet, but even today it’s extremely greenfield opportunity to co-develop research & useful products. in our space, how people interact with the models defines what the data distribution is - and working on this problem end-to-end, from core science to end user, gives us incredible freedom to define the problem and imagine new kinds of experiences.
i expect we’ll use models that continually learn much differently than we’re using them today. it’ll feel different when the models _just know_, and build on our thinking and direction in ways we can’t even imagine. we don’t even know the queries we’re not asking, the things we would do but aren’t able to today.
i’m so excited to share what we’re doing with the world in the coming months!!
and the team is extremely cracked :) tackling this grand challenge and working alongside @jxmnop@EyubogluSabri@dan_biderman@MayeeChen@__howardchen@shizhehe and many others has made every day so fun. come work with us!
Misinterpreting bitter lesson and an incredibly short period when scaling transformers did most of the legwork has reward hacked brains of a whole generation of scientists.
The more compute you’re pouring in training you’re model, the *more* important it is what algorithm you’re using.
There are algorithms that scale and algorithms that don’t and the only bitter lesson is that a lot of energy was spent on directions that were dead ends in the end. At the same time there are whole dimensions of scaling completely unexplored.
@johnroodepic@EngramLab actually part of what RL can teach the model is what notes to trust -- bc ultimately it's rewarded for the task. notes are just a means to an end
We've been doing some science on how training <> harness memory fit together, and consistently find they stack -- the strongest systems use all kinds of memory
Excited to share some results on a clean and simple setting! Teaching models to use their notes with RL augments base models with memory, and taking gradients is actually pretty cheap
I think the trend will be that a lot of harness/prompt eng today will matter less as we just let models learn how to operate with what they have at their disposal. This is especially relevant as text memory and raw context blurs together - as we've seen in codebases exploding with agent-generated content, learning how to juggle docs and skill.mds itself is becoming a skill :)
In our new blog, we share some new science investigating how RL complements text-based memory by letting models learn how to use their notes.
It turns out that training can be cheap compared to building notes, and stays surprisingly robust even through millions of tokens of edits.
@ryanjosebrosas yes! we kept the result here simple but there's similar benefits to training the model with RL to write notes -- see e.g. @Anikait_Singh_ 's work on Librarian Agents https://t.co/taZ9BmJXLh
i'm actually a bit more optimistic than Grant, even -- it'll take time to adjust, there's still the possibility for people to define "what are new valuable uses in society for math" or "what are things people want that math can help us think more clearly about?"
basically, we're mining the space of things that have been formalized (these used to be hard to solve even when formalized), but it's very hard to formalize things in the first place
Grant Sanderson speaking clearly about what math is really about, and what it can be, is super refreshing.
"for so much of the history of math, we have seen proof as _the_ proxy for understanding ... but at this point, we can kind of see proofs to be ~numerical calculation -- a form of raw data that is helpful for theory building, but it's certainly not the end in and of itself."
Even for something as formalizable as math, the goal is actually insight, not outcomes.
There's certainly problems that are more pure outcome-oriented (curing cancer), but I think many more problems than we think are actually not about the stated objective. As we build bigger and better goal-achieving machines, we'll find out that defining goals that are useful / meaningful to achieve is the hard part, and not something models can ever really answer for us.
https://t.co/vgbGAn6uzY