We used to scale models by giving them more data.
Then we scaled how long they could think.
Now we’re approaching something far more interesting:
models that can learn while they think.
If TTT works at scale, “training” stops being a phase.
It becomes a continuous process.
ssi are ploughing the fields of a new paradigm shift, TTT (test time training). it will allow the models to update a subset of their weights on the fly.
so your digital co worker won’t forget everything about your business needs in each new chat, continual learning is here.
much like giving models a scratch pad to think and allocating test time compute, ilya has found the next shift and pushed hard.
here’s a cute silly image to show you the third axis we are scaling.
the laws. they hold 🥹
@iruletheworldmo The third axis isn’t just scaling compute.
It’s scaling the model’s ability to learn from the computation itself.
Training-time scaling made models smarter.
Test-time scaling made them think longer.
TTT makes them change.
That’s a very different game. 🍓👀
ok, apologies in advance because i do so hate to provide actual signal.
openai basically sucked at pre-training for a while and crushed at post. their answer to this was spud. spud was the large new pre-train for the new family of models: sol, terra, and luna.
the upcoming model, ‘astra’, has an even bigger pre-train, codenamed ‘doug’. (this was a running joke, as doug was the name for the world’s largest spud, although that’s not technically accurate — look it up).
still with me?
ok good, so sol is a 5t model based on the spud pre-train. astra is a 10t model based on the doug pre-train. this is the model i’ve discussed that openai researchers have been considerably more excited for. not only does it crush fable on all benchmarks, it’s genuinely delightful to talk to, high eq (4.5 vibes), and this is the first model that is genuinely incredible at writing, with output indistinguishable from that of great human writers.
ok im tired and i have a lot of ssi news coming but ill keep you updated on this. everything here is well sourced and im highly confident. very well sourced.