In case anyone is curious:
We’ve had a ~pixel-perfect world model since last year, which is critical for synthetic data generation.
The (much) smaller version of that in the car is a conditional mean-flow model, based on the recent sequence of Tesla Universal Vision Encoder features, which are the same ones that are plugged into the 3B encoder model that produces the driving actions.
We fork off from these UVE features, then run a separate ~1B diffusion model. This takes in the previous reality-view generation with added noise and then denoises it to the current visualization.
It then produces a small video chunk (8 images at 36 fps, at the resolution of the UI screen). This work is done by 1/2 of a TRIP engine every UVE frame at 18 hz and is spread over 4 cycles.
Currently, the latency is a bit high due to compute constraints, but we have ways to interleave newer UVE features along the diffusion transformer layers to cut roughly 35% of the latency.
To Loop or Not 🧵
@thinkymachines just released the incredible Inkling 2 Mini based on looped Transformers. It's in my view the best model release of the year in its class.
So looping clearly wins somewhere, but should we loop every model ?
I also don't know the answer to this question, but there are scientific ways to test it out.
Before getting into it, let's first understand what looping is.
Looping reruns the same block of layers several times per token.
k unique layers × L loops = the compute of kL layers with the weights of k.
One way to think about it is the connection with MoE. Well, it's basically MoE in reverse. MoE buys parameters without FLOPs. Looping buys FLOPs without parameters.
Based on evidence from previous papers, it is already clear that looping is the right choice for memory-capped deployment (laptops, phones, edge), or fixed parameter budgets or data-limited training. Some models like NVIDIA's Dory report drastic verbosity reductions up to 2.5 times less at 4 loops.
Despite these amazing results, It's not evidently clear whether we should loop for every model. Here's my suggested tests:
Test 1: iso-throughput pretraining
Run a looped vs non-looped study with same data, same measured tokens/s on the same hardware (not FLOPs). Report reasoning and knowledge evals separately (see the caveats at the end to know why).
Do you get more gains by running a looped model vs a non-looped model that takes the same amount of time to run on your hardware ?
Test 2: Does looping introduce new gains ?
Comparisons of looped vs normal transformers usually change three things at once:
(1) Weight sharing: every pass reuses the same weights.
(2) The boundary operator: before each pass, it normalizes the hidden state and adds the original input back in, so every pass still sees the input.
(3) Growth: start shallow and add passes midway through training.
Also. let's distinguish what ties vs untied means. A tied model stores one copy of the block's weights and reuses it on every pass. An untied model has the same number of layers, but each pass has its own weights. At the same depth they cost the same FLOPs per token, but untied has more parameters.
Now, let's define a design space here between two classes of models. Ties vs untied.
For each, let's try 2 variants:
(1) No operator, no growth
(2) Operator + growth
Now this is a 2x2 grid. We can define these experiments navigating this space. All of them have the same depth and training tokens, so all four cost the same FLOPs:
Experiment (A): plain looped model
Experiment (B): looped model (e.g. Dory's recipe)
Experiment (C): plain deep transformer
Experiment (D): B without weight sharing
The most important comparison of the batch is B vs D. It lets us understand whether weight sharing is the reason behind the gains.
Test 3: loop dial vs thinking budget at equal serving cost.
At inference, both looping and thinking tokens attemtp to achieve better performance by allocating more compute. Looping spends it on depth so each token costs more. Thinking spends it on length so the model write more tokens. If the plain model can match a looped one just by thinking longer at the same cost, looping adds nothing that models can't already do at test time.
Pretrain two models on the same data with the same recipe. The first is a looped model whose loop count is sampled at random from 1 to 4 during training, so one checkpoint can run at any of those counts. The second is a plain, non-looped model with the same number of unique parameters. Both get the same reasoning post-training, and the plain model is also trained to follow a thinking budget. We then sweep two dials. We run the looped model at 1, 2, 3 and 4 loops with the thinking budget held at the default, in two versions. One that keeps a separate KV cache for each loop and one that shares it across loops.
We sweep the plain model's thinking budget from low to high, so its costs cover the same range as the loop sweep. We evaluate every configuration on two workloads. The first one is the Long-prompt agentic tasks (for example SWE-bench Verified and Terminal-Bench, with contexts in the tens of thousands of tokens) and the second one is short-prompt reasoning (for example AIME and GPQA Diamond).
All runs use the same hardware, inference stack, prefix caching and concurrency. For each run we record accuracy, GPU-seconds per task, time to first token and tokens per second. Cost per solved task is total GPU-seconds divided by the number of tasks solved.
Test 4: loop dial vs thinking budget at equal serving cost.
Looping's strongest case is memory. It adds depth without adding parameters. But quantization also saves memory and both can be combined. Inkling 2 Mini for example trains its expert layers in NVFP4 from pretraining through RL and also loops. So the real question is whether we should do quantization or looping first.
We fix a weight-memory budget, for example 8 GB, and build a grid of bit-widths (16, 8, 4, 3 and 2) by loop counts (1, 2 and 4). At each bit-width we size the model so its weights fill the budget exactly. 4B parameters at 16-bit, 8B at 8-bit, 16B at 4-bit, about 21B at 3-bit and 32B at 2-bit At each bit-width we train two models on the same data and token budget, both at that precision, using quantization-aware training below 16-bit. One is a plain model. The other is a looped model trained with random loop counts so it can run at 2 or 4 loops.
So, this would be 10 training runs and we report the training FLOPs of each. We evaluate all configurations on reasoning tasks (for example AIME and GPQA Diamond) and on closed-book knowledge tasks (for example SimpleQA and TriviaQA). We measure decode latency at batch size one on the device the budget is meant for, such as a laptop. We then plot quality against latency separately for reasoning and knowledge, find the Pareto frontier
Now that we finished the experiment, I would like to also talk about some caveats:
1. KV Cache sharing: In a standard looped model, each loop keeps its own KV cache, so memory grows linearly with the loop count. This makes long contexts and high concurrency expensive, which is why competitive looped models share the cache. PLT (https://t.co/XhkeCTTjhI) shares the KV cache from the first loop with all later loops. NVIDIA's Dory can keep only the final loop's KV cache, which holds memory at the non-looped size while keeping most of the accuracy gain.
But there is no free lunch. You can do KV cache sharing but you should also expect some accuracy loss. A separate cache per loop gives the best quality,so when you share you are approximating, because sharing KV states across loops can discard loop-specific information when representations differ across loops. Also let's not forget that sharing also saves memory, but not compute. You still have to run every token through every loop. Dory, for example, executes 96 layers per token at 4 loops versus 36 at 1, while storing only 36.
2. Looping adds no knowledge: yes, the truth is that looping adds reasoning but not knowledge because stored knowledge scales with parameters and we don't change the number of parameters in looping. That's why I recommended to report reasoning and knowledge evals separately in experiment 1.
Whether looping come victorious from these set of experiments or not, it is still an exciting idea and has already earned in its place for certain applications in LLM architecture design. But for now, we should put our scientific goggles on and investigate.
Disclaimer: I wrote every token you see in this post. No AI was ever used.
I believe (Embodied-)ICL should be done in LLMs since 2024.
Instruction Understanding, Planning, ICL, Memory, Long-Horizon should ALL go to LLMs instead of VLA/WAM.
Free idea for labs (probably you already do a version of this) : take Tatsu's https://t.co/OXr54K2EUk deep learning alchemy class and feed the quizzes and hw to your synth data generator then hill climb. This should at least get some "rsi" or good science of dl abilities for pretrain.
New slide deck on the biggest open problems in post-training and how the field got to where it is today. Largely, this is a statement of my goals with Trillium Labs (not that we can do every project at once). Stay in touch if you're working on any of these in the open!
Opens with the evolution of post-training and ends with a debate on how RSI is changing this type of research.
The research areas on my mind:
1. Harness specialization and design – How do we design environments for context management that enable new paradigms of using LLMs?
2. Character training – How do we create the personality of models and how does this impact safety, trust, and diffusion of LLMs?
3. Open-source RL environments – How do we create scalable, open data methods?
4. Reasoning efficiency in RL – How do we control and manipulate the reasoning efficiency of LLMs?
5. Scaling laws for RL – How do we design principled experimentation methodology for large-scale post-training?
6. Multi-agent scaling laws – What are the main axes of inquiry into multi-LLM systems and collaborations?
Re-recording video soon :) was from a recent visit to Cornell
We’ve been messing around a bit and inadvertently created an experience that’s a bit too fun internally and now everyone is just having fun instead of working. We might not recover from this.