Looped language models do something subtle:
Each hidden state gets decoded for a prediction AND fed back into future computation. The model carries forward its internal reasoning, not just the token it emitted. It’s the difference between thinking out loud and actually remembering what you thought.
We’ve been wasting the most valuable thing a transformer produces.
Every step, the model builds a rich hidden state, decodes one token from it, then throws the state away. A new paper on looped language models feeds that state back in as runtime memory instead.
For coding agents, context rot is the #1 failure mode. Not reasoning. Not capability.
The model is smart enough to solve the task. The problem is the context doesn't stay clean. Search, exploration, backtracking, all of it piles up noise that poisons every later step.
Clean context beats big context.
Speculative decoding is finally boring enough to be production-ready.
Pair a heavy target model with a lightweight drafter, let idle compute predict ahead, verify the batch in one pass.
If you’re running local/on-device LLMs and not doing this yet, you’re leaving 2-3x on the table.
Everyone’s still optimizing prompt tokens while the actual latency win is at the decode layer.
Gemma 4’s MTP drafters get 3x faster generation with zero quality loss because verification happens in parallel.
Inference is memorybandwidth bound, not compute bound. Build accordingly.
Gemma 4 now runs 2x faster with MTP GGUFs! Run locally on just 6GB RAM. ⚡️
MTP enables Google Gemma 4 run ~1.4–2.2× faster with no accuracy loss.
Gemma 4 12B MTP can run at 162 t/s vs. 52 t/s without MTP. 31B reaches 101 t/s.
GGUFs + Guide: https://t.co/c4gAUlb6YE