🚨This week's top AI/ML research papers:
- DiffusionBlocks
- A Bitter Lesson for Data Filtering
- Neural Weight Norm = Kolmogorov Complexity
- When Does LeJEPA Learn a World Model?
- Do Language Models Need Sleep?
- Parallax
- Gemini Embedding 2
- Qwen-VLA
- The MiniMax-M2 Series
- Looped Diffusion Language Models
- LocateAnything
- Learn from your own latents and not from tokens
overview for each + authors' explanations
read this in thread mode for the best experience
today is (potentially) a great day for the GPU poors
if DiffusionBlocks works on fine-tuning existing models, then literally any reasonable consumer GPU can do LLM fine-tuning
will make a video on this
they were pretty conservative with their paper
so here are some bold and cope potentials if it holds up at scale
> 3-4x memory reduction across the board without much quality loss
> train a small/mid sized LLMs on a single GPU
> if you can train each block independently without much comms: less all-reduce, fewer pipeline bubbles, and reduced comms overhead
> if it works on fine-tuning existing models: consumer GPUs/small clusters can fine-tune SoTA models
> if blocks are independent: partial fine-tuning gets cheaper, since you can update subsets of blocks instead of the whole model
feel free to shut me down
Introducing DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
https://t.co/c9AvsRKybj
What if we didn’t have to hold an entire neural network in memory to train it?
Standard neural net training optimizes all parameters jointly. As a result, the memory required during training grows linearly with the depth of the network.
In our #ICLR2026 paper, we propose DiffusionBlocks, a principled framework to train networks one block at a time, drastically reducing memory requirements while matching end-to-end performance.
With DiffusionBlocks, we split the network into blocks and train them one at a time, so you only need memory for a single block.
How? We explicitly assign each block a role: to move the representation a little closer to the target than the block before it did. That role turns out to be precisely what a diffusion model does, step by step. Each block only needs to optimize its own objective and can be trained independently.
We validated this across five different architectures:
• ViT
• DiT
• Masked diffusion
• Autoregressive transformers
• Recurrent-depth transformers
In each case, performance is competitive with end-to-end training while using a fraction of the memory.
This perspective also extends naturally to recurrent-depth (Looped) transformers, which apply the same network iteratively and normally require expensive backpropagation through time (BPTT). Viewed through DiffusionBlocks, we can replace those multiple iterations with a single forward pass during training.
Read our paper and code, to learn more.
Paper: https://t.co/CRj96VGYQn
GitHub: https://t.co/eNW0K9Xh8E
🐟
🚨This week's top AI/ML research papers:
- Self-Distilled Agentic RL
- Long Context Pre-Training with Lighthouse Attention
- Embedded Language Flows
- Negation Neglect
- Efficient Pre-Training with Token Superposition
- Slicing and Dicing
- SlimQwen
- Registers Matter for Pixel-Space DiT
- Scaling Laws for Mixture Pretraining Under Data Constraints
overview for each + authors' explanations
Reinforcing Recursive Language Models
Can a 4B model learn to recursively call itself to answer hard long-context questions?
We RL fine-tuned a small model to behave as a native RLM.
On evidence selection across scientific papers, our 4B RLM matches Sonnet 4.6 in quality while running significantly faster and cheaper.
"Thinking With Visual Primitives" was taken down without reasons
after reading the paper, my take on why they did that might be because the current version shows that visual primitives can make reasoning much more efficient, but it doesn’t fully answer the big picture that is
How much visual detail can you compress away before better referencing stops being enough?
basically like a trade-off between perception and reference gap
They did something similar with engrams (vs MoE), so maybe they wanted to supplement some more ablation results?
which i hope is the case cuz i would love to see the comparison
🚨This week's top AI/ML research papers:
- The Last Human-Written Paper
- Thinking with Visual Primitives by DeepSeek
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
- Qwen-Scope
- Recursive Multi-Agent Systems
- Co-Evolving Policy Distillation
- Representation Fréchet Loss for Visual Generation
- Tuna-2
- Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
- DORA
overview for each + authors' explanations
read this in thread mode for the best experience
be DeepSeek
>need to achieve batch invariance so bad
>split-k is the only optimal solution but is batch variant
>Thinking Machines ($50b valuation) could barely recover the performance for their solution, gets 1.6x slower
>hold_my_beer.jpg
> dual-kernel strategy
>"match or even surpass the perf of standard split-k in most major scenarios"
>DeepSeek strikes again
>$20b valuation btw
https://t.co/oQ8Hr6oOJE
will talk more about it in my vid but this is just incredible that i wanna post about it and i have not seen anyone talked about it
and i might have to make 2 vids at this point (1 arch 1 infra cuz theres just too much cool stuff