๐ Excited to share our latest work: Delta-Matching โ Closing the Final Gap in Native 8-bit LLM Training!
Delta-Matching lets us train softmax attention with all seven core matmuls in FP8. In our experiments, it matches BF16/FP32 training loss and preserves overall downstream performance where evaluated.
Delta-Matching computes the correction from the same operands as the gradient. We derive the error introduced by stale delta and track selected channel gains growing before loss rises. In our GDN/GQA runs, larger models show faster gain growth; longer runs reveal degradation that short tests miss.
We tested
โข Attention variants with and without QK normalization, RoPE and NoPE, and head dimensions 128 and 256.
โข GDN/GQA, GDN/MLA, KDA/GQA, Mamba-2/GQA and pure-GQA model architectures.
The experiments span 569M to 5.29B parameters, 8K pretraining and lower-learning-rate context extension from 8K to 64K. We also tested AdamW and Muon-based training, different global batch sizes, and FP8 FFNs and attention projections.
๐ Blog: https://t.co/ypspzxOfKu
๐ Paper: https://t.co/kNl7wWVQAT
๐ป Kernel Code: https://t.co/ZHM7xGtmYO
#EfficientAI #LLM #FP8 #DeepLearning #OpenSource
Want to run a 180B model on your own macbook with 16GB RAM, at a decent decoding speed? Check out our new demo on decoupling MoE experts to SSD!
tldr:
โข Retrain the router to predict experts ๐
โข 9GB peak memory ๐ฅ
โข 14ร faster than llama.cpp ๐
๐ Can making an LLM weaker actually make it smarter?
Excited to share our latest work: Weaken to Strengthen โ Weakening-Induced On-Policy Self-Distillation (WISD)!
๐ก Instead of relying on a stronger teacher, we deliberately weaken the model through quantization or pruning, then use the contrast to improve its reasoning.
No external teacher. No reference solutions. No reward signals. Just the model itself and unlabeled prompts!
๐ฅ Key results:
๐ +4.0 / +5.5 percentage points average accuracy gains with quantization / pruning.
๐งฎ HMMT 2025: 57.7% โ 70.5% (Qwen3.5-4B).
๐ป LiveCodeBench v6: 39.1% โ 56.7% (Qwen3.5-4B).
Sometimes, getting stronger starts with getting weaker!
๐ป Code: https://t.co/olArsYU6AZ
๐ Website: https://t.co/zqPuYKIbR8
#LLM #Reasoning #SelfDistillation #EfficientAI
๐ Excited to share our latest work: Delta-Matching โ Closing the Final Gap in Native 8-bit LLM Training!
Delta-Matching lets us train softmax attention with all seven core matmuls in FP8. In our experiments, it matches BF16/FP32 training loss and preserves overall downstream performance where evaluated.
Delta-Matching computes the correction from the same operands as the gradient. We derive the error introduced by stale delta and track selected channel gains growing before loss rises. In our GDN/GQA runs, larger models show faster gain growth; longer runs reveal degradation that short tests miss.
We tested
โข Attention variants with and without QK normalization, RoPE and NoPE, and head dimensions 128 and 256.
โข GDN/GQA, GDN/MLA, KDA/GQA, Mamba-2/GQA and pure-GQA model architectures.
The experiments span 569M to 5.29B parameters, 8K pretraining and lower-learning-rate context extension from 8K to 64K. We also tested AdamW and Muon-based training, different global batch sizes, and FP8 FFNs and attention projections.
๐ Blog: https://t.co/ypspzxOfKu
๐ Paper: https://t.co/kNl7wWVQAT
๐ป Kernel Code: https://t.co/ZHM7xGtmYO
#EfficientAI #LLM #FP8 #DeepLearning #OpenSource
@AlexRoseJo's CroissAttn studies a related centering problem in FP4 attention with stochastic rounding.
https://t.co/sC4WPIZNv7
We also tried a more aggressive training recipe, combining Muon with FP8 attention, FFNs and attention projections. After 30B tokens, stale delta still raised validation loss and degraded downstream performance, while Delta-Matching matched the BF16/FP32 reference overall. Validation CE was 1.3985 with Delta-Matching, 1.3988 with the reference and 1.8077 with stale delta.
We used one shared initialization. Muon updated the hidden weight matrices, while normalization gains and the remaining parameters stayed on AdamW.
๐ฅ Hi everyone! FlashAttention's backward pass has the same zero-sum bug
@jon_barron
just fixed in softmax. With FP4 stochastic rounding it's wrong on every single draw. We fixed it in our ICLR'27 submission, CroissAttn and unlock e2e FP4 training.
This work derives how FP8 inconsistencies produce stale delta and shows how it distorts training. @jon_barron's work on softmax addresses a similar failure of the zero-sum gradient property for BF16.
In GDN/GQA, selected channel gains grow faster in larger models. During lower-learning-rate context extension to 64K, the stale-delta loss gap exceeds 0.01 only after about 19,000 updates, or 20B tokens.
https://t.co/agkFtUaGCW
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
๐ Introducing Jet-Long โ efficient long-context extension for LLMs with Dynamic Bifocal RoPE.
Extend Qwen3 from 32K โ 128K, Qwen3-30B-A3B-Instruct from 256k โ 1M context with:
โข ๐ซ No fine-tuning
โข ๐ +4.79/+2.18/+2.03 pp on RULER (1.7B/4B/8B) over the strongest baseline
โข ๐ง Best overall HELMET-RAG & lowest PG-19 perplexity
โข โก โค4% generation overhead
Jet-Long dynamically adapts RoPE to the current sequence length while exactly preserving the base model within its pretrained context window.
Long context, essentially free at inference.
๐ Paper (NeurIPS 2026): https://t.co/RAldt7h8un
๐ป Github: https://t.co/yNEUpp01RO
Two moments every ML researcher knows. You get onto a new cluster, and week one goes to fitting the framework to your setup, not training. A new architecture lands, and trying it means hacking through a gigantic codebase to stay compatible with the pipeline. What you want to change is small. The code you wade through to change isn't.
This experience is likely not alone, and many researchers weโve talked to run into similar issues. A year of this on CMU's FLAME cluster left us with one question: what if a framework were built for an agent to adapt and evolve, not just for humans to maintain?
So we introduce PithTrain: a compact, agent-native MoE training system, now ~11K lines of Python, on four principles:
- Compact: fits in one context window
- Python-native: readable tracebacks, no compiled-extension rebuilds
- No implicit indirection: direct calls, each model in its own file
- Agent skills: in-repo playbooks for recurring tasks
Then we measured the thing nobody measures. Same agent, same tasks, only the framework underneath changes: on PithTrain it finishes with up to 62% fewer turns and 64% less GPU time than production frameworks, while training just as fast.
We call this second axis agent-task efficiency, and we believe it deserves to sit alongside training throughput as a metric worth optimizing. Excited to see what people build with it.
Built with amazing collaborators @haok1402, Haozhan Tang, Akaash Parthasarathy, @Zichun_Yu, @junrushao, Todd Mowry, @XiongChenyan and @tqchenml.
Blog: https://t.co/byOKPs9rGQ
Code: https://t.co/AH5ZbwYluV
Paper: https://t.co/hkmDGx9Hc6