yucel

learning how not to scale, to understand how to scale better

[Will Held @WilliamBarrHeld]

@giffmana @dlwh @percyliang @Diyi_Yang Part of the issue is how people interpreted the core idea:

Models get predictably *better with scale* (a big model will ~always be better than a small one)

vs.

Models get *predictably* better with scale (you can say meaningful things about big models from small ones)

[yucel @17Ahmetyucel]

@stochasticchasm also very nice paper breakdowns

[yucel @17Ahmetyucel]

@stochasticchasm Do they specifically mention that is a new pretrain somewhere?
I think the base is the same as mimo-v2.5-pro (image from the v2.5-pro model card). So gains seems to be coming from midtraining (stage 2) + rl, which is very impressive.

[Larry Dial @classiclarryd]

At Marin we’ve found that scaling laws let us simulate the entire training trajectory. This became clear in our last 67B MoE run. The run hit the loss target within 1%, but perhaps more interestingly, arbitrary points during training could also be predicted to within 1%. This model is finishing up long context and will soon enter RL. Based on this finding, we’ve used a scaling ladder to simulate the training trajectory of the 535B MoE we kicked off yesterday. Will see how it goes. Now if someone can figure out how to fit a scaling law to each parameter we can just predict our final model and skip this training business.

Something else I've learned from this process is that getting a training run off the ground is way more than just fiddling with the architecture. Lot of herculean efforts from teammates to get trillions of tokens curated, get hardware running on a new GPU stack, improve MFU, and propose great ideas that were tested and integrated.

Given that this is our largest run to date, I anticipate lots of learning and adapting along the way. But thats part of the fun. Details: https://t.co/N7ZxcgvkKs

[yucel @17Ahmetyucel]

@antoine_chaffin @LightOnIO @perplexity_ai Congrats Antoine! Amazing run at lighton, I hope this becomes an even better one.

[Nathan Lambert @natolambert]

It's a good time to share a new @interconnectsai project we made to help make sense of the accelerating open-weight releases these days. The Artifacts Hub builds on our monthly open model roundups and daily monitoring of every model on @huggingface.

The new free resources are:

1. The Artifacts Hub — a curated view of the models trending on Hugging Face, highlighting inference tokens via Open Router, model intelligence via Artificial Analysis, and our tailored adoption metrics building on top of Hugging Face’s data.

2. Our Adoption Dashboard — a living dashboard of download and derivative model numbers by geography and organization. This highlights the US-China gap and growing players in the open ecosystem.

To date, our primary efforts on Interconnects have been release recaps for popular models like Kimi K3, GLM 5.2, DeepSeek R1, etc. and monthly round-ups of the open models that matter, Artifacts Log. We’re expanding on these, building on the tools and internal data we’ve collected for other projects like The ATOM Project (and report). This allows us to capture our ecosystem view of open models, develop methods for understanding adoption of giant MoE models, and everything in between. We’re sharing them freely to help the open ecosystem find its strengths and grow.

The Artifacts Hub right now covers 792 models released in the last two years, across the core text-focused language models and multimodal generative models. At Interconnects we follow the data of every model on Hugging Face, analyze the core few thousand LLMs (this list is public on GitHub and regularly updated), and hand select these core few hundred for further explanation.

For the most popular models, the Hub let’s you quickly see how far behind the model was in terms of frontier intelligence based on Artificial Analysis’s Intelligence Index, compare Hugging Face and Open Router adoption to similar models, glance at relative adoption metric (RAM) scores for time-size normalized downloads, or look at the VAIL similarity index of models with related generations. A snapshot for what you’d see for something like GLM-5.2 is below.

Thanks to @mnshah at VAIL for encouraging us to make this and @huggingface, @OpenRouter, & @ArtificialAnlys to making such useful data openly available.

[yucel @17Ahmetyucel]

@ashley11829 @tilderesearch thanks!

[yucel @17Ahmetyucel]

@antoine_chaffin Now seeing that there are more and better options available is very nice. And not just the dataset I think each of the foundational work done starting with pylate was very important. Thanks to you & the team for open-sourcing and sharing all this work.

[yucel @17Ahmetyucel]

@antoine_chaffin 1/2: When I first started experimenting with training late interaction models for Turkish, I was looking for a dataset to translate as there were very few native ones, and the one I used was of the moderncolbert which helped a lot.

[Antoine Chaffin @antoine_chaffin]

Maybe it looks like "yet another SOTA release from LightOn" (which is already wild), but this one is personal

When I joined LightOn, I looked at the top models and thought: the amount of work to get there is unreachable But it's only truly impossible if you don't try, so we set up a plan and methodically followed it, piece by piece

We built ModernBERT & Ettin, setting a new standard in the encoder world
We built PyLate, so everyone could experiment with late interaction and push the frontier of what was doable
We used it to create strong models, such as LateOn-Code (whose data proved very helpful here!) and ColBERT-Zero, the first fully pre-trained ColBERT model at large scale
We built DenseOn and LateOn, showing that carefully gathered and curated data can create frontier models even outside of closed labs
And now: mDenseOn and mLateOn, compiling everything we've built and extending it to multilinguality, so non-English speakers are not excluded anymore

Everything is public. Every model, every dataset, every piece of code, for anyone to go from random weights to production-ready models

When I look back at the person who thought this was too hard, I could not have dreamed of these resources. I hope they help someone out there who's thinking the same thing right now to give it a shot and come up with their own awesome models

Nothing but pride and gratitude towards everyone who contributed to making this possible. Open source is incredible, and I feel so lucky to be part of it!

[Antoine Chaffin @antoine_chaffin]

"Multilingual when?"
Glad to finally answer today!
Introducing mDenseOn & mLateOn, the next generation of fully open models for multilingual, long-context, and code retrieval
Same openness, same frontier performance, but now for everything

[yucel @17Ahmetyucel]

@antoine_chaffin Congrats on the release! Very nice one as always.

[jianlin.su @Jianlin_S]

arrived as promised

[Kimi.ai @Kimi_Moonshot]

Releasing the model weights and technical report of Kimi K3.

Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.

New model architecture: 2.5x the intelligence per unit of compute, not just more params.

Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.

Model weights: https://t.co/7m7eEg6Y0B
Tech report: https://t.co/yeu6cjpMCT
Tech blog: https://t.co/YTfiMSNM1f

[yucel @17Ahmetyucel]

@swyx @poolsideai @thinkymachines @eliebakouch I think one of the most important thing about their publications is their openness on how they build the models, i.e. the model factory.

[Anton Lozhkov @anton_lozhkov]

For over two years, the largest open code dataset was The Stack v2… until today.

🥞 The Stack v3 is out: the largest open code dataset ever released: 114 TB, 770 languages, 224M repositories, ~5T tokens of deduplicated and filtered source code. Fully open, no restrictively licensed code included.

The upgrade:
- v2 (2024): 68 TB raw -> 2 TB / ~550B tokens, 618 languages filtered
- v3 (2026): 114 TB raw -> 15.9 TB / ~5T tokens, 713 languages filtered

C++ x15, TypeScript x7.5, Rust x7, Python x4.8. Even the COBOL corner of GitHub got a bigger slice.

Part of that is two fresh years of open source. Part of it is a bug we found in v2's deduplication - story below. 👀

Things that make v3 different:
1. Contents inline. The #1 complaint about v2 was "cool dataset, where's the actual code?" v2 shipped file IDs from the Software Heritage graph, and fetching contents was a DIY treasure hunt. v3 is self-contained: sources embedded directly, one row = one repository. Download finishes -> you start training.
2. Fresh crawl. v2 was a 2023 snapshot of even older crawls. v3 is a direct re-crawl of GitHub at the latest commit, completed by August 2025: 224M repositories, 44B files. Forks were only included if they had 5+ stars.

Two ways in:
🥞 stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
https://t.co/px2LW3v6VB
🏗 stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
https://t.co/SQzP5gOmzl

The record isn't really ours to claim - it's millions of everyone's repositories, neatly stacked. Thanks to every developer who keeps their code public, the BigCode community, and friends who built the infrastructure that made this possible!

[Larry Dial @classiclarryd]

Question for the LLM Research Community: Is anyone aware of fully reproducible experimental results showing that MLA beats GQA under the same KV cache? I ask for two reasons: (1) I find it surprising that there is still a divide between Chinese labs using MLA, and Western labs using GQA + sliding window, when certainly many ablations have been run by many parties. (2) At Marin we are considering MLA for our next large scale run, but in early (!) smaller scale ablations it appears worse than heavily tuned feature-rich GQA, even after controlling for KV cache. I would like to run better experiments here. Below I cover my thoughts on general reproducibility, then specifics on MLA.

Every empirical result in ML is only contextually true. Conditioned on the data distribution, optimizer settings, model width, model depth, finer architecture details, hardware, kernel engineering, initialization, token count, tokenizer, context length, and evaluation protocol, one can reach different conclusions.

Contextual results are still useful. Typically if I see a promising method, I will first attempt a full 'context jump', where I apply it to my own context, hoping results transfer. Sometimes they do. If they don't, I can try 2 things: modify the implementation of the method, or modify the context. Ideally I have access to the full context of the original result. Then I can perform a 'context bridge', where I ablate one aspect of the context at a time, isolating exactly why a method performs differently. This lets me make an informed decision to either update my context to let the method shine, or stick with my context and leave the method out.

MLA is tricky to assess at small scale. A core aspect of MLA is compressing hidden_dim->latent_dim. Then for each head, latent_dim->head_dim. Typically head_dim is fixed at 128, partially for hardware reasons, and partially for learning dynamics (head_dim of 8 wouldn't have sufficient representational capacity). To get MLA dynamics, you want hidden_dim>>latent_dim, and latent_dim>head_dim. This window closes at small scale.

The degree of tuning can unfairly alter the scales. In GQA we have partial RoPE, QK Norm, Gated Attention, attention sharpening, sliding window, and other techniques that give a 30%+ training boost. They don't seem to give the same boost to MLA. On one hand, you want to compare techniques apples:apples with equal tuning. On the other hand, there is a finite amount of future tuning you can do, so prior tuning influences which approach is most pragmatic.

Creating controlled tests between MLA and GQA is tricky. Several factors: kv_cache, quadratic attention flops, attention projection flops. kv_cache is controlled by scaling down kv_heads to match MLA, or scaling up kv_latent to match kv_heads. quadratic attention flops are controlled by scaling up GQA's query head count to match MLA head count, or scaling down MLA head count. Also scaling up GQA head_dim 128->192, or scaling down MLA head_dim to 192->128. In general, it's informative to context match to both option A's preferred context and option B's preferred context. Sliding window is another confounder.

MLA is theoretically elegant, if we ignore RoPE. It replaces the 'replicate' op of kv_heads in GQA with a 'mix' op (pic below). Since the 'mix' can learn to 'replicate' if it wants, MLA is purely more expressive, and the cost of 'mix' is hidden at inference with absorb trick. Yet in practice, I find that at small scale this 'mix' op doesn't add much value and interacts poorly with the optimizer dynamics. And the change to RoPE hurts.

My current plan is to first tune and ablate our model features around MLA, then run 3 scaling ladders: MLA, GQA with 2 kv_heads, and GQA with higher kv_heads. For each ladder, fit a loss vs compute projection. If MLA performs worse at our target compute compared to both GQA options, drop it. If MLA beats 2 kv_heads but loses to higher KV_heads, then it becomes a kv_cache tradeoff. Early results indicate MLA will perform worse than both feature-rich GQA ladders, but we will see. Any positive external reproducible results for MLA would help make sure I give it the best chance possible.

[yucel @17Ahmetyucel]

@eliebakouch i would recommend checking out this distillation scaling laws paper: https://t.co/n6JOqbyanV

[yucel @17Ahmetyucel]

@eliebakouch Not frontier but gemma models are also distilled

I think distillation is better objective than pure ntp, also if you have a bigger model and you save the logits while training it is essentially free.