Our work sheds some light on the potential of model growth, but I believe we are only scratching the surface. I am excited by the possibility that Qwen is using it too!
Kudos to @ChangJonathanC and @RhLm2337 for the spotting this! 6/6
Qwen3.5-27B may have been trained from an expanded Qwen3.5-9B checkpoint. If true, what do we know about model growth?
I learned about this technique over the summer and found it interesting. Here's what I learned. 1/6
In our recent work (https://t.co/Omh2LKrLWm), we instead grow models in a single training run. A 5B→7.4B model matches GPT-3 13B on CORE. Moreover, model growth improves scaling exponents, so its compute-efficiency advantage increases with scale. 5/6
Pradheep did a really great job unpacking our paper’s results. I am so glad someone read the appendices and presented them in such a coherent way! (all the hidden efforts are seen 🥺). I strongly recommend giving it a read.
this was a pretty interesting and dense paper on computational depth and the different ways of making models effectively deeper. (the ablations here are insanely dense😌)
my notes: https://t.co/7OsxqqMI6F
the main takeaway is that simply adding more depth isn’t enough. how that depth is introduced, how the passes communicate, and how much of it the model can actually use seem to matter just as much!
kudos to the authors @industrialist@akshayvegesna@charllechen 🙌🏻
@DamienTeney@akshayvegesna@industriaalist@andrewgwils Thanks Damien!
Is that equivalent to no prelude or coda, and with an extra pass over the core blocks? Our preliminary results suggest that prelude and coda layers are beneficial, but we haven't investigated it very rigorously.
1/n We can improve the scaling laws in two ways. While a better constant enables a constant compute efficiency gain across scales, a better exponent leads to a growing one. We show that architectures can improve the latter!
with @akshayvegesna, @industriaalist, and @andrewgwils
We've been obsessed with bending the scaling laws recently and our new paper shows that looping with model growth improves the scaling exponent, leading to gain that compounds with compute!
Everyone assumes architectural changes only give constant factor gains and pretraining progress mostly comes from data (e.g. @dwarkesh_sp's recent post).
We found that model growth, looping, and boundary operators result in compute multipliers over standard transformers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs and 2.7x projected at 1e25.
- Matches GPT-3 13B on CORE with 20x less compute
w/ @charllechen, @akshayvegesna, @andrewgwils
🧵
Scaling laws predict an LLM's pretraining loss, but not its capabilities. Abilities like in-context learning emerge abruptly and only past a certain scale. Our new paper traces this to one bottleneck: learning which tokens attention should focus on. 🧵https://t.co/ja0wc8aK2e
1/ Now that we're running out of data, how do you optimally scale multi-epoch pretraining to hundreds of epochs?
Our first paper from Q! q0 trains a population of models, instead of single model that saturates fast, reaching a dramatically lower loss at *every* epoch budget.
w/ @bishmdl76@akshayvegesna@ShmuelBerman
1/14 Is Muon “better” than Shampoo?
We argue that their relationship parallels Adam's relationship with Signum. Analogous to @lukas_balles and Hennig’s (2018) decomposition of Adam into element-wise scaled Signum, we can decompose Shampoo as left- and right-adapted Muon.