Alexandre TL

Intern at @DragonLLM in Paris. (Pre|post)-training LLMs

[Alexandre TL @AlexandreTL2]

muP works great for Mamba !
Zero-shot transfered the learning rate from a 172k model to a 105 model.

Now part of https://t.co/KVFPYoTN80 👇🧵

[Alexandre TL @AlexandreTL2]

@stochasticchasm awesome thank you

[Alexandre TL @AlexandreTL2]

@NousResearch very similar to patch-level training!! https://t.co/BNJIMx4yc5

[Alexandre TL @AlexandreTL2]

@bibek_poudel_ @hive_echo @aidan_mclau ok interesting thanks!

[Alexandre TL @AlexandreTL2]

@bibek_poudel_ @hive_echo @aidan_mclau do you have any source for that?

[Alexandre TL @AlexandreTL2]

@Ji_Ha_Kim yep second idea is https://t.co/UqqXAgj7d2

[Alexandre TL @AlexandreTL2]

@eliebakouch Fig 10 of HyperP paper, although it's more about predicting stability rather than instability https://t.co/01IfYH9W9c

[Alexandre TL @AlexandreTL2]

@WilliamBarrHeld Ok interesting!

[Alexandre TL @AlexandreTL2]

@QuasarModels @Farahatyoussef0

[Alexandre TL @AlexandreTL2]

@QuasarModels Interesting! The KDA baseline seems already very strong tho, 99% at 10M context length. Is it the same model as in Table 7 ? Also, would it be possible to have the raw NIAH scores for the two models of Table 7?

[Alexandre TL @AlexandreTL2]

@QuasarModels prev image is Table 13. this is Table 7:

[Alexandre TL @AlexandreTL2]

@Ji_Ha_Kim Looks good indeed!

[Alexandre TL @AlexandreTL2]

@Ji_Ha_Kim uhm I meant in general, even if batch size is not fixed, it disappears in the ratio : mB/mD = (B'/B)/(D'/D) where D'=B'*num_iters and D=B*num_iters_base so the B' and the B cancel

[Jean-Gabriel BARTHELEMY @JG_Barthelemy]

We’ve been lucky enough to test Mamba-3 ahead of the curve. 🧪

Here is how it integrates into Hybrid Models (Spoiler: it unlocks Muon for SSMs for the first time). 🧵

[Alexandre TL @AlexandreTL2]

@vfleaking @wen_kaiyue

[Alexandre TL @AlexandreTL2]

This is basically Hyperball + GeoNorm

[Francesco Bertolotti @f14bertolotti]

Interesting optimizer that projects the gradient onto a tangent n-dimensional ball on the loss landscape. Unfortunately, the experiments are fairly basic, but i quite like the idea.

🔗https://t.co/cjljvNkWCJ

[Alexandre TL @AlexandreTL2]

https://t.co/ERht8WNd9X

https://t.co/xWnrdsOqfX

[Alexandre TL @AlexandreTL2]

@wen_kaiyue cool ok thx!