Alexandre TL

Pre-training LLMs @OVHaiLLM in Paris

[Alexandre TL @AlexandreTL2]

muP works great for Mamba !
Zero-shot transfered the learning rate from a 172k model to a 105 model.

Now part of https://t.co/KVFPYoTN80 👇🧵

[Alexandre TL @AlexandreTL2]

@heyyalexwang @leonlufkin @BerenMillidge I dont think their method is possible to train from scratch. They have to learn the attention chunk mass from what a dense model predicts.
HiLS (which is some kind of flat version of ZHA) directly learns the chunk mass so in this case it's trainable from scratch

[Alexandre TL @AlexandreTL2]

@heyyalexwang not with that specific nGPT recipe, but yes, I do think there is something to get out of nGPT, all the recent stream of work around Hyperball/MuonMD are essentially nGPT (it's being tested live in the Marin hero run) https://t.co/FQAM4CABBC

[Alexandre TL @AlexandreTL2]

Got the chance to test out the nGPT implementation of @NousResearch but unfortunately their baseline (and their nGPT) is far behind nanoGPT, let alone modded-nanoGPT (11/08/24 record so very old, before Muon and stuff)

[Alexandre TL @AlexandreTL2]

@loiccabannes Génial !!

[Alexandre TL @AlexandreTL2]

@ZakShark Bonne lecture 😎 J'espère faire une version en français d'ici pas longtemps, si tu as des retours je suis preneur

[Alexandre TL @AlexandreTL2]

@SwayStar123 The font used in Figure 1 also

[Alexandre TL @AlexandreTL2]

Come join us to work on very interesting research & engineering tasks around LLMs !

[Jean-Gabriel BARTHELEMY @JG_Barthelemy]

We're looking for experienced researchers and engineers to work on and train our future large language models. We are hiring for multiple positions in the following roles:

- A research-engineer with experience in advanced data preparation techniques on trillions of tokens

- A research-engineer with experience designing hybrid and linear attention architectures

- A research-engineer with experience in advanced agentic RL techniques for fine-tuning LLMs on complex, long-horizon tasks

If you have a few years of industry or academic experience (PhD or postdoc), you can apply here:

https://t.co/WDVUBNCcN2

If you're interested in working in the field but have less experience, you can apply for our junior researcher position:
https://t.co/0jaBuGeES1

[Alexandre TL @AlexandreTL2]

@seygalare @pp0196 ahah, ben disons que l'attention linéaire donne des capacités plus difficilement apprenables avec l'attention (et vice versa), donc pour moi avoir les 2 dans une architecture est un plus

[Alexandre TL @AlexandreTL2]

@seygalare @pp0196 j'ai aussi fait une série de lives, moins vulagérisés eux https://t.co/LPFeerwt8G

[Alexandre TL @AlexandreTL2]

@ZakShark des GAN à grande échelle

[Alexandre TL @AlexandreTL2]

@hr0nix https://t.co/V3ru9K9m6d

[Alexandre TL @AlexandreTL2]

@serendip410 @XiaotianHan1 @seanliu616 Nice update! Does it still uses a dense MLP for the first layer ? And do you think this can be relaxed thanks to LLAL ?

[Alexandre TL @AlexandreTL2]

@ahatamiz1 Is it possible to see how you built the optimizer on this one ? I mean some parameters in GDN like the conv are a bit tricky, this you stick with Muon for those?

[Alexandre TL @AlexandreTL2]

@rosinality as often, batch size study is left aside :/

[Alexandre TL @AlexandreTL2]

@AlexiGlad really cool! have you tried applying this idea to the auto-regressive regime ?

[Alexandre TL @AlexandreTL2]

@zsophiaaa17 Thank you for the feedback and glad you liked it!

[Sophia Zhang @zsophiaaa17]

Just finished reading The Little Book of RL by @AlexandreTL2.

Super clear, compressed walkthrough of the underlying math paired alongside minimal code. Great entry point for building core RL intuitions!

Excited to dive into my own RL experiments next!

[Alexandre TL @AlexandreTL2]

@From0toDS @francoisfleuret Thanks ! I hope this book will update your prior on RL ;)

[Alexandre TL @AlexandreTL2]

@eliebakouch it's grouped rather than gated, surely referring to https://t.co/k7h7k5iW8J

[Alexandre TL @AlexandreTL2]

@eigen_paul @RichardSSutton 🤗

[Alexandre TL @AlexandreTL2]

@gabrielnoya7 Thanks !!